Probabilistic PARAFAC2

Abstract

The Parallel Factor Analysis 2 (PARAFAC2) is a multimodal factor analysis model suitable for analyzing multi-way data when one of the modes has incomparable observation units, for example, because of differences in signal sampling or batch sizes. A fully probabilistic treatment of the PARAFAC2 is desirable to improve robustness to noise and provide a principled approach for determining the number of factors, but challenging because direct model fitting requires that factor loadings be decomposed into a shared matrix specifying how the components are consistently co-expressed across samples and sample-specific orthogonality-constrained component profiles. We develop two probabilistic formulations of the PARAFAC2 model along with variational Bayesian procedures for inference: In the first approach, the mean values of the factor loadings are orthogonal leading to closed form variational updates, and in the second, the factor loadings themselves are orthogonal using a matrix Von Mises–Fisher distribution. We contrast our probabilistic formulations to the conventional direct fitting algorithm based on maximum likelihood on synthetic data and real fluorescence spectroscopy and gas chromatography–mass spectrometry data showing that the probabilistic formulations are more robust to noise and model order misspecification. The probabilistic PARAFAC2, thus, forms a promising framework for modeling multi-way data accounting for uncertainty.

1. Introduction

Tensor decompositions are multi-way generalizations of matrix decompositions such as principal component analysis (PCA): A matrix is a second-order array with two modes, rows and columns, while a data cube is a third order array with the third mode referred to as slabs. When multi-way data have an inherent multi-linear structure, the advantage of tensor decomposition methods is that they capture this intrinsic information and often provide a unique representation without needing further constraints such as sparsity or statistical independence.

Applications of tensor factorization originated within the field of psychometrics [1,2] and have been widely useful in other fields such as chemometrics [3], for example, to model the relationship between excitation and emission spectra of samples of different mixed compounds obtained by fluorescence spectroscopy [4]. Tensor decomposition is today encountered in practically all fields of research including signal processing, neuroimaging, and information retrieval (see also [5,6,7]).

The two most prominent tensor decomposition methods are (i) the Tucker model [8], where the so-called core array accounts for all multi-linear interactions between the components of each mode, and (ii) the CandeComp/PARAFAC (CP) model [1,2,9], where interactions are restricted to be between components of identical indices across modes, corresponding to a Tucker model with a diagonal core array. Both models can be considered generalizations of PCA to higher-order arrays, with the Tucker model being more flexible at the expense of reduced interpretability. The CP model has been widely used primarily due to its ease of interpretation and its uniqueness [6,10].

In the CP model, the components are assumed identical across measurements, varying only in their scaling. In many situations, this is too restrictive—for example, when the number of samples vary across a mode. Furthermore, violation of the CP structure within chemometrics can be caused by retention time shifts [11,12], whereas in neuroimaging, such violations may be induced by subject and trial variability [6] invalidating the use of the CP model. To handle variability while preserving the uniqueness of the representation, the Parallel Factor Analysis 2 (PARAFAC2) model was proposed [2]. It admits individual loading matrices for each entry in a mode while preserving uniqueness properties of the decomposition by imposing consistency of the Gram matrix (i.e., the loading matrix left multiplied by its transpose, thereby imposing consistency in how components are co-expressed across samples) [13,14,15]. It has since been applied within diverse application domains, including handling variations in elution profiles due to retention shifts in chromatography [11]; monitoring and fault detection facing unequal batch lengths in chemical processes [16]; in neuroimaging to analyze latency changes in frequency resolved evoked EEG potentials [17], to extract common connectivity profiles in multi-subject fMRI data accounting for individual variability [18], and to characterize dynamic functional connectivity [19]; for cross-language information retrieval [20]; as well as for music and image tagging [21,22]. Recently, efforts have been made to scale the PARAFAC2 model to large-scale data [23,24,25], enhance the robustness and efficiency of the conventional direct fitting algorithm [26,27], and apply a non-negativity constraint also on the varying mode [28,29] as well as broader sets of constraints based on alternating directions of the method of multipliers [30].

Traditionally, tensor decompositions have been based on maximum likelihood inference using alternating least squares estimation in which the components of a mode are estimated while keeping the components of other modes fixed. Initial probabilistic approaches defined probability distributions over the component matrices and the core array but relied on maximum likelihood estimates for determining a solution [31,32]. However, the Bayesian approach presented here makes inference with respect to the posterior distributions of the model parameters and can thus be used to assess uncertainty in the parameters and noise estimates. Most work on probabilistic tensor decomposition has focused on the TUCKER and CP models using either Markov Chain Monte Carlo (MCMC) sampling [33,34,35] or variational inference [36,37,38,39]. The CP and Tucker models have been extended to model sparsity [35,40,41], non-negativity [42], and non-linearity [33,43] in component loadings. Heteroscedastic noise modeling has been discussed in the context of the CP model [41,44,45] and Tucker model [46], the latter also providing a generalization of tensor decomposition to exponential family distributions. A review and toolbox for probabilistic tensor decompositions are given in [45]. For component matrices with orthogonal components, recent work has explored using the von Mises–Fisher Matrix (vMF) distribution in the CP model [47] and the block-term decomposition model defined as a sum of Tucker models [48]. The former used a MAP-based estimation—which is not a fully Bayesian approach—and the latter used a variational Bayesian inference approach. In addition to using a variational Bayesian inference approach to the vMF distribution, we also explore another orthogonal formulation that is applicable beyond the PARAFAC2 structure.

Benefits of probabilistic modeling include the ability to account for uncertainty and noise while providing tools for model order selection. Whereas probabilistic modeling can be directly applied to the CP and TUCKER models extending probabilistic PCA [49], a probabilistic treatment of the PARAFAC2 model faces the following two key challenges:

(i)

The ability to impose orthogonality on variational factors (necessary for imposing the PARAFAC2 structure).

(ii)

Handling the coupling of these orthogonal components.

In this paper, we address these two challenges and derive the probabilistic PARAFAC2 model. In particular, we investigate two different formulations of the orthogonality constraint and demonstrate how the orthogonality of variational factors as in the least squares estimation for conventional PARAFAC2 can be obtained in closed form using the singular value decomposition. We exploit how the probabilistic framework admits model order quantification by the evaluation of model evidence and automatic relevance determination. We contrast our probabilistic formulation to conventional maximum likelihood estimation on synthetic data as well as fluorescence spectroscopy and gas chromatography–mass spectrometry data, highlighting the utility of the probabilistic formulation facing noise and model order misspecification (A short workshop contribution in brief presenting the proposed probabilistic PARAFAC2 was presented in [50]).

2. Methods

The three-way CP model can be formulated as a series of coupled matrix decompositions,

$${\mathit{X}}_k=\mathit{A}{\mathit{D}}_k{\mathit{F}}^{\top }+{\mathit{E}}_k,$$

where ${\mathit{X}}_k\in {\mathbb{R}}^{I\times J}$ is the k’th slab of the three-way array $\mathcal{X}$ with dimensions $I\times J\times K$. Let M be the number of components in the model; then, the matrix $\mathit{A}$ with dimensions $I\times M$ contains the loadings for the first mode and $\mathit{F}$ with dimensions $J\times M$ contains the loadings for the second mode. The matrices ${\mathit{D}}_k$, $k=1,\dots ,K$, are diagonal with dimensions $M\times M$ and contain the loadings for the third mode. These are usually written as a single matrix $\mathit{C}\in {\mathbb{R}}^{K\times M}$, where the k’th row contains the diagonal of ${\mathit{D}}_k$. ${\mathit{E}}_k$ denotes the residuals for the k’th slab with dimensions $I\times J$. Notice that the structure of the first and second mode are invariant across the third mode in this model.

The PARAFAC2 model extends the CP structure by letting a mode have individual factors ${\mathit{F}}_k$ for each slab. The extension allows for a varying number of observations in the chosen mode. This model would be as flexible as PCA on the concatenated data $[{\mathit{X}}_1,{\mathit{X}}_2,\dots {\mathit{X}}_K]$ if not for the additional constraint that each Gram matrix of ${\mathit{F}}_k$ be identical, ${\mathit{F}}_k^{\top }{\mathit{F}}_k=\mathsf{\Psi }$, which is a necessary constraint in order to obtain unique solutions [51]. The three-way PARAFAC2 model can thus be written as

$${\mathit{X}}_k=\mathit{A}{\mathit{D}}_k{\mathit{F}}_k^{\top }+{\mathit{E}}_k\mathrm{s}.\mathrm{t}.{\mathit{F}}_k^{\top }{\mathit{F}}_k=\mathsf{\Psi }.$$

Modeling $\mathsf{\Psi }$ explicitly can be difficult, but it is necessary and sufficient [15] to have ${\mathit{F}}_k={\mathit{P}}_k\mathit{F}$, with ${\mathit{P}}_k$ being a columnwise orthogonal $J\times M$ matrix and $\mathit{F}$ a $M\times M$ matrix; thus, the model can be written as

$${\mathit{X}}_k=\mathit{A}{\mathit{D}}_k{\mathit{F}}^{\top }{\mathit{P}}_k^{\top }+{\mathit{E}}_k\mathrm{s}.\mathrm{t}.{\mathit{P}}_k^{\top }{\mathit{P}}_k=\mathbf{I}.$$

(1)

In the following, we describe the conventional direct fitting algorithm [15] for parameter estimation in the PARAFAC2 model before we introduce the probabilistic model formulation in Section 2.3.

2.1. Direct Fitting Algorithm

The parameters in the PARAFAC2 model in (1) can be estimated using the alternating least squares algorithm [15], minimizing the constrained least squares objective function,

$$\underset{\mathit{A},\mathit{F},\{{\mathit{P}}_k,{\mathit{D}}_k\}{\forall }_k}{arg\; min}\sum _k{\parallel {\mathit{X}}_k-{\mathit{A}\mathit{D}}_k{\mathit{F}}^{\top }{\mathit{P}}_k^{\top }\parallel }^2\mathrm{s}.\mathrm{t}.{\mathit{P}}_k^{\top }{\mathit{P}}_k=\mathbf{I}.$$

For fixed $\mathit{A}$, ${\mathit{D}}_k$, and $\mathit{F}$, the ${\mathit{P}}_k$ that minimizes the k’th term in the objective function is equal to

$$\underset{{\mathit{P}}_k}{arg\; max}\mathrm{Tr}\left(\mathit{F}{\mathit{D}}_k{\mathit{A}}^{\top }{\mathit{X}}_k{\mathit{P}}_k\right)$$

(2)

and can be computed as [15,52]

$${\mathit{P}}_k={\mathit{V}}_k{\mathit{U}}_k^{\top }$$

(3)

where ${\mathit{V}}_k$ and ${\mathit{U}}_k$ come from the singular value decomposition (SVD) decomposition

$${\mathit{U}}_k{\mathit{S}}_k{\mathit{V}}_k^{\top }=\mathit{F}{\mathit{D}}_k{\mathit{A}}^{\top }{\mathit{X}}_{\mathit{k}}.$$

Upon fitting ${\mathit{P}}_k$, each slab ${\mathit{X}}_k$ of the tensor can be projected onto ${\mathit{P}}_k$, thereby leaving the remaining parameters to be fitted as a CP model minimizing

$$\underset{\mathit{A},\mathit{F},\left\{{\mathit{D}}_k\right\}}{arg\; min}\sum _k{\parallel {\mathit{X}}_k{\mathit{P}}_k-{\mathit{A}\mathit{D}}_k{\mathit{F}}^{\top }\parallel }^2.$$

(4)

A solution to (4) is well explained by Bro in [3]. A well-known issue with maximum likelihood methods is that it can lead to overfitting due to noise and a lack of uncertainty in the model parameters, resulting in robustness issues, which we attempt to provide a solution for by advancing the PARAFAC2 model to a fully Bayesian setting.

Model Selection

A general problem for latent variable methods is how to choose the model order, M. A popular heuristic can be formed by how well the model fits the data given as

$$\mathrm{R}2=1-{\displaystyle \frac{{\displaystyle \sum _k}{\parallel {\mathit{X}}_k-{\mathit{A}\mathit{D}}_k{\mathit{F}}^{\top }{\mathit{P}}_k^{\top }\parallel }^2}{{\displaystyle \sum _k}{\parallel {\mathit{X}}_k\parallel }^2}}.$$

(5)

However, this measure will simply increase until the model incorporates enough parameters to completely fit the data, thus eventually leading to overfitting. The model selection criterion would only be based on the expected noise level.

Another popular heuristic is the core consistency diagnostic (CCD), originally developed for the CP model [53], but that has shown useful for the PARAFAC2 model as well [54]. It is based on the observation that the CP model can be seen as a constrained Tucker model, where the core array is enforced to be a superdiagonal array of ones. The principle behind CCD is to measure how much the CP model violates this assumption of a superdiagonal core array of ones by re-estimating the core array of the CP model to fit the Tucker model, denoted $\mathcal{G}$, while keeping the loadings fixed and then calculating the CCD according to

$$\mathrm{CCD}=100\left(1-{\displaystyle \frac{||\mathcal{G}-{\mathcal{I}||}_{\mathcal{F}}^2}{{\left|\right|\mathcal{I}\left|\right|}_{\mathcal{F}}^2}}\right),$$

in which $\mathcal{I}$ is the superdiagonal core array and $\left|\right|·{\left|\right|}_{\mathcal{F}}$ denotes the Frobenius norm. The PARAFAC2 model can be written as a CP model for each slab as in (4); thus, the core array can be estimated in the same way as for the standard CP model. This approach was evaluated on synthetic as well as real data sets by [54], where the conclusion was that even though the CCD can be seen as an useful parameter for determining model order, it is not recommended in practice without considering other diagnostic measures, including inspecting the residuals and the loadings.

2.2. Variational Bayesian Inference

In Bayesian modeling, the posterior distribution of the parameters $\mathit{\theta }$ is computed by conditioning on the observed data $\mathit{X}$ using Bayes’ rule, $p\left(\mathit{\theta }\right|\mathit{X})=p(\mathit{X}\left|\mathit{\theta }\right)p\left(\mathit{\theta }\right)/p\left(\mathit{X}\right).$ The posterior is thereby given as the product of the likelihood $p\left(\mathit{X}\right|\mathit{\theta })$ and the prior probability of the parameters $p\left(\mathit{\theta }\right)$ divided by the probability of the observed data $p\left(\mathit{X}\right)$ under the model, also known as the marginal likelihood. Evaluating the marginal likelihood is, in general, intractable; instead, a variational approximation can be found by fitting a distribution $q\left(\mathit{\theta }\right)$—called the variational distribution—to the posterior [55] minimizing the Kullback–Leibler (KL) divergence, given by

$$q^{★}\left(\mathit{\theta }\right)=\arg \min \mathrm{KL}\left[q\left(\mathit{\theta }\right)\parallel p\left(\mathit{\theta }\right|\mathit{X})\right].$$

Minimizing the KL divergence is solved by maximizing a related quantity, the evidence lower bound (ELBO).

$$\mathrm{ELBO}\left(q\right(\mathit{\theta }\left)\right)=\mathbb{E}[logp(\mathit{\theta },\mathit{X}\left)\right]-\mathbb{E}[logq(\mathit{\theta }\left)\right].$$

A common choice is a variational distribution that factorizes over the parameters, known as a mean-field approximation, $q\left(\mathit{\theta }\right)={\prod }_j{q}_j\left({\mathit{\theta }}_j\right)$. Note that, for convenience, we choose distributions belonging to the exponential family, as this allows closed-form solutions to be found. The optimal variational distribution can then be found by iterative updates of the form

$$q_j\left({\theta }_j\right)\propto exp\left({\mathbb{E}}_{-j}[logp({\theta }_j,{\mathit{\theta }}_{-j},\mathit{X})]\right),$$

(6)

where ${\mathbb{E}}_{-j}[·]$ denotes the expectation over the variational distribution except $q_j$. For a comprehensive overview of variational inference, see for example [56,57], and for Bayesian inference in general, see [58].

2.3. Probabilistic PARAFAC2

We propose two probabilistic PARAFAC2 variants using the formulation in (1), which differ only in how the orthogonality of ${\mathit{P}}_k$ is handled. The constraint ${\mathit{P}}_k^{\top }{\mathit{P}}_k={\mathit{I}}_M$ has the probabilistic interpretation that $\mathbb{E}\left[{\mathit{P}}_k^{\top }{\mathit{P}}_k\right]={\mathit{I}}_M$, in which the ${\mathit{P}}_k$ is an orthogonal matrix, which we call model (i). Another interpretation is to enforce that the expected value $\mathbb{E}\left[{\mathit{P}}_k\right]$ is an orthogonal matrix and implies $\mathbb{E}{\left[{\mathit{P}}_k\right]}^{\top }\mathbb{E}\left[{\mathit{P}}_k\right]={\mathit{I}}_M$—which we call model (ii). The main motivation for the latter approach being the interpretation of the orthogonal factor is identical to that of the maximum likelihood estimation. However, the resulting components are no longer themselves restricted to the set of orthogonal matrices, namely, the Stiefel manifold. As such, the model (ii) becomes more flexible as only the mean parameters of the variational approximation are constrained to be orthogonal and not the expectation of their inner product, as required for every realization of the underlying distribution to conform to the PARAFAC2 model. We include the latter model formulation, as it provides simple closed-form updates similar to the conventional direct fitting PARAFAC2 algorithm, as shown below. The updates for (ii) are derived by constraining the mean of a matrix normal ($\mathcal{MN}$) distribution within the variational approximation to the Stiefel manifold, whereas the model (i) formulation is based on [59] and uses a matrix von Mises–Fisher (vMF) Matrix distribution, which only has support on the Stiefel manifold. We accordingly present the following two generative models, (i) and (ii), for the probabilistic PARAFAC2:

$$\begin{array}{cccc}\hfill & \hfill & \hfill {\mathit{a}}_{i·}& \sim \mathcal{N}(\mathbf{0},{\mathbf{I}}_M)\hfill \\ \hfill & \hfill & \hfill {\mathit{f}}_{m·}& \sim \mathcal{N}(\mathbf{0},{\mathbf{I}}_M)\hfill \\ \hfill & \hfill & \hfill {\mathit{c}}_{k·}& \sim \mathcal{N}(\mathbf{0},\mathrm{diag}\left({\mathit{\alpha }}^{-1}\right))\hfill \\ \hfill \left(\mathrm{i}\right)& \hfill & \hfill {\mathit{P}}_k& \sim \mathrm{vMF}\left(\mathbf{0}\right)\hfill \\ \hfill \left(\mathrm{ii}\right)& \hfill & \hfill {\mathit{P}}_k& \sim \mathcal{MN}\left(\mathbf{0},{\mathbf{I}}_J,{\mathbf{I}}_M\right)\mathrm{s}.\mathrm{t}.\mathbb{E}{\left[{\mathit{P}}_k\right]}^{\top }\mathbb{E}\left[{\mathit{P}}_k\right]={\mathit{I}}_M\hfill \\ \hfill & \hfill & \hfill {\tau }_k& \sim \mathrm{Gamma}(a_{{\tau }_k},b_{{\tau }_k})\hfill \\ \hfill & \hfill & \hfill {\mathit{X}}_k& \sim \mathcal{N}(\mathit{A}{\mathit{D}}_k{\mathit{F}}^{\top }{\mathit{P}}_k^{\top },{\tau }_k^{-1}{\mathbf{I}}_J),\hfill \end{array}$$

where ${\mathit{a}}_{i·}$ denotes the ith row of the matrix $\mathit{A}$ and similarly for ${\mathit{f}}_{m·}$ and ${\mathit{c}}_{k·}$. We denote the set of all ${\left\{{\mathit{P}}_k\right\}}_{k=1,2,\dots ,K}$ as $\mathcal{P}$. For the rate-scale Gamma distribution, the hyper-parameters $a_{{\tau }_k}$ and $b_{{\tau }_k}$ are user defined. $\mathit{\alpha }$ defines the length scale of each component and can thus be used for automatic relevance determination (ARD) by turning off excess components by concentrating their distributions at zero when ${\alpha }_m$ is large [56]. In this paper, we use the MAP estimate of ${\alpha }_m$ as we are more interested in the model’s pruning ability than uncertainty on ${\alpha }_m$. Pruning excess components is a challenging task, see [45] for ARD within Bayesian inference in the CP and Tucker models, and [60] for Bayesian shrinkage priors in general. Lastly, we allow the noise ${\tau }_k$ to vary across slabs, thereby accounting for potential different levels of the noise (i.e., assuming heteroscedastic noise) across slabs.

2.4. Variational Update Rules

The inference is based on the following factorized distribution,

$$\mathrm{q}\left(\mathit{\theta }\right)=q\left(\mathit{A}\right)q\left(\mathit{C}\right)\prod _{m}q\left({\mathit{f}}_{m·}\right)\prod _{k}q\left({\mathit{P}}_k\right)q\left({\tau }_k\right)$$

leading to the following ELBO,

$$\begin{array}{cc}\hfill \mathrm{ELBO}\left(\mathrm{q}\right(\mathit{\theta }\left)\right)=& \mathbb{E}[logp(\mathcal{X},\mathit{\theta }\left)\right]-\mathbb{E}[log\mathrm{q}(\mathit{\theta }\left)\right]\hfill \\ \hfill =& \mathbb{E}[log\mathrm{p}(\mathcal{X}\mid \mathit{A},\mathit{C},\mathit{F},\mathcal{P},\mathit{\tau }\left)\right]+\mathbb{E}[log\mathrm{p}(\mathit{A}\left)\right]\hfill \\ \hfill & +\mathbb{E}[log\mathrm{p}(\mathit{C}\mid \mathit{\alpha }\left)\right]+\mathbb{E}[log\mathrm{p}(\mathit{F}\left)\right]\hfill \\ \hfill & +\mathbb{E}[log\mathrm{p}(\mathcal{P}\left)\right]+\mathbb{E}[log\mathrm{p}(\mathit{\tau }\left)\right]\hfill \\ \hfill & +\mathrm{h}\left(\mathrm{q}\right(\mathit{A}\left)\right)+\mathrm{h}\left(\mathrm{q}\right(\mathit{C}\left)\right)+\mathrm{h}\left(\mathrm{q}\right(\mathit{F}\left)\right)\hfill \\ \hfill & +\mathrm{h}\left(\mathrm{q}\right(\mathcal{P}\left)\right)+\mathrm{h}\left(\mathrm{q}\right(\mathit{\tau }\left)\right).\hfill \end{array}$$

(7)

Expanding the variational factors, as given by (6), the resulting variational distributions and update rules are given in

Table 1. Overview of all the variational factors and their updates. Note that $\mathcal{P}={\left\{{\mathit{P}}_k\right\}}_{k=1,2,\dots ,K}$ is the set of projection matrices and (SVD) indicates the expression is decomposed by singular value decomposition (SVD) to obtain ${\mathit{U}}_k{\mathit{S}}_k{\mathit{V}}_k$.

Variational Factor	Update
$\mathrm{q}\left(\mathit{A}\right)\sim {\displaystyle \prod _i}\mathcal{N}({\mathit{\mu }}_{{\mathit{a}}_{i·}},{\mathsf{\Sigma }}_{{\mathit{a}}_{i·}})$	${\mathsf{\Sigma }}_{{\mathit{a}}_{i·}}={({\mathbf{I}}_M+{\displaystyle \sum _k}\mathbb{E}\left[{\tau }_k\right]\mathbb{E}\left[{\mathit{D}}_k{\mathit{F}}^{\top }{\mathit{P}}_k^{\top }{\mathit{P}}_k\mathit{F}{\mathit{D}}_k\right])}^{-1}$
	${\mathit{\mu }}_{{\mathit{a}}_{i·}}={\mathsf{\Sigma }}_{{\mathit{a}}_{i·}}{\displaystyle \sum _k}\mathbb{E}\left[{\tau }_k\right]\mathbb{E}\left[{\mathit{D}}_k{\mathit{F}}^{\top }{\mathit{P}}_k^{\top }{\mathit{x}}_{i·k}^{\top }\right]$
$\mathrm{q}\left(\mathit{C}\right)\sim \prod \mathcal{N}({\mathit{\mu }}_{{\mathit{c}}_{k·}},{\mathsf{\Sigma }}_{{\mathit{c}}_{k·}})$	${\mathsf{\Sigma }}_{{\mathit{c}}_{k·}}={\left(\mathrm{diag}\left(\mathit{\alpha }\right)+\mathbb{E}\left[{\tau }_k\right]\mathbb{E}\left[{\mathit{F}}^{\top }{\mathit{P}}_k^{\top }{\mathit{P}}_k\mathit{F}\right]\circ \mathbb{E}\left[{\mathit{A}}^{\top }\mathit{A}\right]\right)}^{-1}$
	${\mathit{\mu }}_{{\mathit{c}}_{k·}}={\mathsf{\Sigma }}_{{\mathit{c}}_{\mathit{k}·}}\mathbb{E}\left[{\tau }_k\right]\mathrm{diag}\left(\mathbb{E}\left[{\mathit{F}}^{\top }\right]\mathbb{E}\left[{\mathit{P}}_k^{\top }\right]{\mathit{X}}_k^{\top }\mathbb{E}\left[\mathit{A}\right]\right)$
$\mathrm{q}\left(\mathit{F}\right)\sim {\displaystyle \prod _m}\mathcal{N}({\mathit{\mu }}_{{\mathit{f}}_{m·}},{\mathsf{\Sigma }}_{{\mathit{f}}_{m·}})$	${\mathsf{\Sigma }}_{{\mathit{f}}_{\mathit{m}·}}={(\sum _k\mathbb{E}\left[{\tau }_k\right]\mathbb{E}\left[{\mathit{D}}_k{\mathit{A}}^{\top }\mathit{A}{\mathit{D}}_k\right]\mathbb{E}\left[{\mathit{p}}_{·mk}^{\top }{\mathit{p}}_{·mk}\right]+{\mathbf{I}}_M)}^{-1}$
	${\mathit{\mu }}_{{\mathit{f}}_{\mathit{m}·}}={\mathsf{\Sigma }}_{{\mathit{f}}_{\mathit{m}·}}({\displaystyle \sum _k}\mathbb{E}\left[{\tau }_k\right]\{\mathbb{E}\left[{\left({\mathit{P}}_k^{\top }\right)}_m\right]{\mathit{X}}_k^{\top }\mathbb{E}\left[\mathit{A}\right]\mathbb{E}\left[{\mathit{D}}_k\right]$
	$-\mathbb{E}\left[{\mathit{D}}_k{\mathit{A}}^{\top }\mathit{A}{\mathit{D}}_k\right]{\displaystyle \sum _{m^{\prime }\setminus m}}\mathbb{E}\left[{\mathit{p}}_{·mk}^{\top }{\mathit{p}}_{·m^{\prime }k}\right]{\mathit{f}}_{m^{\prime }·}^{\top }\})$
$\mathrm{q}\left(\mathcal{P}\right)\sim {\displaystyle \prod _k}\mathrm{vMF}\left({\mathit{B}}_{{\mathit{P}}_k}\right)$	${\mathit{B}}_{{\mathit{P}}_k}=\mathbb{E}\left[{\tau }_k\right]\mathbb{E}\left[\mathit{F}\right]\mathbb{E}\left[{\mathit{D}}_k\right]\mathbb{E}\left[{\mathit{A}}^{\top }\right]{\mathit{X}}_k$
	$\begin{array}{c}\hfill \mathbb{E}\left[{\mathit{P}}_k\right]={\mathit{V}}_k\mathsf{\Psi }{\mathit{U}}_k^{\top },\mathrm{where}{\mathit{B}}_{{\mathit{P}}_k}={\mathit{U}}_k{\mathit{S}}_k{\mathit{V}}_k^{\top }\left(\mathrm{SVD}\right)\end{array}$
	($\mathsf{\Psi }$ given by [59], Appendix A.2)
$\mathrm{q}\left(\mathcal{P}\right)\sim {\displaystyle \prod _k}c\mathcal{MN}\left({\mathit{M}}_{{\mathit{P}}_k},{\mathbf{I}}_J,{\mathsf{\Sigma }}_{{\mathit{P}}_k}\right)$	${\mathsf{\Sigma }}_{{\mathit{P}}_k}={(\mathbb{E}\left[\mathit{F}{\mathit{D}}_k{\mathit{A}}^{\top }\mathit{A}{\mathit{D}}_k{\mathit{F}}^{\top }\right]+\mathbf{I})}^{-1}$
	$\begin{array}{c}\hfill {\mathit{M}}_{{\mathit{P}}_k}={\mathit{V}}_k{\mathit{U}}_k^{\top },\mathrm{where}\mathbb{E}\left[{\tau }_k\right]\mathbb{E}\left[\mathit{F}\right]\mathbb{E}\left[{\mathit{D}}_k\right]\mathbb{E}\left[{\mathit{A}}^{\top }\right]{\mathit{X}}_k={\mathit{U}}_k{\mathit{S}}_k{\mathit{V}}_k^{\top }\left(\mathrm{SVD}\right)\end{array}$
$\mathrm{q}\left(\mathit{\tau }\right)\sim {\displaystyle \prod _k}\mathrm{Gamma}(a_{{\tau }_k},b_{{\tau }_k})$	$a_{{\tau }_k}=a_{\mathit{\tau }}+{\displaystyle \frac{I·J}{2}}$
	$b_{{\tau }_k}=(b_{\mathit{\tau }}^{-1}+{\displaystyle \frac{1}{2}}\mathrm{Tr}\left({\mathit{X}}_k{\mathit{X}}_k^{\top }\right)+{\displaystyle \frac{1}{2}}\mathbb{E}\left[\mathrm{Tr}\left(\mathit{A}{\mathit{D}}_k{\mathit{F}}^{\top }{\mathit{P}}_k^{\top }{\mathit{P}}_k\mathit{F}{\mathit{D}}_k{\mathit{A}}^{\top }\right)\right]$
	$-\mathbb{E}\left[\mathrm{Tr}\left(\mathit{A}{\mathit{D}}_k{\mathit{F}}^{\top }{\mathit{P}}_k^{\top }{\mathit{X}}_k^{\top }\right)\right]{)}^{-1}$
$\underset{{\alpha }_m}{\arg \max }\mathrm{ELBO}\left({\alpha }_m\right)$	${\alpha }_m=K{({\displaystyle \sum _k}\mathbb{E}\left[c_{km}^2\right])}^{-1}$

. The update for the factor matrix $\mathit{F}$ is non-trivial, and to obtain a closed-form solution we employ a componentwise updating scheme inspired by the non-negative matrix factorization literature [61,62,63]. For each latent parameter, we use (6) and moment matching to determine the optimal variational distributions.

These updates rules are used for implementing a computational algorithm for probabilistic PARAFAC2, where each factor $\mathit{A},\mathit{C},\mathit{F},\mathcal{P},$ and $\mathit{\tau }$ is updated conditionally on all other factors. This leads to an alternating optimization algorithm that, given an initial solution (randomized or starting from the MAP solution), iteratively maximizes the evidence lower bound, Equation (7), until the relative change in ELBO is below a convergence criteria or a maximum number of iterations is reached. Finding the optimal solution is a non-convex optimization problem that is sensitive to initialization and the order of the updates.

2.4.1. Von Mises–Fisher Loading

In the von Mises–Fisher model for the loading ${\mathit{P}}_k$, the variational distribution is given by

$$\mathrm{vMF}\left({\mathit{P}}_k\right|{\mathit{B}}_{{\mathit{P}}_k})=\kappa {(J,{\mathit{B}}_{{\mathit{P}}_k}^{\top }{\mathit{B}}_{{\mathit{P}}_k})}^{-1}\exp \left(\mathrm{tr}\left[{\mathit{B}}_{{\mathit{P}}_k}^{\top }{\mathit{P}}_k\right]\right),$$

which is defined on the Stiefel manifold, ${\mathit{P}}_k^{\top }{\mathit{P}}_k=\mathit{I}$. The normalization constant is given by $\kappa ={}_{0}F_1\left({\displaystyle \frac{1}{2}}J,{\displaystyle \frac{1}{4}}{\mathit{B}}_{{\mathit{P}}_k}^{\top }{\mathit{B}}_{{\mathit{P}}_k}\right){\mathrm{v}}_{J,M}$, where ${\mathrm{v}}_{J,M}$ is the volume of the J-dimensional Stiefel manifold described by M components [64]. The hypergeometric function with matrix argument ${}_{0}F_1\left(·,·\right)$ can be calculated more efficiently using the SVD of ${\mathit{B}}_{{\mathit{P}}_k}={\mathit{U}}_k{\mathit{S}}_k{\mathit{V}}_k^{\top }$, since ${}_{0}F_1\left({\displaystyle \frac{1}{2}}J,{\displaystyle \frac{1}{4}}{\mathit{B}}_{{\mathit{P}}_k}^{\top }{\mathit{B}}_{{\mathit{P}}_k}\right)={}_{0}F_1\left({\displaystyle \frac{1}{2}}J,{\displaystyle \frac{1}{4}}{\mathit{S}}_k^2\right)$ [64].

Computing expectations over the vMF matrix distribution requires evaluating the hypergeometric function and can be performed as described by [59]. (^† Source code for approximating the hypergeometric function is available online http://staff.utia.cz/smidl/files/mat/OVPCA.zip (accessed on 28 February 2017). This code was used with default settings and without modifications in the experiments. We also share it with the probabilistic PARAFAC2 code at https://github.com/philipjhj/VBParafac2 (accessed on 28 February 2017)). Note that it follows from the vMF matrix distribution that $\mathbb{E}\left[{\mathit{P}}_k^{\top }{\mathit{P}}_k\right]=\mathbf{I}$, but in general $\mathbb{E}{\left[{\mathit{P}}_k\right]}^{\top }\mathbb{E}\left[{\mathit{P}}_k\right]\ne \mathbf{I}$. However, if an orthogonal summary representation is desired, one can inspect the mode of the vMF given by ${\mathbf{U}}_k{\mathbf{V}}_k^{\top }$.

2.4.2. Constrained Matrix Normal Loading

In the constrained matrix normal ($c\mathcal{MN}$) model for the variational factor of the loadings ${\mathit{P}}_k$, we consider the distribution

$$\begin{array}{cc}\hfill c\mathcal{MN}\left({\mathit{P}}_k\right|{\mathit{M}}_{{\mathit{P}}_k},{\mathit{I}}_J,{\mathsf{\Sigma }}_{{\mathit{P}}_k})=& {\displaystyle \frac{\exp \left\{-\frac{1}{2}\mathrm{trace}\left({\mathsf{\Sigma }}_{{\mathit{P}}_k}^{-1}{({\mathit{P}}_k-{\mathit{M}}_{{\mathit{P}}_k})}^{\top }{\mathit{I}}_J^{-1}({\mathit{P}}_k-{\mathit{M}}_{{\mathit{P}}_k})\right)\right\}}{{\left(2\pi \right)}^{IM/2}|{\mathsf{\Sigma }}_{{\mathit{P}}_{\mathit{k}}}{|}^{I/2}{\left|{\mathit{I}}_J\right|}^{M/2}}},\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.{\mathit{M}}_{{\mathit{P}}_k}^{\top }{\mathit{M}}_{{\mathit{P}}_k}=\mathit{I}.\hfill \end{array}$$

Instead of using the free form variational approach, we maximize (7) as a function of the mean parameter ${\mathit{M}}_{{\mathit{P}}_k}$ subject to the orthogonality constraint ${\mathit{M}}_{{\mathit{P}}_k}{\mathit{M}}_{{\mathit{P}}_k}^{\top }={\mathbf{I}}_M.$

The constraint consequently causes (7) to be constant except for the linear term of the expected log of the probability density function of the data. The reason for this is that all other terms do not depend on ${\mathit{M}}_{{\mathit{P}}_k}$ or only on the matrix product ${\mathit{M}}_{{\mathit{P}}_k}{\mathit{M}}_{{\mathit{P}}_k}^{\top }$, which is equivalent to the identity matrix, resulting in the optimization problem

$$\begin{array}{c}\hfill \underset{{\mathit{M}}_{{\mathit{P}}_k}}{\arg \max }\mathrm{ELBO}\left({\mathit{M}}_{{\mathit{P}}_k}\right)\mathrm{s}.\mathrm{t}.{\mathit{M}}_{{\mathit{P}}_k}{{\mathit{M}}_{{\mathit{P}}_k}}^{\top }=\mathbf{I}\end{array}$$

where

$$\begin{array}{c}\hfill \mathrm{ELBO}\left({\mathit{M}}_{{\mathit{P}}_k}\right)=\sum _k\mathbb{E}\left[{\tau }_k\right]\mathrm{Tr}\left(\mathbb{E}\left[\mathit{F}\right]\mathbb{E}\left[{\mathit{D}}_k\right]\mathbb{E}\left[{\mathit{A}}^{\top }\right]{\mathit{X}}_k{\mathit{M}}_{{\mathit{P}}_k}\right)+c.\end{array}$$

This is equal to (2) except for a scalar leading to the same solution as for the maximum likelihood estimation method, as given in (3). Detailed derivations of the expression above are given in the Appendix A and Appendix B. The variance parameter ${\mathsf{\Sigma }}_{{\mathit{P}}_k}$ in the variational distribution follows from moment matching using (6).

2.4.3. The F Matrix

The updates for ${\mathit{f}}_{m·}$ are non-trivial due to an inter-component dependency. The quadratic term in (6) for $\mathit{F}$ is

$$\begin{array}{l}{\mathbb{E}}_{-\mathit{F}}\left[{\mathit{a}}_{i·}{\mathit{D}}_k{\mathit{F}}^{\top }{{\mathit{P}}_k}^{\top }{\mathit{P}}_k\mathit{F}{\mathit{D}}_k{{\mathit{a}}_{i·}}^{\top }\right]\\ ={\mathbb{E}}_{-\mathit{F}}\left[\mathrm{Tr}\left(\mathit{F}{\mathit{D}}_k{{\mathit{a}}_{i·}}^{\top }{\mathit{a}}_{i·}{\mathit{D}}_k{\mathit{F}}^{\top }{{\mathit{P}}_k}^{\top }{\mathit{P}}_k\right)\right]\\ =\mathrm{Tr}\left(\mathit{F}{\mathbb{E}}_{-\mathit{F}}\left[{\mathit{D}}_k{{\mathit{a}}_{i·}}^{\top }{\mathit{a}}_{i·}{\mathit{D}}_k\right]{\mathit{F}}^{\top }{\mathbb{E}}_{-\mathit{F}}\left[{{\mathit{P}}_k}^{\top }{\mathit{P}}_k\right]\right)\\ ={\displaystyle \sum _{m{m}^{\prime }}}{\left(\mathit{F}\mathbb{E}\left[{\mathit{D}}_k{{\mathit{a}}_{i·}}^{\top }{\mathit{a}}_{i·}{\mathit{D}}_k\right]{\mathit{F}}^{\top }\right)}_{m{m}^{\prime }}{\left(\mathbb{E}\left[{{\mathit{P}}_k}^{\top }{\mathit{P}}_k\right]\right)}_{m{m}^{\prime }}\\ ={\displaystyle \sum _{m{m}^{\prime }}}{\mathit{f}}_{m·}\mathbb{E}\left[{\mathit{D}}_k{{\mathit{a}}_{i·}}^{\top }{\mathit{a}}_{i·}{\mathit{D}}_k\right]{\mathbf{f}}_{m^{\prime }·}^{\mathrm{T}}\mathbb{E}\left[{\mathbf{p}}_{·mk}^{\mathrm{T}}{\mathbf{p}}_{·m^{\prime }k}\right]\\ ={\displaystyle \sum _m}{\mathit{f}}_{m·}\mathbb{E}\left[{\mathit{D}}_k{{\mathit{a}}_{i·}}^{\top }{\mathit{a}}_{i·}{\mathit{D}}_k\right]\mathbb{E}\left[{\mathbf{p}}_{·mk}^{\mathrm{T}}{\mathbf{p}}_{·mk}\right]{\mathbf{f}}_{m·}^{\mathrm{T}}\\ +2{\displaystyle \sum _m}{\displaystyle \sum _{m^{\prime }\setminus m}}{\mathit{f}}_{m·}\mathbb{E}\left[{\mathit{D}}_k{{\mathit{a}}_{i·}}^{\top }{\mathit{a}}_{i·}{\mathit{D}}_k\right]\mathbb{E}\left[{\mathbf{p}}_{·mk}^{\mathrm{T}}{\mathbf{p}}_{·m^{\prime }k}\right]{\mathbf{f}}_{m^{\prime }·}^{\mathrm{T}},\end{array}$$

where we see that the quadratic term separates into a quadratic and linear part, revealing the linear inter-component dependency.

2.4.4. Non-Trivial Expectations

An overview of all the factors and their updates are given in Table 1. Below, we detail some non-trivial expectations and the necessary steps to compute them. The first group of expectations deals with having the diagonal matrix ${\mathit{D}}_k$ left and right multiplied with an inner term. The first case is the following expectation,

$$\begin{array}{c}\hfill \mathbb{E}\left[{\mathit{D}}_k{{\mathit{a}}_{i·}}^{\top }{\mathit{a}}_{i·}{\mathit{D}}_k\right]\end{array}$$

which is equivalent to the Hadamard product of the outer product of the diagonal of the surrounding matrix with itself and the inner matrix; so, we can separate the expectation into two parts

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\mathit{D}}_k{{\mathit{a}}_{i·}}^{\top }{\mathit{a}}_{i·}{\mathit{D}}_k\right]=& \mathbb{E}\left[{{\mathit{c}}_{k·}}^{\top }{\mathit{c}}_{k·}\right]\circ \mathbb{E}\left[{{\mathit{a}}_{i·}}^{\top }{\mathit{a}}_{i·}\right],\hfill \end{array}$$

where ${\mathit{c}}_{k·}$ is the vector containing the diagonal elements of ${\mathit{D}}_k$. The same rule applies for the following expectation:

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\mathit{D}}_k{\mathit{F}}^{\top }{{\mathit{P}}_k}^{\top }{\mathit{P}}_k\mathit{F}{\mathit{D}}_k\right]=& \mathbb{E}\left[{{\mathit{c}}_{k·}}^{\top }{\mathit{c}}_{k·}\right]\circ \mathbb{E}\left[{\mathit{F}}^{\top }{{\mathit{P}}_k}^{\top }{\mathit{P}}_k\mathit{F}\right],\hfill \end{array}$$

where the second expectation becomes trivial when using the vMF prior (ii) as the matrix product ${{\mathit{P}}_k}^{\top }{\mathit{P}}_k$ is the identity matrix. However, when using the matrix normal distribution (i), we obtain

$$\begin{array}{cc}\hfill \mathbb{E}\left[{{\mathit{P}}_k}^{\top }{\mathit{P}}_k\right]=& \mathrm{Tr}\left({\mathsf{\Sigma }}_{{\mathit{P}}_k}\right)+{\mathbf{I}}_M,\hfill \end{array}$$

which leads to the element with index $ij$ of the expectation to be equal to

$$\begin{array}{cc}\hfill \mathbb{E}{\left[{\mathit{F}}^{\top }{{\mathit{P}}_k}^{\top }{\mathit{P}}_k\mathit{F}\right]}_{ij}& =\mathbb{E}[\sum _m{\left({\mathit{F}}^{\top }\right)}_{im}{\left({{\mathit{P}}_k}^{\top }{\mathit{P}}_k\mathit{F}\right)}_{mj}]\hfill \\ \hfill & =\mathbb{E}[\sum _m{\mathit{F}}_{mi}^{\top }\sum _{m^{\prime }}{\left({{\mathit{P}}_k}^{\top }{\mathit{P}}_k\right)}_{m{m}^{\prime }}{\mathit{F}}_{m^{\prime }j}]\hfill \\ \hfill & =\sum _m\sum _{m^{\prime }}\mathbb{E}\left[{\mathit{F}}_{mi}^{\top }{\mathit{F}}_{m^{\prime }j}\right]\mathbb{E}\left[{\left({{\mathit{P}}_k}^{\top }{\mathit{P}}_k\right)}_{m{m}^{\prime }}\right].\hfill \end{array}$$

Since the m’th and $m^{\prime }$ components are independent, we have

$$\begin{array}{cc}\hfill & \mathbb{E}\left[{\mathit{F}}_{mi}^{\top }{\mathit{F}}_{m^{\prime }j}\right]=\hfill \\ \hfill & \left\{\begin{array}{cc}\mathbb{E}\left[{\mathit{F}}_{mi}^{\top }\right]\mathbb{E}\left[{\mathit{F}}_{m^{\prime }j}\right]+{\left({\mathsf{\Sigma }}_{{\mathit{f}}_{\mathit{m}·}}\right)}_{ij}\hfill & \mathrm{for}m=m^{\prime }.\hfill \\ \mathbb{E}\left[{\mathit{F}}_{mi}^{\top }\right]\mathbb{E}\left[{\mathit{F}}_{m^{\prime }j}\right]\hfill & \mathrm{for}m\ne m^{\prime }.\hfill \end{array}\right.\hfill \end{array}$$

These are the most involved expectations when computing the update rules, and the remaining are either simpler or depend upon the expectations derived here.

2.5. Noise Modeling

The probabilistic formulation of PARAFAC2 requires the specification and estimation of the noise precision $\mathit{\tau }$. We presently consider two specifications, i.e., homoscedastic noise in which the noise of each slab ${\mathit{X}}_k$ is identical—i.e., ${\tau }_1=\dots ={\tau }_K$—as assumed in the direct fitting algorithm, and heteroscedastic noise, where the model includes a separate precision for each of the K slabs.

2.6. Model Selection

A benefit of a fully probabilistic formulation of the PARAFAC2 model is that it provides model order quantification using tools from Bayesian inference, see [45,60], respectively, for details in the context of probabilistic tensor models and Bayesian inference in general. We presently exploit automatic relevance determination by learning the length scale $\mathit{\alpha }$, see also [56]. In practice, we use the MAP estimates for the automatic relevance determination because we are more interested in the pruning ability than the uncertainty estimates on $\mathit{\alpha }$. If desired, a variational estimate is easily found by letting ${\alpha }_m$ follow a Gamma distribution, c.f. [49]. Finally, the estimated ELBO on the data can also be used to compare different model orders.

2.7. Computational Complexity

The computational complexity of probabilistic PARAFAC2—for a third-order tensor $\mathcal{X}\in {\mathbb{R}}^{I\times J\times K}$ with M components—is the same as its maximum likelihood alternative, namely, $O(I·J·K·M+K·M^3)$ where the first term stems from the matricized tensor Khatri–Rao product (MTTKRP) and the second from the inversion (or SVD) of an $M\times M$ matrix in connection with updating ${\mathit{P}}_k$ for $k=1,2,\dots ,K$. The MTTKRP cannot be avoided, but caching of the sufficient statistics can make resulting calculations more efficient, although the computational complexity remains unchanged. Usually, $I,J,$ and K are much greater than M; so, the MTTKRP becomes the limiting factor. Importantly, a limitation of the variational Bayesian formulation of PARAFAC2 is that one cannot directly use the projection trick of PARAFAC2, where the K mode is projected such that ${\mathit{X}}_k{\mathit{P}}_k={\mathit{Y}}_{\mathit{k}}$, and $\mathcal{Y}$ then has a PARAFAC structure. The trick relies on ${\mathit{P}}_k$ being orthogonal, but in the variational formulation, the expectation of $\mathbb{E}\left[{\mathit{P}}_k\right]$ is used instead of ${\mathit{P}}_k$; thus, it is no longer exactly orthogonal. A remedy to this is either using sampling or a maximum a priori estimate for which ${\mathit{P}}_k$ is exactly orthogonal, although neither approach changes the computational complexity of probabilistic PARAFAC2.

3. Results and Discussion

We evaluate the proposed models on both synthetic data and three real data sets: an amino acid fluorescence (AAF) data set and two gas chromatography–mass spectrometry (GC-MS) data sets. For comparison, we include the least squares PARAFAC2 direct-fit (Direct Fit) [15], probabilistic CP with normal distributed factors and a Gamma ARD-prior with either homoscedastic (VB PARAFAC $\Delta$) or heteroscedastic (VB Parafac $\mathsf{\Omega }$) noise modeling, probabilistic Tucker (VB TUCKER) [48], and Bayesian relaxed matrix factorization (rMFT) [65]. For the proposed probabilistic PARAFAC2 methods, we initialize the model parameters as the PARAFAC2 solution computed using the direct fitting algorithm (as implemented by Bro [15] at http://www.models.life.ku.dk/go?filename=parafac2.m (accessed on 13 October 2017)) and repeat the initialization five times for the synthetic data and 50 times for the real data to minimize the risk of getting stuck in a local extrema. The final model parameters are chosen as the parameters with the lowest R2 for the direct fitting models and the highest ELBO for the probabilistic models among the fitted models. Each model estimation is limited to ${10}^4$ iterations for the synthetic data and $5\times {10}^4$ iterations for the real data. If the relative improvement in R2 for the direct fitting models and the ELBO for the probabilistic models after an iteration goes below ${10}^{-9}$, we invoke an early stop. Empirically, we experienced better learning of the probabilistic models by keeping the precision parameter of the added noise fixed for some number of iterations while estimating the length scale $\mathit{\alpha }$. We choose this delay to last for the first 50 iterations. The hyper-parameters of the precision were set to (shape, scale) = $(a_{{\tau }_k},b_{{\tau }_k})=(1,{10}^{32})$ in order to be uninformative for the variational distribution, as their influence on the updated parameters is very small on the considered data sets.

3.1. Synthetic Data

To investigate the performance of the proposed model, we generate synthetic data sets in a similar manner as in [15]. We generated the data tensor $\mathcal{X}$ by sampling $\mathit{A}$ from a zero-mean isotropic multivariate normal distribution with unit variance. $\mathit{F}$ was taken from a Cholesky factorization of an $M\times M$ matrix with 1’s in its diagonal and $0.4$ in all the off-diagonal elements. This essentially keeps the M components from being too similar. Each element of $\mathit{C}$ was sampled from a uniform distribution on the interval 0 to 30. ${\mathit{P}}_k$ was constructed by the standard orthonormalization function in MATLAB of a set of vectors sampled from a zero-mean isotropic multivariate normal distribution with unit variance. The synthetic data sets were generated with either homoscedastic or heteroscedastic additive noise at different signal-to-noise ratios (SNR) in the interval $[-20,10]$ dB, with increments of 2 dB. Each configuration was generated 10 times, resulting in 320 data sets. Each data set was given the dimensions $50\times 50\times 10$ with $M=4$ components.

The probabilistic PARAFAC2 models were fitted to the data sets with the results on the synthetic data shown in Figure 1 and Figure 2. To investigate the effect of the principled model selection approach based on the ELBO, we compare it to the existing model order selection heuristics by plotting the different selection criteria as a function of the number of components used in the model in Figure 1a,b. The figures show the mean result of the models fitted on the 10 synthetic data sets with four components and an SNR of 4. Overall, the ELBO suggests the same number of components as the other two criteria, R2 and CCD. When the data have heteroscedastic noise, the two probabilistic models that incorporate this have a substantially higher ELBO compared to the homoscedastic models.

The results for varying SNR using the true number of components in each model are shown in Figure 2a for data with homoscedastic noise and in Figure 2b for data with heteroscedastic noise. We report the R2 on the noiseless data, i.e., using the formula from (5), with the modification that the noise ${\mathit{E}}_k$ has been subtracted from ${\mathit{X}}_k$ for each slab. Thereby, we measure the different models’ ability to capture the true underlying structure in the data.

On the homoscedastic data, we see a small advantage of using the two vMF models compared to the direct fitting algorithm when we decrease the SNR of the data. The $c\mathcal{MN}$ model performs slightly worse compared to the direct fitting algorithm. When we move to the heteroscedastic data, we see a stronger separation of the four different probabilistic methods. Naturally, the models with heteroscedastic noise outperform the ones with homoscedastic noise. It is also evident that the penalty of modeling the noise as heteroscedastic in a setting where the true noise is homoscedastic is small.

If the number of components is misspecified, see Figure 2c,d, we see a larger difference between the performance of the probabilistic models accounting for the heteroscedastic noise and the direct fitting algorithm. Here, we also observe that the vMF models perform better compared to the $c\mathcal{MN}$ parameterization and see a larger positive effect of using the probabilistic models over the direct fitting algorithm. This is mainly explained by the reduced tendency to overfit when accounting for the uncertainty and the automatic relevance determination (ARD) pruning irrelevant components, as the Bayesian modeling promotes simpler representations by the ARD.

3.2. Real Data

As our synthetic results suggest, both formulations of the orthogonality constraint appear to be reasonable; we further investigate their performance on three real-world data sets. The first is an amino acid fluorescence (AAF) data set (available at www.ucphchemometrics.com (accessed on 28 February 2017), previously http://www.models.life.ku.dk/Amino_Acid_fluo) described in [61,66], in which the core-consistency diagnostic based on the PARAFAC2 model has previously successfully identified the three underlying constituents; tyrosine, tryptophan, and phenylalanine [54]. The data set contains five samples with 201 emission and 61 excitation intervals.

In addition, the models were evaluated on two gas chromatography–mass spectrometry (GC-MS) data sets. The first of these originated from wine (GC-MS-WINE) (available at www.ucphchemometrics.com (accessed on 28 February 2017), previously http://www.models.life.ku.dk/Wine_GCMS_FTIR) and was described in detail in [67]. PARAFAC2 has previously been used on GC-MS data obtained from measuring wine [54,68]. The second data set based on tobacco (GC-MS-TOBAC) was produced by [69] and kindly made available by the authors upon request. The GC-MS-WINE data contain 44 samples of wine; here, we specifically consider the unaligned data at the elution times 4.5903–4.7527 min over the mass range m/z 5–204. The GC-MS-TOBAC data analyzed here contain 65 samples of tobacco, and we consider the elution times between 4.95 and 5.03 min over the mass range m/z 50–350.

In Figure 3, Figure 4, Figure 5 and Figure 6, we consider the estimated components using the direct fitting algorithm and the proposed probabilistic PARAFAC2 with homo- and heteroscedastic noise, respectively. In Figure 3, we report the ELBO using the probabilistic models as well as the R2 and CCD using the direct fitting algorithm, and in Figure 4, Figure 5 and Figure 6, we present the extracted profiles for each data set.

For the amino acid fluorescence data, we observe that both the R2 and CCD strongly suggest that a three-component model sufficiently describes the data, and the ELBO also finds no substantial improvements beyond three components (Figure 3a). In Figure 4, we investigate the extracted excitation loadings and observe that both the probabilistic and direct fitting PARAFAC2 models extract similar components when too few or the correct number of components are specified, i.e., $M\le 3$. However, facing misspecification by having chosen too many components, the direct fitting algorithm extracts noisy profiles that incorrectly reflect the underlying three constituents. In contrast, the probabilistic PARAFAC2 models more robustly recover the three constituents when overspecifying the number of components—in particular, when assuming homoscedastic noise.

For the GC-MS-WINE data, the R2 and CCD point to a four- or five-component model, whereas the ELBO points to adding additional components (cf. Figure 3b). Inspecting the extracted components in Figure 5, we again observe close agreement between the extracted components using the probabilistic and direct fitting PARAFAC2 approaches when specifying a low number of components ($M\le 5$). Furthermore, the estimated elution profiles facing model order misspecification appear less influenced by noise than the elution profiles extracted using the direct fitting algorithm, emphasizing the improved robustness by the Bayesian approach.

For the GC-MS-TOBAC data given in Figure 3c, we observe support for a three-component model according to R2 and CCD, whereas it is harder to decide a suitable model order based on the ELBO. The change in the ELBO from two to three components for the homoscedastic noise models suggests that local maxima have been identified. Inspecting the extracted components in Figure 6, it is also evident that local maxima have been reached for most of the probabilistic PARAFAC2 models with $M<4$. For $M>3$, most of the probabilistic models successfully recover the three components without using the extra components, where the direct fitting algorithm splits the three components into multiple components.

Of the three considered data sets, the ELBO itself does not strongly indicate an optimal number of components; however, most of the probabilistic models still manage to recover the underlying structure given by the ground-truth or expert conclusion in spite of being overspecified. This is in sharp contrast to MAP estimation, where overspecification typically leads to degenerate solutions. We attribute this to the regularization invoked by accounting for uncertainty and the automatic relevance determination promoting the pruning of excess components. The relative importance of each component can be observed from the Hinton diagrams in Figure 4, Figure 5 and Figure 6. Each square in the Hinton diagrams indicates the relative contribution of each component to the full data reconstruction, computed as the squared Frobenius norm of the componentwise data reconstruction divided by the sum of the squared Frobenius norms of all the componentwise data reconstructions.

4. Conclusions

We developed a fully probabilistic PARAFAC2 model and demonstrated how orthogonality can be imposed in the context of variational inference in two different ways: Firstly, using the von Mises–Fisher matrix distribution, assuming $\mathbb{E}\left[{\mathit{Y}}^{\top }\mathit{Y}\right]=\mathit{I}$, as proposed in the context of variational PCA in [59]. Using this distribution forces all the realizations of the given matrix parameter to be orthogonal. Secondly, using the constrained matrix normal distribution, assuming $\mathbb{E}\left[{\mathit{Y}}^{\top }\right]\mathbb{E}\left[\mathit{Y}\right]=\mathit{I}$, in which the mean is constrained to the Stiefel manifold. This effectively results in a more flexible model as only the expectation of the realizations of the matrix are orthogonal and not the realizations themselves. For the latter approach, we presently derived a simple closed-form solution based on optimizing the lower bound.

Both probabilistic PARAFAC2 approaches were able to successfully recover the underlying signal in synthetic data when considering homoscedastic or heteroscedastic added noise. However, we found that the specification of orthogonality based on vMF was more robust to noise than the specification based on $c\mathcal{MN}$. In particular, we observed substantial noise robustness in the probabilistic PARAFAC2 models when compared to the conventional direct fitting approach, both when the correct model order was specified and when overestimating the number of components.

On the AAF data, the probabilistic PARAFAC2 framework was able to correctly identify the underlying constituents and demonstrated improved robustness to model misspecification when compared to the conventional direct fitting algorithm. The ELBOs of the probabilistic models suggest a model order of three components similar to the CCD and R2 heuristics computed from the direct fitting estimations. For the two gas chromatography–mass spectrometry data sets, GC-MS-WINE and GC-MS-TOBAC, we also observed agreement between the probabilistic and direct fitting PARAFAC2 models but with more mixed results. The model order is not so clearly evident from the ELBO on these data sets. However, we see that the automatic relevance determination suppresses unnecessary components fairly well on both data sets, ensuring robustness to overspecification of the model, which otherwise leads to degenerate solutions when the direct fitting approach is used. A few results from the probabilistic PARAFAC2 did not match the results of the direct fitting approach. This can most likely be explained by encountering local maxima, since variational methods are known to suffer from issues of underestimating uncertainty and thereby becoming overly confident on estimated parameters.

We attribute the performance improvements of probabilistic PARAFAC2 over conventional PARAFAC2 to the casting of PARAFAC2 as a Bayesian model, which approximates the posterior distribution of the parameters—rather than a point estimate as conventional PARAFAC2. Additionally, Bayesian inference, in general, enjoys more robustness to noise and overspecification of the model [48]. The proposed probabilistic PARAFAC2 models form an important step in the direction of applying probabilistic approaches to more advanced tensor decomposition approaches and a new direction for handling orthogonality constraints in probabilistic modeling—in general, using the proposed constrained matrix normal distribution framework, which has a simple variational update. In particular, we anticipate that the orthogonality constraints within a probabilistic setting may also be useful for the Tucker decomposition, in which orthogonality is typically imposed [5]; the block-term decompositions [70], in which orthogonality may be beneficial to impose within each block as previously considered using the vMF [48]; or to improve identifiability within the CP decomposition by imposing orthogonality as implemented in the n-way toolbox (http://www.models.life.ku.dk/nwaytoolbox, accessed on 28 February 2017). PARAFAC2 is actively being advanced and employed for new applications, e.g., recently, the higher-order block term decomposition has been embedded with a PARAFAC2 structure [71].