Generalized Alpha-Beta Divergences and Their Application to Robust Nonnegative Matrix Factorization

Abstract

We propose a class of multiplicative algorithms for Nonnegative Matrix Factorization (NMF) which are robust with respect to noise and outliers. To achieve this, we formulate a new family generalized divergences referred to as the Alpha-Beta-divergences (AB-divergences), which are parameterized by the two tuning parameters, alpha and beta, and smoothly connect the fundamental Alpha-, Beta- and Gamma-divergences. By adjusting these tuning parameters, we show that a wide range of standard and new divergences can be obtained. The corresponding learning algorithms for NMF are shown to integrate and generalize many existing ones, including the Lee-Seung, ISRA (Image Space Reconstruction Algorithm), EMML (Expectation Maximization Maximum Likelihood), Alpha-NMF, and Beta-NMF. Owing to more degrees of freedom in tuning the parameters, the proposed family of AB-multiplicative NMF algorithms is shown to improve robustness with respect to noise and outliers. The analysis illuminates the links of between AB-divergence and other divergences, especially Gamma- and Itakura-Saito divergences.

1. Introduction

In many applications such as image analysis, pattern recognition and statistical machine learning, it is beneficial to use the information-theoretic divergences rather than the Euclidean squared distance. Among them the Kullback-Leibler, Hellinger, Jensen-Shannon and Alpha-divergences have been pivotal in estimating similarity between probability distributions [1,2,3,4,5,6]. Such divergences have been successfully applied as cost functions to derive multiplicative and additive projected gradient algorithms for nonnegative matrix and tensor factorizations [7,8,9,10]. Such measures also play an important role in the areas of neural computation, pattern recognition, estimation, inference and optimization. For instance, many machine learning algorithms for classification and clustering employ a variety of (dis)similarity measures which are formulated using information theory, convex optimization, and information geometry [11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27]. Apart from the most popular squared Euclidean distance and Kullback-Leibler divergence, recently, alternative generalized divergences such as the Csiszár-Morimoto f-divergence and Bregman divergences have become attractive alternatives for advanced machine learning algorithms [7,8,9,10,28,29,30,31,32,33,34].

In this paper, we present a novel (dis)similarity measure which smoothly connects or integrates many existing divergences and allows us to develop new flexible and robust algorithms for NMF and related problems. The scope of the results presented in this paper is vast, since the generalized Alpha-Beta-divergence (or simply AB-divergence) function includes a large number of useful cost functions containing those based on the relative entropies, generalized Kullback-Leibler or I-divergence, Hellinger distance, Jensen-Shannon divergence, J-divergence, Pearson and Neyman Chi-square divergences, Triangular Discrimination and Arithmetic-Geometric divergence. Furthermore, the AB-divergence provides a natural extension of the families of Alpha- and Beta-divergences, gives smooth connections between them and links to other fundamental divergences. The divergence functions discussed in this paper are flexible because they allow us to generate a large number of well-known and often used particular divergences (for specific values of tuning parameters). Moreover, by adjusting adaptive tuning parameters of factor matrices, we can optimize the cost function for NMF algorithms and estimate desired parameters of the underlying factorization model in the presence of noise and/or outliers.

1.1. Introduction to NMF and Basic Multiplicative Algorithms for NMF

The Nonnegative Matrix Factorization (NMF) problem has been investigated by many researchers, e.g., Paatero and Tapper [36], and has gained popularity through the work of Lee and Seung [37,38]. Based on the argument that the nonnegativity is important in human perception, they proposed simple algorithms (often called the Lee-Seung algorithms) for finding physically meaningful nonnegative representations of nonnegative signals and images [38].

The basic NMF problem can be stated as follows: Given a nonnegative data matrix $\mathbf{Y}=\mathbf{P}=[p_{it}]\in {\mathbb{R}}_{+}^{I\times T}$ (with $p_{it}\ge 0$ or equivalently $\mathbf{P}\ge 0$) and a reduced rank J ($J\le min(I,T)$, typically $J<<min(I,T)$), find two nonnegative matrices $\mathbf{A}=[{\mathit{a}}_1,{\mathit{a}}_2,\dots ,{\mathit{a}}_J]\in {\mathbb{R}}_{+}^{I\times J}$ and $\mathbf{X}={\mathbf{B}}^T={[{\mathit{b}}_1,{\mathit{b}}_2,\dots ,{\mathit{b}}_J]}^T\in {\mathbb{R}}_{+}^{J\times T}$ which factorize $\mathbf{P}$ as faithfully as possible, that is

$$\begin{array}{c}\hfill \mathbf{Y}=\mathbf{A}\mathbf{X}+\mathbf{E}=\mathbf{A}{\mathbf{B}}^T+\mathbf{E},\end{array}$$

(1)

where the matrix $\mathbf{E}\in {\mathbb{R}}^{I\times T}$ represents approximation error. Since we usually operate on column vectors of matrices for convenience we shall use often the matrix $\mathbf{B}={\mathbf{X}}^T$ instead of the matrix $\mathbf{X}$.

The factors $\mathbf{A}$ and $\mathbf{X}$ may have different physical meanings in different applications: in a Blind Source Separation (BSS) $\mathbf{A}$ denotes mixing matrix and $\mathbf{X}$ source signals; in clustering problems, $\mathbf{A}$ is the basis matrix and $\mathbf{X}$ is a weight matrix; in acoustic analysis, $\mathbf{A}$ represents the basis patterns, while each row of $\mathbf{X}$ corresponds to sound patterns activation [7].

Standard NMF only assumes nonnegativity of factor matrices $\mathbf{A}$ and $\mathbf{X}$. Unlike blind source separation methods based on independent component analysis (ICA), here we do not assume that the sources are independent, although we can impose some additional constraints such as smoothness, sparseness or orthogonality of $\mathbf{A}$ and/or $\mathbf{X}$ [7].

Although the NMF can be applied to the BSS problems for nonnegative sources and nonnegative mixing matrices, its application is not limited to the BSS and it can be used in various and diverse applications far beyond BSS [7].

In NMF, our aim is to find the entries of nonnegative matrices $\mathbf{A}$ and $\mathbf{X}$ assuming that a data matrix $\mathbf{P}$ is known:

$$\begin{array}{c}\hfill \mathbf{Y}=\mathbf{A}\mathbf{X}+\mathbf{E}=\mathbf{A}{\mathbf{B}}^T+\mathbf{E}=\sum _{j=1}^J{\mathit{a}}_j{\mathit{b}}_j^T+\mathbf{E},\end{array}$$

(2)

or equivalently in a scalar form:

$$\begin{array}{c}\hfill {\displaystyle y_{it}=p_{it}=\sum _{j=1}^J{a}_{ij}{x}_{jt}+e_{it}=\sum _{j=1}^J{a}_{ij}{b}_{tj}+e_{it}.}\end{array}$$

(3)

In order to estimate nonnegative factor matrices $\mathbf{A}$ and $\mathbf{X}$ in the standard NMF, we need to consider the similarity measures to quantify a difference between the data matrix $\mathbf{P}$ and the approximative NMF model matrix $\widehat{\mathbf{P}}=\mathbf{Q}=\mathbf{A}\mathbf{X}$.

The choice of the similarity measure (also referred to as distance, divergence or measure of dissimilarity) mostly depends on the probability distribution of the estimated signals or components and on the structure of data and a distribution of a noise.

The best known and the most frequently used adaptive multiplicative algorithms for NMF are based on the two loss functions: Squared Euclidean distance and generalized Kullback-Leibler divergence also called the I-divergence.

The squared Euclidean distance is based on the Frobenius norm:

$$\begin{array}{c}\hfill D_E(\mathbf{Y}||\mathbf{A}\mathbf{X})=\frac{1}{2}\left|\right|\mathbf{Y}-{\mathbf{A}\mathbf{X}\left|\right|}_F^2,\end{array}$$

(4)

which is optimal for additive Gaussian noise [7]. It should be noted that the above cost function is convex with respect to either elements of matrix $\mathbf{A}$ or matrix $\mathbf{X}$, but not both.

Remark: 

Although the NMF optimization problem is not convex, the objective functions are separately convex in each of the two factors $\mathbf{A}$ and $\mathbf{X}$, which implies that finding the optimal factor matrix $\mathbf{A}$ corresponding to a fixed matrix $\mathbf{X}$ reduces to a convex optimization problem and vice versa. However, the convexity is lost as soon as we try to optimize factor matrices simultaneously [39].

Using a gradient descent approach for cost function (4) and switching alternatively between the two sets of parameters, we obtain the simple multiplicative update formulas (see Section 3 for derivation of algorithms in a more general form):

$$\begin{array}{ccc}\hfill a_{ij}& \leftarrow & a_{ij}\frac{{[\mathbf{P}{\mathbf{X}}^T]}_{ij}}{{[\mathbf{Q}{\mathbf{X}}^T]}_{ij}+\epsilon },\hfill \end{array}$$

(5)

$$\begin{array}{ccc}\hfill x_{jt}& \leftarrow & x_{jt}\frac{{[{\mathbf{A}}^T\mathbf{P}]}_{jt}}{{[{\mathbf{A}}^T\mathbf{Q}]}_{jt}+\epsilon },\hfill \end{array}$$

(6)

where a small positive constant ε prevents division by zero, $\mathbf{P}=\mathbf{Y}$ and $\mathbf{Q}=\mathbf{A}\mathbf{X}$.

The above algorithm (5)-(6), called often the Lee-Seung NMF algorithm can be considered as a natural extension of the well known algorithm ISRA proposed first by Daube-Witherspoon and Muehllehner [40] and investigated extensively by De Pierro and Byrne [41,42,43,44,45].

The above update rules can be written in a compact matrix form as

$$\begin{array}{ccc}\hfill \mathbf{A}& \leftarrow & \mathbf{A}⊛\left[(\mathbf{P}{\mathbf{X}}^T)\oslash (\mathbf{Q}{\mathbf{X}}^T+\epsilon )\right],\hfill \end{array}$$

(7)

$$\begin{array}{ccc}\hfill \mathbf{X}& \leftarrow & \mathbf{X}⊛\left[({\mathbf{A}}^T\mathbf{P})\oslash ({\mathbf{A}}^T\mathbf{Q}+\epsilon )\right],\hfill \end{array}$$

(8)

where ⊛ is the Hadamard (components-wise) product and ⊘ is element-wise division between two matrices. In practice, the columns of the matrix $\mathbf{A}$ should be normalized to the unit ${\ell }_p$-norm (typically, $p=1$).

The original ISRA algorithm is relatively slow, and several heuristic approaches have been proposed to speed it up. For example, a relaxation approach rises the multiplicative coefficients to some power $w\in (0,2]$, that is [7,47],

$$\begin{array}{ccc}\hfill a_{ij}& \leftarrow & a_{ij}{\left(\frac{{[\mathbf{P}{\mathbf{X}}^T]}_{ij}}{{[\mathbf{Q}{\mathbf{X}}^T]}_{ij}}\right)}^w,\hfill \end{array}$$

(9)

$$\begin{array}{ccc}\hfill x_{jt}& \leftarrow & x_{jt}{\left(\frac{{[{\mathbf{A}}^T\mathbf{P}]}_{jt}}{{[{\mathbf{A}}^T\mathbf{Q}]}_{jt}}\right)}^w,\hfill \end{array}$$

(10)

in order to achieve faster convergence.

Another frequently used cost function for the NMF is the generalized Kullback-Leibler divergence (also called the I-divergence) [38]:

$$\begin{array}{c}\hfill D_{KL}(\mathbf{P}||\mathbf{Q})=\sum _{it}\left(p_{it}ln\frac{p_{it}}{q_{it}}-p_{it}+q_{it}\right),\end{array}$$

(11)

where $\mathbf{Q}=\widehat{\mathbf{P}}=\mathbf{A}\mathbf{X}$ with entries $q_{it}={[\mathbf{A}\mathbf{X}]}_{it}$.

Similar to the squared Euclidean cost function, the I-divergence is convex with respect to either $\mathbf{A}$ or $\mathbf{X}$, but it is not generally convex with respect to $\mathbf{A}$ and $\mathbf{X}$ jointly, so the minimization of such a cost function can yield many local minima.

By minimizing the cost function (11) subject to the nonnegativity constraints, we can easily derive the following multiplicative learning rule, referred to as the EMML algorithm (Expectation Maximization Maximum Likelihood) [44,45]. The EMML algorithm is sometimes called the Richardson-Lucy algorithm (RLA) or simply the EM algorithm [48,49,50]. In fact, the EMML algorithm was developed for a fixed and known $\mathbf{A}$. The Lee-Seung algorithm (which is in fact the EMML algorithm) based on the I-divergence employs alternative switching between $\mathbf{A}$ and $\mathbf{X}$ [37,38]:

$$\begin{array}{ccc}\hfill x_{jt}& \leftarrow & x_{jt}\frac{{\sum }_{i=1}^I{a}_{ij}(p_{it}/q_{it})}{{\sum }_{i=1}^I{a}_{ij}},\hfill \\ \hfill \end{array}$$

(12)

$$\begin{array}{ccc}\hfill a_{ij}& \leftarrow & a_{ij}\frac{{\sum }_{t=1}^T{x}_{jt}(p_{it}/q_{it})}{{\sum }_{t=1}^T{x}_{jt}}.\hfill \end{array}$$

(13)

We shall derive the above algorithm in a much more general and flexible form in Section 3.

To accelerate the convergence of the EMML, we can apply the following heuristic extension of the EMML algorithm [8,51]:

$$\begin{array}{c}\hfill x_{jt}\leftarrow x_{jt}{\left(\frac{{\sum }_{i=1}^I{a}_{ij}(p_{it}/q_{it})}{{\sum }_{i=1}^I{a}_{ij}}\right)}^w,\end{array}$$

(14)

$$\begin{array}{c}\hfill a_{ij}\leftarrow a_{ij}{\left(\frac{{\sum }_{t=1}^T{x}_{jt}(p_{it}/q_{it})}{{\sum }_{t=1}^T{x}_{jt}}\right)}^w,\end{array}$$

(15)

where the positive relaxation parameter w helps to improve the convergence (see Section 3.3).

In an enhanced form to reduce the computational cost, the denominators in (14) and (15) can be ignored due to normalizing ${\mathit{a}}_j$ to the unit length of ${\ell }_1$-norm:

$$\begin{array}{ccc}\hfill \mathbf{X}& \leftarrow & \mathbf{X}⊛{\left({\mathbf{A}}^T(\mathbf{P}\oslash \mathbf{Q})\right)}^{.[w]},\hfill \end{array}$$

(16)

$$\begin{array}{ccc}\hfill \mathbf{A}& \leftarrow & \mathbf{A}⊛{\left((\mathbf{P}\oslash \mathbf{Q}){\mathbf{X}}^T\right)}^{.[w]},\hfill \end{array}$$

(17)

$$\begin{array}{ccc}\hfill a_{ij}& \leftarrow & a_{ij}/\parallel {\mathit{a}}_j{\parallel }_1.\hfill \end{array}$$

(18)

One of the objectives of this paper is develop generalized multiplicative NMF algorithms which are robust with respect to noise and/or outliers and integrate (combine) both the ISRA and the EMML algorithms into the more general a flexible one.

2. The Alpha-Beta Divergences

For positive measures $\mathbf{P}$ and $\mathbf{Q}$ consider the following new dissimilarity measure, which we shall refer to as the AB-divergence:

$$\begin{array}{ccc}\hfill D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})& =& {\displaystyle -\frac{1}{\alpha \beta }\sum _{it}\left(p_{it}^{\alpha }{q}_{it}^{\beta }-\frac{\alpha }{\alpha +\beta }{p}_{it}^{\alpha +\beta }-\frac{\beta }{\alpha +\beta }{q}_{it}^{\alpha +\beta }\right)}\hfill \\ & & \text{for}\alpha ,\beta ,\alpha +\beta \ne 0,\hfill \end{array}$$

(19)

or equivalently

$$\begin{array}{ccc}\hfill D_{AB}^{(\alpha ,\lambda -\alpha )}(\mathbf{P}\parallel \mathbf{Q})& =& {\displaystyle \frac{1}{(\alpha -\lambda )\alpha }\sum _{it}\left(p_{it}^{\alpha }{q}_{it}^{\lambda -\alpha }-\frac{\alpha }{\lambda }{p}_{it}^{\lambda }-\frac{\lambda -\alpha }{\lambda }{q}_{it}^{\lambda }\right),}\hfill \\ & & \text{for}\alpha \ne 0,\alpha \ne \lambda ,\lambda =\alpha +\beta \ne 0,\hfill \end{array}$$

(20)

Note that, Equation (19) is a divergence since the following relationship holds:

$$\begin{array}{ccc}\hfill \frac{1}{\alpha \beta }{p}_{it}^{\alpha }{q}_{it}^{\beta }& \le & \frac{1}{\beta (\alpha +\beta )}{p}_{it}^{\alpha +\beta }+\frac{1}{\alpha (\alpha +\beta )}{q}_{it}^{\alpha +\beta },\hfill \\ & & \text{for}\alpha ,\beta ,\alpha +\beta \ne 0,\hfill \end{array}$$

(21)

with equality holding for $p_{it}=q_{it}$. In Appendix A, we show that Equation (21) is a summarization of three different inequalities of Young’s type, each one holding true for a different combination of the signs of the constants: $\alpha \beta$, $\alpha (\alpha +\beta )$ and $\beta (\alpha +\beta )$.

In order to avoid indeterminacy or singularity for certain values of parameters, the AB-divergence can be extended by continuity (by applying l’Hôpital formula) to cover all the values of $\alpha ,\beta \in \mathbb{R}$, thus the AB-divergence can be expressed or defined in a more explicit form:

$$\begin{array}{ccc}\hfill D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})& =& \sum _{it}{d}_{AB}^{(\alpha ,\beta )}(p_{it},q_{it}),\hfill \end{array}$$

(22)

where

$$\begin{array}{lll}{d}_{AB}^{(\alpha ,\beta )}(p_{it},q_{it})& =& \left\{\begin{array}{ll}{\displaystyle ll-\frac{1}{\alpha \beta }\left(p_{it}^{\alpha }{q}_{it}^{\beta }-\frac{\alpha }{(\alpha +\beta )}{p}_{it}^{\alpha +\beta }-\frac{\beta }{(\alpha +\beta )}{q}_{it}^{\alpha +\beta }\right)}& \text{for}\alpha ,\beta ,\alpha +\beta \ne 0\\ {\displaystyle \frac{1}{{\alpha }^2}\left({\displaystyle p_{it}^{\alpha }ln\frac{p_{it}^{\alpha }}{q_{it}^{\alpha }}-p_{it}^{\alpha }+q_{it}^{\alpha }}\right)}& \text{for}\alpha \ne 0,\beta =0\\ {\displaystyle \frac{1}{{\alpha }^2}\left({\displaystyle ln\frac{q_{it}^{\alpha }}{p_{it}^{\alpha }}+{\left({\displaystyle \frac{q_{it}^{\alpha }}{p_{it}^{\alpha }}}\right)}^{-1}-1}\right)}& \text{for}\alpha =-\beta \ne 0\\ {\displaystyle \frac{1}{{\beta }^2}\left({\displaystyle q_{it}^{\beta }ln\frac{q_{it}^{\beta }}{p_{it}^{\beta }}-q_{it}^{\beta }+p_{it}^{\beta }}\right)}& \text{for}\alpha =0,\beta \ne 0\\ {\displaystyle \frac{1}{2}{\left(ln{p}_{it}-ln{q}_{it}\right)}^2}& \text{for}\alpha ,\beta =0.\end{array}\right.\end{array}$$

(23)

2.1. Special Cases of the AB-Divergence

We shall now illustrate that a suitable choice of the $(\alpha ,\beta )$ parameters simplifies the AB-divergence into some existing divergences, including the well-known Alpha- and Beta-divergences [4,7,19,35].

When $\alpha +\beta =1$ the AB-divergence reduces to the Alpha-divergence [4,33,34,35,52]

$$\begin{array}{ccc}\hfill D_{AB}^{(\alpha ,1-\alpha )}\left(\mathbf{P}\parallel \mathbf{Q}\right)& =& D_A^{(\alpha )}\left(\mathbf{P}\parallel \mathbf{Q}\right)\hfill \end{array}$$

(24)

$$\begin{array}{ccc}& \doteq & \left\{\begin{array}{ll}{\displaystyle \frac{1}{\alpha (\alpha -1)}\sum _{it}\left(p_{it}^{\alpha }{q}_{it}^{1-\alpha }-\alpha p_{it}+(\alpha -1)q_{it}\right)}\hfill & \text{for}\alpha \ne 0,\alpha \ne 1,\\ {\displaystyle \sum _{it}\left({\displaystyle p_{it}ln\frac{p_{it}}{q_{it}}-p_{it}+q_{it}}\right)}\hfill & \text{for}\alpha =1,\\ {\displaystyle \sum _{it}\left({\displaystyle q_{it}ln\frac{q_{it}}{p_{it}}-q_{it}+p_{it}}\right)}\hfill & \text{for}\alpha =0.\end{array}\right.\hfill \end{array}$$

(25)

On the other hand, when $\alpha =1$, it reduces to the Beta-divergence [8,9,19,32,53]

$$\begin{array}{ccc}\hfill D_{AB}^{(1,\beta )}\left(\mathbf{P}\parallel \mathbf{Q}\right)& =& D_B^{(\beta )}(\mathbf{P}\parallel \mathbf{Q})\hfill \end{array}$$

(26)

$$\begin{array}{ccc}& \doteq & \left\{\begin{array}{ll}{\displaystyle -\frac{1}{\beta }\sum _{it}\left({\displaystyle p_{it}{q}_{it}^{\beta }-\frac{1}{(1+\beta )}{p}_{it}^{1+\beta }-\frac{\beta }{(1+\beta )}{q}_{it}^{1+\beta }}\right)}\hfill & \text{for}\beta ,1+\beta \ne 0,\\ {\displaystyle \frac{1}{2}\sum _{it}{\left(p_{it}-q_{it}\right)}^2}\hfill & \text{for}\beta =1,\\ {\displaystyle \sum _{it}\left({\displaystyle p_{it}ln\frac{p_{it}}{q_{it}}-p_{it}+q_{it}}\right)}\hfill & \text{for}\beta =0,\\ {\displaystyle \sum _{it}\left({\displaystyle ln\frac{q_{it}}{p_{it}}+{\left({\displaystyle \frac{q_{it}}{p_{it}}}\right)}^{-1}-1}\right)}\hfill & \text{for}\beta =-1.\end{array}\right.\hfill \end{array}$$

(27)

The AB-divergence gives to the standard Kullback-Leibler (KL) divergence ($D_{KL}(·,·)$) for $\alpha =1$ and $\beta =0$

$$\begin{array}{ccc}\hfill D_{AB}^{(1,0)}(\mathbf{P}\parallel \mathbf{Q})& =& {\displaystyle D_{KL}\left(\mathbf{P}\right.∥\left.\mathbf{Q}\right)=\sum _{it}\left(p_{it}ln\frac{p_{it}}{q_{it}}-p_{it}+q_{it}\right),}\hfill \end{array}$$

(28)

and reduces to the standard Itakura-Saito divergence ($D_{IS}(·,·)$) for $\alpha =1$ and $\beta =-1$ [8,53,54]

$$\begin{array}{ccc}\hfill D_{AB}^{(1,-1)}(\mathbf{P}\parallel \mathbf{Q})& =& {\displaystyle D_{IS}\left(\mathbf{P}\right.∥\left.\mathbf{Q}\right)=\sum _{it}\left(ln\frac{q_{it}}{p_{it}}+\frac{p_{it}}{q_{it}}-1\right).}\hfill \end{array}$$

(29)

Using the $1-\alpha$ deformed logarithm defined as

$$\begin{array}{c}\hfill {ln}_{1-\alpha }(z)=\left\{\begin{array}{cc}{\displaystyle \frac{z^{\alpha }-1}{\alpha },}\hfill & \alpha \ne 0,\hfill \\ lnz,\hfill & \alpha =0,\hfill \end{array}\right.\end{array}$$

(30)

where $z>0$, observe that the AB-divergence is symmetric with respect to both arguments for $\alpha =\beta \ne 0$ and takes the form of a metric distance

$$\begin{array}{ccc}\hfill D_{AB}^{(\alpha ,\alpha )}(\mathbf{P}\parallel \mathbf{Q})& =& D_E({ln}_{1-\alpha }(\mathbf{P})\parallel {ln}_{1-\alpha }(\mathbf{Q}))=\frac{1}{2}\sum _{it}{\left({ln}_{1-\alpha }(p_{it})-{ln}_{1-\alpha }(q_{it})\right)}^2,\hfill \end{array}$$

(31)

in the transform domain $\varphi (x)={ln}_{1-\alpha }(x)$. As particular cases, this includes the scaled squared Euclidean distance (for $\alpha =1$) and the Hellinger distance (for $\alpha =0.5$).

For both $\alpha \to 0$ and $\beta \to 0$, the AB-divergence converges to the Log-Euclidean distance defined as

$$\begin{array}{c}\hfill D_{AB}^{(0,0)}(\mathbf{P}\parallel \mathbf{Q})=\underset{\alpha \to 0}{lim}{D}_{AB}^{(\alpha ,\alpha )}(\mathbf{P}\parallel \mathbf{Q})=\frac{1}{2}\sum _{it}{\left(ln{p}_{it}-ln{q}_{it}\right)}^2.\end{array}$$

(32)

2.2. Properties of AB-Divergence: Duality, Inversion and Scaling

Let us denote by ${\mathbf{P}}^{.[r]}$ the one to one transformation that raises each positive element of the matrix $\mathbf{P}$ to the power r, i.e., each entry is raised to power r, that is $p_{it}^r$. According to the definition of the AB-divergence, we can easily check that it satisfies the following duality property (see Figure 1)

$$\begin{array}{ccc}\hfill D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})& =& D_{AB}^{(\beta ,\alpha )}(\mathbf{Q}\parallel \mathbf{P}),\hfill \end{array}$$

(33)

and inversion property

$$\begin{array}{ccc}\hfill D_{AB}^{(-\alpha ,-\beta )}(\mathbf{P}\parallel \mathbf{Q})& =& D_{AB}^{(\alpha ,\beta )}({\mathbf{P}}^{.[-1])}\parallel {\mathbf{Q}}^{.[-1]}).\hfill \end{array}$$

(34)

The above properties can be considered as a particular case of the scaling property of parameters α and β by a common factor $\omega \in \mathbb{R}\backslash \{0\}$. The divergence whose parameters has been rescaled is proportional to original divergence with both arguments raised to the common factor, i.e.,

$$\begin{array}{ccc}\hfill D_{AB}^{(\omega \alpha ,\omega \beta )}(\mathbf{P}\parallel \mathbf{Q})& =& \frac{1}{{\omega }^2}{D}_{AB}^{(\alpha ,\beta )}({\mathbf{P}}^{.[\omega ]}\parallel {\mathbf{Q}}^{.[\omega ]}).\hfill \end{array}$$

(35)

The scaling of α and β by a common factor $\omega <1$ can be seen as a ‘zoom-in’ on the arguments $\mathbf{P}$ and $\mathbf{Q}$. This zoom gives more relevance to the smaller values over the large values. On the other hand, for $\omega >1$ yields a ‘zoom-out’ effect were the smaller values decrease their relevance at the expense of large values whose relevance is increased (see Figure 1).

Figure 1. Graphical illustration of duality and inversion properties of the AB-divergence. On the alpha-beta plane are indicated as special important cases particular divergences by points and lines, especially Kullback-Leibler divergence $D_{KL}$, Hellinger Distance $D_H$, Euclidean distance $D_E$, Itakura-Saito distance $D_{IS}$, Alpha-divergence $D_A^{(\alpha )}$, and Beta-divergence $D_B^{(\beta )}$.

Figure 1. Graphical illustration of duality and inversion properties of the AB-divergence. On the alpha-beta plane are indicated as special important cases particular divergences by points and lines, especially Kullback-Leibler divergence $D_{KL}$, Hellinger Distance $D_H$, Euclidean distance $D_E$, Itakura-Saito distance $D_{IS}$, Alpha-divergence $D_A^{(\alpha )}$, and Beta-divergence $D_B^{(\beta )}$.

By scaling arguments of the AB-divergence by a positive scaling factor $c>0$, it yields the following relation

$$\begin{array}{ccc}\hfill D_{AB}^{(\alpha ,\beta )}(c\mathbf{P}\parallel c\mathbf{Q})& =& c^{\alpha +\beta }{D}_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q}).\hfill \end{array}$$

(36)

These basic properties imply that whenever $\alpha \ne 0$, we can rewrite the AB-divergence in terms of a $\frac{\beta }{\alpha }$-order Beta-divergence combined with an α-zoom of its arguments as

$$\begin{array}{ccc}\hfill D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})& =& \frac{1}{{\alpha }^2}{D}_{AB}^{\left(1,\frac{\beta }{\alpha }\right)}({\mathbf{P}}^{.[\alpha ]}\parallel {\mathbf{Q}}^{.[\alpha ]})\hfill \end{array}$$

(37)

$$\begin{array}{ccc}& =& \frac{1}{{\alpha }^2}{D}_B^{\left(\frac{\beta }{\alpha }\right)}({\mathbf{P}}^{.[\alpha ]}\parallel {\mathbf{Q}}^{.[\alpha ]}).\hfill \end{array}$$

(38)

Similarly, for $\alpha +\beta \ne 0$, the AB-divergence can also be expressed in terms of $\frac{\alpha }{\alpha +\beta }$-order Alpha-divergence with an $(\alpha +\beta )$-zoom of the arguments (see Figure 2) as

$$\begin{array}{ccc}\hfill D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})& =& \frac{1}{{(\alpha +\beta )}^2}{D}_{AB}^{\left(\frac{\alpha }{\alpha +\beta },\frac{\beta }{\alpha +\beta }\right)}({\mathbf{P}}^{.[\alpha +\beta ]}\parallel {\mathbf{Q}}^{.[\alpha +\beta ]})\hfill \end{array}$$

(39)

$$\begin{array}{ccc}& =& \frac{1}{{(\alpha +\beta )}^2}{D}_A^{(\frac{\alpha }{\alpha +\beta })}\left({\mathbf{P}}^{.[\alpha +\beta ]}\right.∥\left.{\mathbf{Q}}^{.[\alpha +\beta ]}\right).\hfill \end{array}$$

(40)

illustrating that the AB-divergence equals to an $\frac{\alpha }{\alpha +\beta }$-order Alpha-divergence with an $(\alpha +\beta )$-zoom in of the arguments. On the other hand $D_{AB}^{({\alpha }_1,{\beta }_1)}$ can be seen as an Alpha-divergence of order ${\alpha }_1/({\alpha }_1+{\beta }_1)$ of a zoom-out of the arguments (with ${\alpha }_1+{\beta }_1>1$). Note that, $D_{AB}^{({\alpha }_2,{\beta }_2)}$ can be seen as the Alpha-divergence of order ${\alpha }_2/({\alpha }_2+{\beta }_2)$ with a zoom-in of its arguments with ${\alpha }_2+{\beta }_2<1$ (see Figure 2).

Figure 2. Illustrations how the AB-divergence for $\alpha +\beta \ne 0$ can be expressed via scaled Alpha-divergences.

Figure 2. Illustrations how the AB-divergence for $\alpha +\beta \ne 0$ can be expressed via scaled Alpha-divergences.

When $\alpha \ne 0$ and $\beta =0$, the AB-divergence can be expressed in terms of the Kullback-Leibler divergence with an α-zoom of its arguments

$$\begin{array}{ccc}\hfill D_{AB}^{(\alpha ,0)}(\mathbf{P}\parallel \mathbf{Q})& =& \frac{1}{{\alpha }^2}{D}_{AB}^{(1,0)}\left({\mathbf{P}}^{.[\alpha ]}\right.∥\left.{\mathbf{Q}}^{.[\alpha ]}\right)\hfill \end{array}$$

(41)

$$\begin{array}{ccc}& =& \frac{1}{{\alpha }^2}{D}_{KL}\left({\mathbf{P}}^{.[\alpha ]}\right.∥\left.{\mathbf{Q}}^{.[\alpha ]}\right)\hfill \end{array}$$

(42)

$$\begin{array}{ccc}& =& {\displaystyle \frac{1}{{\alpha }^2}\sum _{it}\left(p_{it}^{\alpha }ln\frac{p_{it}^{\alpha }}{q_{it}^{\alpha }}-p_{it}^{\alpha }+q_{it}^{\alpha }\right).}\hfill \end{array}$$

(43)

When $\alpha +\beta =0$ with $\alpha \ne 0$ and $\beta \ne 0$, the AB-divergence can also be expressed in terms of a generalized Itakura-Saito distance with an α-zoom of the arguments

$$\begin{array}{ccc}\hfill D_{AB}^{(\alpha ,-\alpha )}(\mathbf{P}\parallel \mathbf{Q})& =& \frac{1}{{\alpha }^2}{D}_{AB}^{\left(1,-1\right)}({\mathbf{P}}^{.[\alpha ]}\parallel {\mathbf{Q}}^{.[\alpha ]})\hfill \end{array}$$

(44)

$$\begin{array}{ccc}& =& \frac{1}{{\alpha }^2}{D}_{IS}\left({\mathbf{P}}^{.[\alpha ]}\right.∥\left.{\mathbf{Q}}^{.[\alpha ]}\right)\hfill \end{array}$$

(45)

$$\begin{array}{ccc}& =& {\displaystyle \frac{1}{{\alpha }^2}\sum _{it}\left(ln\frac{q_{it}^{\alpha }}{p_{it}^{\alpha }}+\frac{p_{it}^{\alpha }}{q_{it}^{\alpha }}-1\right).}\hfill \end{array}$$

(46)

Remark: 

The generalized Itakura-Saito distance is scale-invariant, i.e,

$$\begin{array}{c}\hfill D_{AB}^{(\alpha ,-\alpha )}(\mathbf{P}\parallel \mathbf{Q})=D_{AB}^{(\alpha ,-\alpha )}(c\mathbf{P}\parallel c\mathbf{Q}),\end{array}$$

(47)

with any $c>0$.

Note that, from the AB-divergence, a more general scale-invariant divergence can be obtained by the monotonic nonlinear transformations:

$$\begin{array}{c}\hfill \sum _{it}{p}_{it}^{\alpha }{q}_{it}^{\beta }\to ln\left(\sum _{it}{p}_{it}^{\alpha }{q}_{it}^{\beta }\right)\forall \alpha ,\beta \end{array}$$

(48)

leading to the following scale-invariant divergence given by

$$\begin{array}{ccc}\hfill D_{AC}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})& =& \frac{1}{\beta (\alpha +\beta )}ln\left(\sum _{it}{p}_{it}^{\alpha +\beta }\right)+\frac{1}{\alpha (\alpha +\beta )}ln\left(\sum _{it}{q}_{it}^{\alpha +\beta }\right)-\frac{1}{\alpha \beta }ln\left(\sum _{it}{p}_{it}^{\alpha }{q}_{it}^{\beta }\right)\hfill \\ & =& \frac{1}{\alpha \beta }ln\frac{{\left({\displaystyle \sum _{it}{p}_{it}^{\alpha +\beta }}\right)}^{\frac{\alpha }{\alpha +\beta }}{\left({\displaystyle \sum _{it}{q}_{it}^{\alpha +\beta }}\right)}^{\frac{\beta }{\alpha +\beta }}}{{\displaystyle \sum _{it}{p}_{it}^{\alpha }{q}_{it}^{\beta }}}\mathrm{for}\alpha \ne 0,\beta \ne 0,\alpha +\beta \ne 0.\hfill \end{array}$$

(49)

which generalizes a family of Gamma-divergences [35].

The divergence (49) is scale-invariant in a more general context

$$\begin{array}{c}\hfill D_{AC}^{(\alpha ,\beta )}(\mathbf{P}||\mathbf{Q})=D_{AC}^{(\alpha ,\beta )}(c_1\mathbf{P}\left|\right|c_2\mathbf{Q})\end{array}$$

(50)

for any positive scale factors $c_1$ and $c_2$.

2.3. Why is AB-Divergence Potentially Robust?

To illustrate the role of the hyperparameters α and β on the robustness of the AB-divergence with respect to errors and noises, we shall compare the behavior of the AB-divergence with the standard Kullback-Leibler divergence. We shall assume, without loss of generality, that the proposed factorization model $\mathbf{Q}$ (for given noisy $\mathbf{P}$) is a function of the vector of parameters $\mathit{\theta }$ and that each of its elements $q_{it}(\mathit{\theta })>0$ is non-negative for a certain range of parameters Θ.

The estimator $\widehat{\mathit{\theta }}$ obtained for the Kullback-Leibler divergence between two discrete positive measures $\mathbf{P}$ and $\mathbf{Q}$, is a solution of

$$\begin{array}{c}\hfill \frac{\partial D_{KL}\left(\mathbf{P}\parallel \mathbf{Q}\right)}{\partial \mathit{\theta }}=-\sum _{it}\frac{\partial q_{it}}{\partial \mathit{\theta }}{ln}_0\left(\frac{p_{it}}{q_{it}}\right)=\mathbf{0},\end{array}$$

(51)

while, for the Beta-divergence, the estimator solves

$$\begin{array}{c}\hfill \frac{\partial D_B^{(\beta )}\left(\mathbf{P}\parallel \mathbf{Q}\right)}{\partial \mathit{\theta }}=-\sum _{it}\frac{\partial q_{it}}{\partial \mathit{\theta }}{q}_{it}^{\beta }{ln}_0\left(\frac{p_{it}}{q_{it}}\right)=\mathbf{0}.\end{array}$$

(52)

The main difference between both equations is in the weighting factors $q_{it}^{\beta }$ for the Beta-divergence which are controlled by the parameter β. In the context of probability distributions, these weighting factors may control the influence of likelihood ratios $p_{it}/q_{it}$. It has been shown [55] and [56] that the parameter β determines a tradeoff between robustness to outliers (for $\beta >0$) and efficiency (for β near 0). In the special case of $\beta =1$ the Euclidean distance is obtained, which is known to be more robust and less efficient than the Kullback-Leibler divergence $\beta =0$.

On the other hand, for the Alpha-divergence, the estimating equation takes a different form

$$\begin{array}{c}\hfill \frac{\partial D_A^{(\alpha )}\left(\mathbf{P}\parallel \mathbf{Q}\right)}{\partial \mathit{\theta }}=-\sum _{it}\frac{\partial q_{it}}{\partial \mathit{\theta }}{ln}_{1-\alpha }\left(\frac{p_{it}}{q_{it}}\right)=\mathbf{0}.\end{array}$$

(53)

In this case, the influence of the values of individual ratios $p_{it}/q_{it}$ are controlled not by weighting factors but by the deformed logarithm of order $1-\alpha$. This feature can be interpreted as a zoom or over-weight of the interesting details of the likelihood ratio. For $\alpha >1$ (a zoom-out), we emphasize the relative importance of larger values of the ratio $p_{it}/q_{it}$, whereas for $\alpha <1$ (a zoom-in), we put more emphasis on smaller values of $p_{it}/q_{it}$ (see Figure 3). The major consequence is the inclusive ($\alpha \to \infty$) and exclusive ($\alpha \leftarrow -\infty$) behavior of the Alpha-divergence discussed in [52].

The estimating equation for the Alpha-Beta divergence combines both effects:

$$\begin{array}{c}\hfill \frac{\partial D_{AB}^{(\alpha ,\beta )}\left(\mathbf{P}\parallel \mathbf{Q}\right)}{\partial \mathit{\theta }}=-\sum _{it}\frac{\partial q_{it}}{\partial \mathit{\theta }}\underset{weights}{\underbrace{q_{it}^{\alpha +\beta -1}}}\underset{\alpha -zoom}{\underbrace{{ln}_{1-\alpha }\left(p_{it}/q_{it}\right)}}=\mathbf{0},\end{array}$$

(54)

and therefore is much more flexible and powerful regarding insensitivity to error and noise.

As illustrated in Figure 3, depending on value of α, we can zoom-in or zoom-out the interesting sets of the ratios $p_{it}/q_{it}$ and simultaneously weight these ratios by scaling factors $q_{it}^{\lambda -1}$ controlled by the parameter $\lambda =\alpha +\beta$. Therefore, the parameter α can be used to control the influence of large or small ratios in the estimator, while the parameter β provides some control on the weighting of the ratios depending on the demand to better fit to larger or smaller values of the model.

Figure 3. Graphical illustration how the set parameters $(\alpha ,\beta )$ can control influence of individual ratios $p_{it}/q_{it}$. The dash-doted line ($\alpha +\beta =1$) shows the region where the multiplicative weighting factor $q_{it}^{\alpha +\beta -1}$ in the estimating equations is constant and equal to unity. The dashed line ($\alpha =1$) shows the region where the order of the deformed logarithm of $p_{it}/q_{it}$ is constant and equal to that of the standard Kullback-Leibler divergence.

Figure 3. Graphical illustration how the set parameters $(\alpha ,\beta )$ can control influence of individual ratios $p_{it}/q_{it}$. The dash-doted line ($\alpha +\beta =1$) shows the region where the multiplicative weighting factor $q_{it}^{\alpha +\beta -1}$ in the estimating equations is constant and equal to unity. The dashed line ($\alpha =1$) shows the region where the order of the deformed logarithm of $p_{it}/q_{it}$ is constant and equal to that of the standard Kullback-Leibler divergence.

3. Generalized Multiplicative Algorithms for NMF

3.1. Derivation of Multiplicative NMF Algorithms Based on the AB-Divergence

We shall now develop generalized and flexible NMF algorithms (see Equation (1)) by employing the AB-divergence with $\mathbf{Y}=\mathbf{P}=[p_{it}]\in {\mathbb{R}}_{+}^{I\times T}$, $\mathbf{Q}=[q_{it}]=\mathbf{A}\mathbf{X}\in {\mathbb{R}}_{+}^{I\times T}$, where $q_{it}={\widehat{p}}_{it}={[\mathbf{A}\mathbf{X}]}_{it}={\sum }_j{a}_{ij}{x}_{jt}$. In this case, the gradient of the AB-divergence (20) can be expressed in a compact form (for any $\alpha ,\beta \in \mathbb{R}$) in terms of an $1-\alpha$ deformed logarithm (see (30))

$$\begin{array}{ccc}\hfill \frac{\partial D_{AB}^{(\alpha ,\beta )}}{\partial x_{jt}}& =& {\displaystyle -\sum _{i=1}^I{q}_{it}^{\lambda -1}{a}_{ij}{ln}_{1-\alpha }\left(\frac{p_{it}}{q_{it}}\right),}\hfill \end{array}$$

(55)

$$\begin{array}{ccc}\hfill \frac{\partial D_{AB}^{(\alpha ,\beta )}}{\partial a_{ij}}& =& {\displaystyle -\sum _{t=1}^T{q}_{it}^{\lambda -1}{x}_{jt}{ln}_{1-\alpha }\left(\frac{p_{it}}{q_{it}}\right).}\hfill \end{array}$$

(56)

The new multiplicative learning algorithm is gradient descent based, in the natural parameter space $\varphi (x)$ of the proposed divergence:

$$\begin{array}{ccc}\hfill x_{jt}& \leftarrow & {\varphi }^{-1}\left(\varphi (x_{jt})-{\eta }_{jt}\frac{\partial D_{AB}^{(\alpha ,\beta )}}{\partial \varphi (x_{jt})}\right),\hfill \end{array}$$

(57)

$$\begin{array}{ccc}\hfill a_{ij}& \leftarrow & {\varphi }^{-1}\left(\varphi (a_{ij})-{\eta }_{ij}\frac{\partial D_{AB}^{(\alpha ,\beta )}}{\partial \varphi (a_{ij})}\right).\hfill \end{array}$$

(58)

These updates can be considered as a generalization of the exponentiated gradient (EG) [57].

In general, such a nonlinear scaling (or transformation) provides a stable solution and the obtained gradients are much better behaved in the ϕ space. We use the $1-\alpha$ deformed logarithm transformation $\varphi (z)={ln}_{1-\alpha }(z)$, whose inverse transformation is a $1-\alpha$ deformed exponential

$$\begin{array}{c}\hfill {\varphi }^{-1}(z)={exp}_{1-\alpha }(z)=\left\{\begin{array}{ccc}exp(z)& \text{for}\hfill & \alpha =0,\hfill \\ {(1+\alpha z)}^{\frac{1}{\alpha }}& \text{for}\hfill & \alpha \ne 0\text{and}1+\alpha z\ge 0,\hfill \\ 0& \text{for}\hfill & \alpha \ne 0\text{and}1+\alpha z<0.\hfill \end{array}\right.\end{array}$$

(59)

For positive measures $z>0$, the direct transformation $\varphi (z)$ and the composition ${\varphi }^{-1}(\varphi (z))$ are bijective functions which define a one to one correspondence, so we have ${\varphi }^{-1}(\varphi (z))=z$.

By choosing suitable learning rates

$$\begin{array}{c}\hfill {\eta }_{jt}=\frac{x_{jt}^{2\alpha -1}}{{\displaystyle \sum _{i=1}^I{a}_{ij}{q}_{it}^{\lambda -1}}},{\eta }_{ij}=\frac{a_{ij}^{2\alpha -1}}{{\displaystyle \sum _{t=1}^T{x}_{jt}{q}_{it}^{\lambda -1}}}.\end{array}$$

(60)

a new multiplicative NMF algorithm (refereed to as the AB-multiplicative NMF algorithm) is obtained as:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill x_{jt}& \leftarrow & {\displaystyle x_{jt}{exp}_{1-\alpha }\left(\sum _{i=1}^I\frac{a_{ij}{q}_{it}^{\lambda -1}}{{\sum }_{i=1}^I{a}_{ij}{q}_{it}^{\lambda -1}}{ln}_{1-\alpha }\left(\frac{p_{it}}{q_{it}}\right)\right),}\hfill \\ \hfill a_{ij}& \leftarrow & {\displaystyle a_{ij}{exp}_{1-\alpha }\left(\sum _{t=1}^T\frac{x_{jt}{q}_{it}^{\lambda -1}}{{\sum }_{t=1}^t{x}_{jt}{q}_{it}^{\lambda -1}}{ln}_{1-\alpha }\left(\frac{p_{it}}{q_{it}}\right)\right).}\hfill \end{array}\end{array}$$

(61)

In these equations, the deformed logarithm of order $1-\alpha$ of the quotients $p_{it}/q_{it}$ plays a key role to control relative error terms whose weighted mean provides the multiplicative corrections. This deformation in the relative error is controlled by the parameter α. The parameter $\alpha >1$ gives more relevance to the large values of the quotient, while the case of $\alpha <1$ puts more emphasis on smaller values of the quotient. On the other hand, the parameter $\lambda -1$ (where $\lambda =\alpha +\beta$) controls the influence of the values of the approximation ($q_{it}$) on the weighing of the deformed error terms. For $\lambda =1$, this influence disappears.

It is interesting to note that the multiplicative term of the main updates

$$\begin{array}{ccc}\hfill M_{\alpha }(\mathbf{z},\mathbf{w},S)& =& {exp}_{1-\alpha }\left(\frac{1}{{\displaystyle \sum _{i\in S}{w}_i}}\sum _{i\in S}{w}_i{ln}_{1-\alpha }\left(z_i\right)\right),\hfill \end{array}$$

(62)

can be interpreted as a weighted generalized mean across the elements with indices in the set S.

Depending on the value of α, we obtain as particular cases: the minimum of the vector $\mathbf{z}$ (for $\alpha \to -\infty$), its weighted harmonic mean ($\alpha =-1$), the weighted geometric mean ($\alpha =0$), the arithmetic mean ($\alpha =1$), the weighted quadratic mean ($\alpha =2$) and the maximum of the vector ($\alpha \to \infty$), i.e.,

$$\begin{array}{ccc}\hfill M_{\alpha }(\mathbf{z},\mathbf{w},\{1,\dots ,n\})& =& \left\{\begin{array}{ll}min\{z_1,\dots ,z_n\},\hfill & \alpha \to -\infty ,\\ \left({\displaystyle \sum _{i=1}^n{w}_i}\right){\left({\displaystyle \sum _{i=1}^n\frac{w_i}{z_i}}\right)}^{-1},\hfill & \alpha =-1,\\ {\displaystyle \prod _{i=1}^n{z}^{\frac{w_i}{{\sum }_{i=1}^n{w}_i}},}\hfill & \alpha =0,\\ {\displaystyle \frac{1}{{\sum }_{i=1}^n{w}_i}\sum _{i=1}^n{w}_i{z}_i,}\hfill & \alpha =1,\\ {\left({\displaystyle \frac{1}{{\sum }_{i=1}^n{w}_i}\sum _{i=1}^n{w}_i{z}_i^2}\right)}^{1/2},\hfill & \alpha =2,\\ max\{z_1,\dots ,z_n\},\hfill & \alpha \to \infty .\end{array}\right.\hfill \end{array}$$

(63)

The generalized weighted means are monotonically increasing functions of α, i.e., if ${\alpha }_1<{\alpha }_2$, then

$$\begin{array}{c}\hfill M_{{\alpha }_1}(\mathbf{z},\mathbf{w},S)<M_{{\alpha }_2}(\mathbf{z},\mathbf{w},S).\end{array}$$

(64)

Thus, by increasing the values of α, we puts more emphasis on large relative errors in the update formulas (61).

In the special case of $\alpha \ne 0$, the above update rules can be simplified as:

$$\begin{array}{ccc}\hfill x_{jt}& \leftarrow & x_{jt}{\left(\frac{{\displaystyle \sum _{i=1}^I{a}_{ij}{p}_{it}^{\alpha }{q}_{it}^{\beta -1}}}{{\displaystyle \sum _{i=1}^I{a}_{ij}{q}_{it}^{\alpha +\beta -1}}}\right)}^{1/\alpha },\hfill \end{array}$$

(65)

$$\begin{array}{ccc}\hfill a_{ij}& \leftarrow & a_{ij}{\left(\frac{{\displaystyle \sum _{t=1}^T{x}_{jt}{p}_{it}^{\alpha }{q}_{it}^{\beta -1}}}{{\displaystyle \sum _{t=1}^T{x}_{jt}{q}_{it}^{\alpha +\beta -1}}}\right)}^{1/\alpha },\hfill \end{array}$$

(66)

where $q_{it}={[\mathbf{A}\mathbf{X}]}_{it}$ and at every iteration the columns of $\mathbf{A}$ are normalized to the unit length.

The above multiplicative update rules can be written in a compact matrix forms as

$$\begin{array}{ccc}\hfill \mathbf{X}& \leftarrow & \mathbf{X}⊛{\left(\left({\mathbf{A}}^T({\mathbf{P}}^{.[\alpha ]}⊛{\mathbf{Q}}^{.[\beta -1]})\right)\oslash \left({\mathbf{A}}^T{\mathbf{Q}}^{.[\alpha +\beta -1]}\right)\right)}^{.[1/\alpha ]},\hfill \end{array}$$

(67)

$$\begin{array}{ccc}\hfill \mathbf{A}& \leftarrow & \mathbf{A}⊛{\left(\left(\left({\mathbf{P}}^{.[\alpha ]}⊛{\mathbf{Q}}^{.[\beta -1]}\right){\mathbf{X}}^T\right)\oslash \left({\mathbf{Q}}^{.[\alpha +\beta -1]}{\mathbf{X}}^T\right)\right)}^{.[1/\alpha ]}\hfill \end{array}$$

(68)

or even more compactly:

$$\begin{array}{ccc}\hfill \mathbf{X}& \leftarrow & \mathbf{X}⊛{\left(\left({\mathbf{A}}^T\mathbf{Z}\right)\oslash \left({\mathbf{A}}^T{\mathbf{Q}}^{.[\alpha +\beta -1]}\right)\right)}^{.[1/\alpha ]},\hfill \end{array}$$

(69)

$$\begin{array}{ccc}\hfill \mathbf{A}& \leftarrow & \mathbf{A}⊛{\left(\left(\mathbf{Z}{\mathbf{X}}^T\right)\oslash \left({\mathbf{Q}}^{.[\alpha +\beta -1]}{\mathbf{X}}^T\right)\right)}^{.[1/\alpha ]},\hfill \end{array}$$

(70)

where $\mathbf{Z}={\mathbf{P}}^{.[\alpha ]}⊛{\mathbf{Q}}^{.[\beta -1]}$.

In order to fix the scaling indeterminacy between the columns of $\mathbf{A}$ and the rows of $\mathbf{X}$, in practice, after each iteration, we can usually evaluate the $l_1$-norm of the columns of $\mathbf{A}$ and normalize the elements of the matrices as

$$\begin{array}{ccc}\hfill x_{ij}\leftarrow & x_{ij}\sum _p{a}_{pj},a_{ij}\leftarrow & a_{ij}/\sum _p{a}_{pj}.\hfill \end{array}$$

(71)

This normalization does not alter $\mathbf{Q}=\mathbf{A}\mathbf{X}$, thus, preserving the value of the AB-divergence.

The above novel algorithms are natural extensions of many existing algorithms for NMF, including the ISRA, EMML, Lee-Seung algorithms and Alpha- and Beta-multiplicative NMF algorithms [7,58]. For example, by selecting $\alpha +\beta =1$, we obtain the Alpha-NMF algorithm, for $\alpha =1$, we have Beta-NMF algorithms, for $\alpha =-\beta \ne 0$ we obtain a family of multiplicative NMF algorithms based on the extended Itakura-Saito distance [8,53]. Furthermore, for $\alpha =1$ and $\beta =1$, we obtain the ISRA algorithm and for $\alpha =1$ and $\beta =0$ we obtain the EMML algorithm.

It is important to note that in low-rank approximations, we do not need access to all input data $p_{it}$. In other words, the above algorithms can be applied for low-rank approximations even if some data are missing or they are purposely omitted or ignored. For large-scale problems, the learning rules can be written in a more efficient form by restricting the generalized mean only to those elements whose indices belong to a preselected subsets $S_T\subset \{1,\cdots ,T\}$ and $S_I\subset \{1,\cdots ,I\}$ of the whole set of indices.

Using a duality property ($D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})=D_{AB}^{(\beta ,\alpha )}(\mathbf{Q}\parallel \mathbf{P})$) of the AB-divergence, we obtain now the dual update rules:

$$\begin{array}{ccc}\hfill x_{jt}& \leftarrow & x_{jt}{exp}_{1-\beta }\left(\sum _{i\in S_I}\frac{a_{ij}{p}_{it}^{\lambda -1}}{{\sum }_{i\in S_I}{a}_{ij}{p}_{it}^{\lambda -1}}{ln}_{1-\beta }\left(\frac{q_{it}}{p_{it}}\right)\right),\hfill \\ \hfill a_{ij}& \leftarrow & a_{ij}{exp}_{1-\beta }\left(\sum _{t\in S_T}\frac{x_{jt}{p}_{it}^{\lambda -1}}{{\sum }_{t\in S_T}{x}_{jt}{p}_{it}^{\lambda -1}}{ln}_{1-\beta }\left(\frac{q_{it}}{p_{it}}\right)\right),\hfill \end{array}$$

(72)

which for $\beta \ne 0$ reduce to

$$\begin{array}{ccc}\hfill a_{ij}& \leftarrow & a_{ij}{\left(\frac{{\displaystyle \sum _{t\in S_T}{x}_{jt}{p}_{it}^{\alpha -1}{q}_{it}^{\beta }}}{{\displaystyle \sum _{t\in S_T}{x}_{jt}{p}_{it}^{\alpha +\beta -1}}}\right)}^{1/\beta },\hfill \\ \hfill x_{jt}& \leftarrow & x_{jt}{\left(\frac{{\displaystyle \sum _{i\in S_I}{a}_{ij}{p}_{it}^{\alpha -1}{q}_{it}^{\beta }}}{{\displaystyle \sum _{i\in S_I}{a}_{ij}{p}_{it}^{\alpha +\beta -1}}}\right)}^{1/\beta }.\hfill \end{array}$$

(73)

3.2. Conditions for a Monotonic Descent of AB-Divergence

In this section, we first explain the basic principle of auxiliary function method, which allows us to establish conditions for which NMF update rules provide monotonic descent of the cost function during iterative process. In other words, we analyze the conditions for the existence of auxiliary functions that justify the monotonous descent in the AB-divergence under multiplicative update rules of the form (61). NMF update formulas with such property have been previously obtained for certain particular divergences: in [38,41] for the Euclidean distance and Kullback-Leibler divergence, in [58] for the Alpha-Divergence and in [59,60,61] for the Beta-divergence.

Let ${\mathbf{Q}}^{(k)}={\mathbf{A}}^{(k)}{\mathbf{X}}^{(k)}$ denote our current factorization model of the observations in the k-th iteration, and let $\{{\mathbf{Q}}^{(k+1)}\}$ denote our candidate model for the next $k+1$-iteration. In the following, we adopt the notation $\{{\mathbf{Q}}^{(k)}\}\equiv \{{\mathbf{A}}^{(k)},{\mathbf{X}}^{(k)}\}$ to refer compactly to the non-negative factors of the decompositions.

An auxiliary function $G(\{{\mathbf{Q}}^{(k+1)}\},\{{\mathbf{Q}}^{(k)}\};\mathbf{P})$ for the surrogate optimization of the divergence should satisfy

$$\begin{array}{ccc}\hfill G(\{{\mathbf{Q}}^{(k+1)}\},\{{\mathbf{Q}}^{(k)}\};\mathbf{P})\ge G(\{{\mathbf{Q}}^{(k+1)}\},\{{\mathbf{Q}}^{(k+1)}\};\mathbf{P})& =& D_{AB}^{(\alpha ,\beta )}(\mathbf{P}||{\mathbf{Q}}^{(k+1)}),\hfill \end{array}$$

(74)

with equality holding for $\{{\mathbf{Q}}^{(k)}\}=\{{\mathbf{Q}}^{(k+1)}\}$.

In analogy to the Expectation Maximization techniques developed in [38,41,58], our objective is to find a convex upper bound of the AB divergence and to replace the minimization of the original AB-divergence by an iterative optimization of the auxiliary function.

Given the factors of the current factorization model $\{{\mathbf{Q}}^{(k)}\}$, the factors of the next candidate $\{{\mathbf{Q}}^{(k+1)}\}$ are chosen as

$$\begin{array}{c}\hfill \{{\mathbf{Q}}^{(k+1)}\}=arg\underset{\{\mathbf{Q}\}}{min}G(\{\mathbf{Q}\},\{{\mathbf{Q}}^{(k)}\};\mathbf{P}).\end{array}$$

(75)

where, for simplicity, the minimization is usually carried out with respect to only one of the factors $\mathbf{A}$ or $\mathbf{X}$ of $\{\mathbf{Q}\}$, while keeping the other factor fixed and common in $\{{\mathbf{Q}}^{(k+1)}\}$, $\{{\mathbf{Q}}^{(k)}\}$ and $\{\mathbf{Q}\}$.

Assuming that auxiliary function is found, a monotonic descent of the AB-divergence is a consequence of the chain of inequalities:

$$\begin{array}{ccc}\hfill D_{AB}^{(\alpha ,\beta )}(\mathbf{P}||{\mathbf{Q}}^{(k)})& =& G(\{{\mathbf{Q}}^{(k)}\},\{{\mathbf{Q}}^{(k)}\};\mathbf{P})\hfill \end{array}$$

(76)

$$\begin{array}{ccc}& \stackrel{(a)}{\ge }& G(\{{\mathbf{Q}}^{(k+1)}\},\{{\mathbf{Q}}^{(k)}\};\mathbf{P})\hfill \end{array}$$

(77)

$$\begin{array}{ccc}& \stackrel{(b)}{\ge }& G(\{{\mathbf{Q}}^{(k+1)}\},\{{\mathbf{Q}}^{(k+1)}\};\mathbf{P})\hfill \end{array}$$

(78)

$$\begin{array}{ccc}& =& D_{AB}^{(\alpha ,\beta )}(\mathbf{P}\parallel {\mathbf{Q}}^{(k+1)}).\hfill \end{array}$$

(79)

The inequality $(a)$ is due to the optimization in (75), while the inequality $(b)$ reflects the definition of the auxiliary function in (74).

3.3. A Conditional Auxiliary Function

It is shown in Appendix B that under a certain condition, the function

$$\begin{array}{ccc}\hfill & & G(\{{\mathbf{Q}}^{(k+1)}\},\{{\mathbf{Q}}^{(k)}\};\mathbf{P})=\sum _{i,t}\sum _j{\gamma }_{it}^{(j)}(\{{\mathbf{Q}}^{(k)}\})d_{AB}^{(\alpha ,\beta )}\left(p_{it},{\widehat{q}}_{it}^{(j)}\right)\hfill \end{array}$$

(80)

$$\begin{array}{ccc}& & \text{where}{\gamma }_{it}^{(j)}(\{{\mathbf{Q}}^{(k)}\})=\frac{a_{ij}^{(k)}{x}_{jt}^{(k)}}{q_{it}^{(k)}}\text{and}{\widehat{q}}_{it}^{(j)}=\frac{a_{ij}^{(k+1)}{x}_{jt}^{(k+1)}}{{\gamma }_{it}^{(j)}(\{{\mathbf{Q}}^{(k)}\})},\hfill \end{array}$$

(81)

is an auxiliary function for the surrogate optimization of $D_{AB}^{(\alpha ,\beta )}(\mathbf{P}||\mathbf{Q})$. The required condition is, at each iteration, the convexity of $d_{AB}^{(\alpha ,\beta )}(p_{it},{\widehat{q}}_{it})$ with respect to all ${\widehat{q}}_{it}\in [{min}_j{\widehat{q}}_{it}^{(j)},{max}_j{\widehat{q}}_{it}^{(j)}]$.

Recall that ${\mathbf{Q}}^{(k)}={\mathbf{A}}^{(k)}{\mathbf{X}}^{(k)}$ corresponds to the current iteration, while a candidate model ${\mathbf{Q}}^{(k+1)}={\mathbf{A}}^{(k+1)}{\mathbf{X}}^{(k+1)}$ for the next iteration, is the solution of the optimization problem (75). For updating $\mathbf{A}$ at each iteration, we keep ${\mathbf{X}}^{(k+1)}$ equal to ${\mathbf{X}}^{(k)}$ and find the global minimum of the auxiliary function $G(\{{\mathbf{A}}^{(k+1)},{\mathbf{X}}^{(k)}\},\{{\mathbf{A}}^{(k)},{\mathbf{X}}^{(k)}\};\mathbf{P})$ with respect to ${\mathbf{A}}^{(k+1)}$.

By setting the following derivative to zero

$$\begin{array}{ccc}\hfill \frac{\partial G(\mathbf{A}\mathbf{X},{\mathbf{Q}}^{(k)};\mathbf{P})}{\partial a_{ij}}& =& a_{ij}^{\beta -1}\sum _{t=1}^T{(q_{it}^{(k)})}^{\lambda -1}{x}_{jt}^{(k)}\left[{ln}_{1-\alpha }\left(\frac{a_{ij}}{a_{ij}^{(k)}}\right)-{ln}_{1-\alpha }\left(\frac{p_{it}}{q_{it}^{(k)}}\right)\right]=0,\hfill \end{array}$$

(82)

and solving with respect to $a_{ij}=a_{ij}^{(k+1)}$ yields

$$\begin{array}{ccc}\hfill a_{ij}^{(k+1)}& =& a_{ij}^{(k)}{exp}_{1-\alpha }\left(\sum _{t=1}^T\frac{{(q_{it}^{(k)})}^{\lambda -1}{x}_{jt}^{(k)}}{{\sum }_{t=1}^T{(q_{it}^{(k)})}^{\lambda -1}{x}_{jt}^{(k)}}{ln}_{1-\alpha }\left(\frac{p_{it}}{q_{it}^{(k)}}\right)\right).\hfill \end{array}$$

(83)

Analogously, for the intermediate factorized model ${\tilde{\mathbf{Q}}}^{(k)}={\mathbf{A}}^{(k+1)}{\mathbf{X}}^{(k)}$, the global minimum of the auxiliary function $G({\mathbf{A}}^{(k+1)}{\mathbf{X}}^{(k+1)},\{{\mathbf{A}}^{(k+1)},{\mathbf{X}}^{(k)}\};\mathbf{P})$ with respect to the new update ${\mathbf{X}}^{(k+1)}$, while keeping $\mathbf{A}={\mathbf{A}}^{(k+1)}$, is given by

$$\begin{array}{ccc}\hfill x_{jt}^{(k+1)}& =& x_{jt}^{(k)}{exp}_{1-\alpha }\left(\sum _{i=1}^I\frac{{({\tilde{q}}_{it}^{(k)})}^{\lambda -1}{a}_{ij}^{(k+1)}}{{\sum }_{i=1}^I{({\tilde{q}}_{it}^{(k)})}^{\lambda -1}{a}_{ij}^{(k+1)}}{ln}_{1-\alpha }\left(\frac{p_{it}}{{\tilde{q}}_{it}^{(k)}}\right)\right).\hfill \end{array}$$

(84)

The above two equations match with the updates proposed in (61). As previously indicated, a sufficient condition for a monotonic descent of the AB-divergence is the convexity of terms $d_{AB}^{(\alpha ,\beta )}(p_{it},{\widehat{q}}_{it})$ for all ${\widehat{q}}_{it}\in [{min}_j{\widehat{q}}_{it}^{(j)},{max}_j{\widehat{q}}_{it}^{(j)}]$. At this point, it is illustrative to interpret the elements ${\widehat{q}}_{it}^{(j)}$ defined in (81) as linear predictions the factorization model $q_{it}={\sum }_j{a}_{ij}{x}_{jt}$, obtained from the components $a_{ij}{x}_{jt}$ and factors ${\gamma }_{it}^{(j)}(\{{\mathbf{Q}}^{(k)}\})$.

Appendix C, provides necessary and sufficient conditions for the convexity of $d_{AB}^{(\alpha ,\beta )}(p_{it},{\widehat{q}}_{it})$ with respect to ${\widehat{q}}_{it}$. Depending on the parameter β, it is required that one of the following condition be satisfied:

$$\begin{array}{c}\hfill \left\{\begin{array}{ccc}{\displaystyle \frac{p_{it}}{{\widehat{q}}_{it}}\ge c(\alpha ,\beta )}\hfill & \text{for}\hfill & \beta <min\{1,1-\alpha \},\hfill \\ \text{always}\text{convex}\hfill & \text{for}\hfill & \beta \in [min\{1,1-\alpha \},max\{1,1-\alpha \}],\hfill \\ {\displaystyle \frac{p_{it}}{{\widehat{q}}_{it}}\le c(\alpha ,\beta )}\hfill & \text{for}\hfill & \beta >max\{1,1-\alpha \},\hfill \end{array}\right.\end{array}$$

(85)

where the upper and lower bounds depend on the function

$$\begin{array}{c}\hfill c(\alpha ,\beta )={exp}_{1-\alpha }\left(\frac{1}{\beta -1}\right),\end{array}$$

(86)

whose contour plot is shown in Figure 4(b).

The convexity of the divergence w.r.t. $q_{it}$ holds for the parameters $(\alpha ,\beta )$ within the convex cone of $\beta \in [min\{1,1-\alpha \},max\{1,1-\alpha \}]$. Therefore, for this set of parameters, the monotonic descent in the AB-divergence with the update formulas (83), (84) is guaranteed.

Figure 4. Convexity analysis of $d_{AB}^{(\alpha ,\beta )}(p_{it},{\widehat{q}}_{it})$ w.r.t. ${\widehat{q}}_{it}$ in the $(\alpha ,\beta )$ plane.

Figure 4. Convexity analysis of $d_{AB}^{(\alpha ,\beta )}(p_{it},{\widehat{q}}_{it})$ w.r.t. ${\widehat{q}}_{it}$ in the $(\alpha ,\beta )$ plane.

On the other hand, even when $\beta \notin [min\{1,1-\alpha \},max\{1,1-\alpha \}]$, we can still guarantee the monotonic descent if the set of estimators ${\widehat{q}}_{it}\in [{min}_j{\widehat{q}}_{it}^{(j)},{max}_j{\widehat{q}}_{it}^{(j)}]$ is bounded and sufficiently close to $p_{it}$ so that they satisfy the conditions (85). Figure 4(b) illustrates this property demonstrating that if the ratios $\frac{p_{it}}{{\widehat{q}}_{it}}$ approach unity, the convexity of the divergence w.r.t. $q_{it}$ holds for an increasing size of $(\alpha ,\beta )$ plane. In other words, for sufficiently small relative errors between $p_{it}$ and ${\widehat{q}}_{it}$, the update formulas (83), (84) still guarantee a monotonic descent of the AB-divergence, within a reasonable wide range of the hyperparameters $(\alpha ,\beta )$.

3.4. Unconditional Auxiliary Function

The conditions of the previous section for a monotonic descent can be avoided by upper-bounding $d_{AB}^{(\alpha ,\beta )}\left(p_{it},{\widehat{q}}_{it}\right)$ with another function ${\overline{d}}_{AB}^{(\alpha ,\beta )}\left(p_{it},{\widehat{q}}_{it};q_{it}^{(k)}\right)$ which is convex with respect to ${\widehat{q}}_{it}$ and, at the same time, tangent to the former curve at ${\widehat{q}}_{it}=q_{it}^{(k)}$, i.e.,

$$\begin{array}{c}\hfill d_{AB}^{(\alpha ,\beta )}\left(p_{it},q_{it}\right)\le {\overline{d}}_{AB}^{(\alpha ,\beta )}\left(p_{it},{\widehat{q}}_{it};q_{it}^{(k)}\right),\end{array}$$

(87)

where

$$\begin{array}{c}\hfill d_{AB}^{(\alpha ,\beta )}\left(p_{it},q_{it}^{(k)}\right)={\overline{d}}_{AB}^{(\alpha ,\beta )}\left(p_{it},q_{it}^{(k)},q_{it}^{(k)}\right),\end{array}$$

(88)

and

$$\begin{array}{c}\hfill \frac{{\partial }^2{\overline{d}}_{AB}^{(\alpha ,\beta )}\left(p_{it},{\widehat{q}}_{it};q_{it}^{(k)}\right)}{\partial {\widehat{q}}_{it}^2}\ge 0.\end{array}$$

(89)

Similarly to the approach in [59,60,61] for the Beta-divergence, we have constructed an auxiliary function for the AB-divergence by linearizing those additive concave terms of the AB-divergence:

$$\begin{array}{c}\hfill {\overline{d}}_{AB}^{(\alpha ,\beta )}\left(p_{it},{\widehat{q}}_{it};q_{it}^{(k)}\right)=\left\{\begin{array}{ll}{\displaystyle \frac{p_{it}^{\alpha +\beta }}{\beta (\alpha +\beta )}+\frac{{(q_{it}^{(k)})}^{\alpha +\beta }}{\alpha (\alpha +\beta )}-\frac{p_{it}^{\alpha }{\widehat{q}}_{it}^{\beta }}{\alpha \beta }+\frac{{(q_{it}^{(k)})}^{\alpha +\beta -1}}{\alpha }({\widehat{q}}_{it}-q_{it}^{(k)}),}& \frac{\beta }{\alpha }<\frac{1}{\alpha }-1,\\ {\displaystyle \frac{p_{it}^{\alpha +\beta }}{\beta (\alpha +\beta )}+\frac{{\widehat{q}}_{it}^{\alpha +\beta }}{\alpha (\alpha +\beta )}-\frac{p_{it}^{\alpha }{\widehat{q}}_{it}^{\beta }}{\alpha \beta },}& \frac{\beta }{\alpha }\in [\frac{1}{\alpha }-1,\frac{1}{\alpha }],\\ {\displaystyle \frac{p_{it}^{\alpha +\beta }}{\beta (\alpha +\beta )}+\frac{{\widehat{q}}_{it}^{\alpha +\beta }}{\alpha (\alpha +\beta )}-\frac{p_{it}^{\alpha }{(q_{it}^{(k)})}^{\beta }}{\alpha \beta }-\frac{p_{it}^{\alpha }{(q_{it}^{(k)})}^{\beta -1}}{\alpha }({\widehat{q}}_{it}-q_{it}^{(k)}),}& \frac{\beta }{\alpha }>\frac{1}{\alpha }.\end{array}\right.\end{array}$$

The upper-bounds for singular cases can be obtained by continuity using L’Hópitals formula.

The unconditional auxiliary function for the surrogate minimization of the AB-divergence $D_{AB}^{(\alpha ,\beta )}(\mathbf{P}||\mathbf{Q})$ is now given by

$$\begin{array}{ccc}\hfill & & \overline{G}(\{\mathbf{Q}\},\{{\mathbf{Q}}^{(k)}\};\mathbf{P})=\sum _{i,t}\sum _j{\gamma }_{it}^{(j)}(\{{\mathbf{Q}}^{(k)}\}){\overline{d}}_{AB}^{(\alpha ,\beta )}\left(p_{it},{\widehat{q}}_{it}^{(j)};q_{it}^{(k)}\right),\hfill \end{array}$$

(90)

$$\begin{array}{ccc}& & \text{where}{\gamma }_{it}^{(j)}(\{{\mathbf{Q}}^{(k)}\})=\frac{a_{ij}^{(k)}{x}_{jt}^{(k)}}{q_{it}^{(k)}}\text{and}{\widehat{q}}_{it}^{(j)}=\frac{a_{ij}^{(k+1)}{x}_{jt}^{(k+1)}}{{\gamma }_{it}^{(j)}(\{{\mathbf{Q}}^{(k)}\})}.\hfill \end{array}$$

(91)

The alternating minimization of the above auxiliary function with respect to $\mathbf{A}$ and $\mathbf{X}$ yields the following stabilized iterations (with monotonic descent of the AB-divergence):

$$\begin{array}{ccc}\hfill a_{ij}^{(k+1)}& =& a_{ij}^{(k)}{\left[{exp}_{1-\alpha }\left(\sum _{t=1}^T\frac{{(q_{it}^{(k)})}^{\lambda -1}{x}_{jt}^{(k)}}{{\sum }_{t=1}^T{(q_{it}^{(k)})}^{\lambda -1}{x}_{jt}^{(k)}}{ln}_{1-\alpha }\left(\frac{p_{it}}{q_{it}^{(k)}}\right)\right)\right]}^{w(\alpha ,\beta )},\hfill \end{array}$$

(92)

$$\begin{array}{ccc}\hfill x_{jt}^{(k+1)}& =& x_{jt}^{(k)}{\left[{exp}_{1-\alpha }\left(\sum _{i=1}^I\frac{{({\tilde{q}}_{it}^{(k)})}^{\lambda -1}{a}_{ij}^{(k+1)}}{{\sum }_{i=1}^I{({\tilde{q}}_{it}^{(k)})}^{\lambda -1}{a}_{ij}^{(k+1)}}{ln}_{1-\alpha }\left(\frac{p_{it}}{{\tilde{q}}_{it}^{(k)}}\right)\right)\right]}^{w(\alpha ,\beta )},\hfill \end{array}$$

(93)

where

$$\begin{array}{c}\hfill w(\alpha ,\beta )=\left\{\begin{array}{ccc}1& & \alpha =0,\beta =1,\hfill \\ 0& & \alpha =0,\beta \ne 1,\hfill \\ {\displaystyle \frac{\alpha }{1-\beta }}& & \alpha \ne 0,\frac{\beta }{\alpha }<\frac{1}{\alpha }-1,\hfill \\ 1& & \alpha \ne 0,\frac{\beta }{\alpha }\in [\frac{1}{\alpha }-1,\frac{1}{\alpha }],\hfill \\ {\displaystyle \frac{\alpha }{\alpha +\beta -1}}& & \alpha \ne 0,\frac{\beta }{\alpha }>\frac{1}{\alpha }.\hfill \end{array}\right.\end{array}$$

(94)

The stabilized formulas (92), (93) coincide with (83), (84), except for the exponent $w(\alpha ,\beta )$, which is shown in Figure 5. This exponent is bounded between zero and one, and plays a similar role in the multiplicative update to that of the normalized step-size in an additive gradient descent update. Its purpose is to slow down the convergence so as to guarantee monotonic descent of the cost function. This is a consequence of the fact that the multiplicative correction term is progressively contracted towards the unity as $w(\alpha ,\beta )\to 0$. For the same reason, the stabilized formulas completely stop the updates for $\alpha =0$ and $\beta \ne 1$. Therefore, for α close to zero, we recommend to prevent this undesirable situation by enforcing a positive lower-bound in the value of exponent $w(\alpha ,\beta )$.

Figure 5. Surface plot of the exponent $w(\alpha ,\beta )$, whose role in the multiplicative update is similar to that of a normalized step-size in an additive gradient descent update.

Figure 5. Surface plot of the exponent $w(\alpha ,\beta )$, whose role in the multiplicative update is similar to that of a normalized step-size in an additive gradient descent update.

3.5. Multiplicative NMF Algorithms for Large-Scale Low-Rank Approximation

In practice, for low-rank approximations with $J\ll min\{I,T\}$ we do not need to process or save the whole data matrix $\mathbf{Y}$, nor is it necessary to perform computations at each iteration step of the products of the whole estimated matrices ${\mathbf{Y}}^T\mathbf{A}$ or $\mathbf{Y}{\mathbf{X}}^T$ (see Figure 6).

Figure 6. Conceptual factorization model of block-wise data processing for a large-scale NMF. Instead of processing the whole matrix $\mathbf{P}=\mathbf{Y}\in {\mathbb{R}}_{+}^{I\times T}$, we can process much smaller block matrices ${\mathbf{Y}}_c\in {\mathbb{R}}_{+}^{I\times C}$ and ${\mathbf{Y}}_r\in {\mathbb{R}}_{+}^{R\times T}$ and the corresponding factor matrices ${\mathbf{X}}_c={\mathbf{B}}_c^T={[{\mathit{b}}_{1,r},{\mathit{b}}_{2,r},\dots ,{\mathit{b}}_{J,r}]}^T\in {\mathbb{R}}_{+}^{J\times C}$ and ${\mathbf{A}}_r=[{\mathit{a}}_{1,r},{\mathit{a}}_{2,r},\dots ,{\mathit{a}}_{J,r}]\in {\mathbb{R}}_{+}^{R\times J}$ with $J<C<<T$ and $J<R<<I$. For simplicity of graphical illustration, we have selected the first C columns of the matrices $\mathbf{P}$ and $\mathbf{X}$ and the first R rows of $\mathbf{A}$.

Figure 6. Conceptual factorization model of block-wise data processing for a large-scale NMF. Instead of processing the whole matrix $\mathbf{P}=\mathbf{Y}\in {\mathbb{R}}_{+}^{I\times T}$, we can process much smaller block matrices ${\mathbf{Y}}_c\in {\mathbb{R}}_{+}^{I\times C}$ and ${\mathbf{Y}}_r\in {\mathbb{R}}_{+}^{R\times T}$ and the corresponding factor matrices ${\mathbf{X}}_c={\mathbf{B}}_c^T={[{\mathit{b}}_{1,r},{\mathit{b}}_{2,r},\dots ,{\mathit{b}}_{J,r}]}^T\in {\mathbb{R}}_{+}^{J\times C}$ and ${\mathbf{A}}_r=[{\mathit{a}}_{1,r},{\mathit{a}}_{2,r},\dots ,{\mathit{a}}_{J,r}]\in {\mathbb{R}}_{+}^{R\times J}$ with $J<C<<T$ and $J<R<<I$. For simplicity of graphical illustration, we have selected the first C columns of the matrices $\mathbf{P}$ and $\mathbf{X}$ and the first R rows of $\mathbf{A}$.

In other words, in order to perform the basic low-rank NMF

$$\begin{array}{c}\hfill \mathbf{Y}=\mathbf{P}=\mathbf{A}\mathbf{X}+\mathbf{E},\end{array}$$

we need to perform two associated nonnegative matrix factorizations using much smaller-scale matrices for large-scale problems, given by

$$\begin{array}{c}\hfill {\mathbf{Y}}_r={\mathbf{A}}_r\mathbf{X}+{\mathbf{E}}_r,\mathrm{for}\mathrm{fixed}(\mathrm{known}){\mathbf{A}}_r,\end{array}$$

(95)

$$\begin{array}{c}\hfill {\mathbf{Y}}_c=\mathbf{A}{\mathbf{X}}_c+{\mathbf{E}}_c,\mathrm{for}\mathrm{fixed}(\mathrm{known}){\mathbf{X}}_c,\end{array}$$

(96)

where ${\mathbf{Y}}_r\in {\mathbb{R}}_{+}^{R\times T}$ and ${\mathbf{Y}}_c\in {\mathbb{R}}_{+}^{I\times C}$ are the matrices constructed from the selected rows and columns of the matrix $\mathbf{Y}$, respectively. Analogously, we can construct the reduced matrices: ${\mathbf{A}}_r\in {\mathbb{R}}_{+}^{R\times J}$ and ${\mathbf{X}}_c\in {\mathbb{R}}_{+}^{J\times C}$ by using the same indices for the columns and rows as those used for the construction of the data sub-matrices ${\mathbf{Y}}_c$ and ${\mathbf{Y}}_r$. In practice, it is usually sufficient to choose: $J<R\le 4J$ and $J<C\le 4J$.

Using this approach, we can formulate the update learning rule for a large-scale multiplicative NMF as (see Figure 6)

$$\begin{array}{ccc}\hfill \mathbf{X}& \leftarrow & \mathbf{X}⊛{\left(\left({\mathbf{A}}_r^T({\mathbf{Y}}_r^{.[\alpha ]}⊛{\mathbf{Q}}_r^{.[\beta -1]})\right)\oslash \left({\mathbf{A}}_r^T{\mathbf{Q}}_r^{.[\alpha +\beta -1]}\right)\right)}^{.[w/\alpha ]},\hfill \end{array}$$

(97)

$$\begin{array}{ccc}\hfill \mathbf{A}& \leftarrow & \mathbf{A}⊛{\left(\left(\left({\mathbf{Y}}_c^{.[\alpha ]}⊛{\mathbf{Q}}_c^{.[\beta -1]}\right){\mathbf{X}}_c^T\right)\oslash \left({\mathbf{Q}}_c^{.[\alpha +\beta -1]}{\mathbf{X}}_c^T\right)\right)}^{.[w/\alpha ]},\hfill \end{array}$$

(98)

$$\begin{array}{ccc}\hfill {\mathbf{Q}}_r& =& {\mathbf{A}}_r\mathbf{X},{\mathbf{Q}}_c=\mathbf{A}{\mathbf{X}}_c.\hfill \end{array}$$

(99)

In fact, we need to save only two reduced set of data matrices ${\mathbf{Y}}_r\in {\mathbb{R}}_{+}^{R\times T}$ and ${\mathbf{Y}}_c\in {\mathbb{R}}_{+}^{I\times C}$. For example, for large dense data matrix $\mathbf{Y}\in {\mathbb{R}}_{+}^{I\times T}$ with $I=T={10}^5$ and $J=10$ and $R=C=50$, we need to save only ${10}^7$ nonzero entries instead of ${10}^{10}$ entries, thus reducing memory requirement 1000 times.

We can modify and extend AB NMF algorithms in several ways. First of all, we can use for update of matrices $\mathbf{A}$ and $\mathbf{X}$ two different cost functions, under the assumption that both the functions are convex with respect to one set of updated parameters (another set is assumed to be fixed). In a special case, we can use two AB-divergences (with two different sets of parameters: one $D_{AB}^{({\alpha }_A,{\beta }_A)}(\mathbf{Y}||\mathbf{A}\mathbf{X})$ for the estimation of $\mathbf{A}$ and fixed $\mathbf{X}$, and another one $D_{AB}^{({\alpha }_X,{\beta }_X)}(\mathbf{Y}||\mathbf{A}\mathbf{X})$ for the estimation of $\mathbf{X}$ and fixed $\mathbf{A}$). This leads to the following updates rules

$$\begin{array}{ccc}\hfill \mathbf{X}& \leftarrow & \mathbf{X}⊛{\left(\left({\mathbf{A}}_r^T({\mathbf{Y}}_r^{.[{\alpha }_X]}⊛{\mathbf{Q}}_r^{.[{\beta }_X-1]})\right)\oslash \left({\mathbf{A}}_r^T{\mathbf{Q}}_r^{.[{\alpha }_X+{\beta }_X-1]}\right)\right)}^{.[w_X/{\alpha }_X]},\hfill \end{array}$$

(100)

$$\begin{array}{ccc}\hfill \mathbf{A}& \leftarrow & \mathbf{A}⊛{\left(\left(\left({\mathbf{Y}}_c^{.[{\alpha }_A]}⊛{\mathbf{Q}}_c^{.[{\beta }_A-1]}\right){\mathbf{X}}_c^T\right)\oslash \left({\mathbf{Q}}_c^{.[{\alpha }_A+{\beta }_A-1]}{\mathbf{X}}_c^T\right)\right)}^{.[w_A/{\alpha }_A]},\hfill \end{array}$$

(101)

$$\begin{array}{ccc}\hfill {\mathbf{Q}}_r& =& {\mathbf{A}}_r\mathbf{X},{\mathbf{Q}}_c=\mathbf{A}{\mathbf{X}}_c.\hfill \end{array}$$

(102)

In order to accelerate the convergence of the algorithm, we can estimate one of the factor matrices, e.g., $\mathbf{A}$ by using the ALS (Alternating Least Squares) algorithm [7]. This leads to the following modified update rules which can still be robust with respect to $\mathbf{X}$ for suitably chosen set of the parameters.

$$\begin{array}{ccc}\hfill \mathbf{X}& \leftarrow & \mathbf{X}⊛{\left(\left({\mathbf{A}}_r^T({\mathbf{Y}}_r^{.[\alpha ]}⊛{({\mathbf{A}}_r\mathbf{X})}^{.[\beta -1]})\right)\oslash \left({\mathbf{A}}_r^T{({\mathbf{A}}_r\mathbf{X})}^{.[\alpha +\beta -1]}\right)\right)}^{.[w/\alpha ]},\hfill \end{array}$$

(103)

$$\begin{array}{ccc}\hfill \mathbf{A}& \leftarrow & max\{{\mathbf{Y}}_r{\mathbf{X}}_r^T{({\mathbf{X}}_r{\mathbf{X}}_r^T)}^{-1},\epsilon \}.\hfill \end{array}$$

(104)

Another alternative exists, which allows us to use different cost functions and algorithms for each of the factor matrices.

Furthermore, it would be very interesting to apply AB-multiplicative NMF algorithms for inverse problems in which matrix $\mathbf{A}$ is known and we need to estimate only matrix $\mathbf{X}$ for ill-conditioned and noisy data [62,63].

4. Simulations and Experimental Results

We have conducted extensive simulations with experiments designed specifically to address the following aspects:

• What is approximately a range of parameters alpha and beta for which the AB-multiplicative NMF algorithm exhibits the balance between relatively fastest convergence and good performance.
• What is approximately the range of parameters of alpha and beta for which the AB-multiplicative NMF algorithm provides a stable solution independent of how many iterations are needed.
• How robust is the AB-multiplicative NMF algorithm to noisy mixtures under multiplicative Gaussian noise, additive Gaussian noise, spiky biased noise? In other words, find a reasonable range of parameters for which the AB-multiplicative NMF algorithm gives improved performance when the data are contaminated by the different types of noise.

In order to test the performance of the AB-divergence for the NMF problem, we considered the matrix ${\mathbf{X}}^{*}$ with three non-negative sources (rows) shown in Figure 7(a). These sources were obtained by superposing pairs of signals from the “ACsincpos10” benchmark of the NMFLAB toolbox and truncating their length to the first 250 samples [7]. We mix these sources with a random mixing matrix ${\mathbf{A}}^{*}$ of dimension $25\times 5$, whose elements were drawn independently from a uniform random distribution in the unit interval. This way, we obtain a noiseless observation matrix $\mathbf{P}={\mathbf{A}}^{*}{\mathbf{X}}^{*}$ whose first five rows are displayed in Figure 7(b).

Figure 7. Illustration of simulation experiments with three nonnegative sources and their typical mixtures using a randomly generated (uniformly distributed) mixing matrix (rows of a data matrix $\mathbf{P}=\mathbf{A}\mathbf{X}+\mathbf{E}$ are denoted by ${\mathbf{p}}_1,{\mathbf{p}}_2,...$).

Figure 7. Illustration of simulation experiments with three nonnegative sources and their typical mixtures using a randomly generated (uniformly distributed) mixing matrix (rows of a data matrix $\mathbf{P}=\mathbf{A}\mathbf{X}+\mathbf{E}$ are denoted by ${\mathbf{p}}_1,{\mathbf{p}}_2,...$).

Our objective is to reconstruct from the noisy data $\mathbf{P}$, the matrix $\mathbf{X}$ of the nonnegative sources and the mixing matrix $\mathbf{A}$, by ignoring the scaling and permutation ambiguities.

The performance was evaluated with the mean Signal to Interference Ratio (SIR) of the estimated factorization model $\mathbf{A}\mathbf{X}$ and the mean SIR of the estimated sources (the rows of the $\mathbf{X}$) [7].

We evaluate the proposed NMF algorithm in (61) based on the AB-divergence for very large number of pairs of values of $(\alpha ,\beta )$, and a wide range of their values. The algorithm used a single trial random initialization, followed by a refinement which consist of running ten initial iterations of the algorithm with $(\alpha ,\beta )=(0.5,0.5)$. This initialization phase serves to approach the model to the observations ($\mathbf{A}\mathbf{X}\to \mathbf{P}$) which is important for guaranteing the posterior monotonic descent in the divergence when the parameters are arbitrary, as discussed in Section 3.3. Then, we ran only 250 iterations of the proposed NMF algorithm for the selected pair of parameters $(\alpha ,\beta )$.

To address the influence of noise, the observations were modeled as $\mathbf{P}={\mathbf{Q}}^{*}+\mathbf{E}$, where ${\mathbf{Q}}^{*}={\mathbf{A}}^{*}{\mathbf{X}}^{*}$ and $\mathbf{E}$ denote, respectively, the desired components and the additive noise. To cater for different types of noises, we assume that the elements of the additive noise $e_{it}$ are functions of the noiseless model $q_{it}^{*}$ and of another noise $z_{it}$, which was independent of $q_{it}^{*}$. Moreover, we also assume that the signal $q_{it}^{*}$ and the noise $z_{it}$ combine additively to give the observations in the deformed logarithm domain as

$$\begin{array}{c}\hfill {ln}_{1-{\alpha }^{*}}(p_{it})={ln}_{1-{\alpha }^{*}}(q_{it}^{*})+{ln}_{1-{\alpha }^{*}}(z_{it}),\end{array}$$

(105)

which is controlled by the parameter ${\alpha }^{*}$. Solving for the observations, we obtain

$$\begin{array}{c}\hfill p_{it}={exp}_{1-{\alpha }^{*}}\left({ln}_{1-{\alpha }^{*}}(q_{it}^{*})+{ln}_{1-{\alpha }^{*}}(z_{it})\right).\end{array}$$

(106)

or equivalently,

$$\begin{array}{c}\hfill p_{it}=\left\{\begin{array}{ll}ll{q}_{it}^{*}{z}_{it}& ,{\alpha }^{*}=0,\mathrm{multiplicative}\mathrm{noise},\\ q_{it}^{*}+z_{it}& ,{\alpha }^{*}=1,\mathrm{additive}\mathrm{noise},\\ {\left({(q_{it}^{*})}^{{\alpha }^{*}}+{(z_{it})}^{{\alpha }^{*}}\right)}^{\frac{1}{{\alpha }^{*}}}& ,{\alpha }^{*}\ne 0,\mathrm{additive}\mathrm{noise}\mathrm{in}\mathrm{a}\mathrm{deformed}\mathrm{log-domain}.\end{array}\right.\end{array}$$

(107)

Such approach allows us to model, under one single umbrella, noises of multiplicative type, additive noises, and other noises that act additively in a transformed domain, together with their distributions.

In order to generate the observations, we should assume first a probability density function $g({\overline{z}}_{it})$ for ${\overline{z}}_{it}\equiv {ln}_{1-{\alpha }^{*}}(z_{it})$, the noise in the transformed domain. This distribution is corrected, when necessary, to the nearby distribution that satisfies the positivity constraint of the observations.

In the following, we assume that the noise in the transformed domain ${\overline{z}}_{it}$ is Gaussian. Figure 8 presents mixtures obtained for this noise under different values of ${\alpha }^{*}$. The transformation (106) of the Gaussian density $g({\overline{z}}_{it})$ leads to the marginal distribution of the observations

$$\begin{array}{c}\hfill f(p_{it}|q_{it}^{*})=\frac{1}{\sqrt{2\pi }\sigma p_{it}^{1-{\alpha }^{*}}}{e}^{{\displaystyle -\frac{{\left({ln}_{1-{\alpha }^{*}}(p_{it})-{ln}_{1-{\alpha }^{*}}(q_{it}^{*})\right)}^2}{2{\sigma }^2}}},p_{it}>0,\end{array}$$

(108)

which can be recognized as a deformed log-Gaussian distribution (of order $1-{\alpha }^{*}$), with mean ${ln}_{1-{\alpha }^{*}}(q_{it}^{*})$ and variance ${\sigma }^2$. This distribution is exact for multiplicative noise (${\alpha }^{*}=0$), since in this case no correction of the distribution is necessary to guarantee the non-negativity of the observations. For the remaining cases the distribution is only approximate, but the approximation improves when ${\alpha }^{*}$ is not far from zero or when $q_{it}^{*}$ is sufficient large for all $i,t$, and for ${\sigma }^2$ sufficient small.

Figure 8. Illustration of the effect of the parameter ${\alpha }^{*}$ on the noisy observations (${\mathbf{p}}_1$ denotes the first row of the matrix $\mathbf{P}$). Dashed lines corresponds to noiseless mixtures and solid lines to the noisy mixtures that obtained when adding noise in the deformed logarithm (${ln}_{1-{\alpha }^{*}}(·)$) domain. The noise distribution was Gaussian of zero mean and with a variance chosen so as to obtain an SNR of 20 dB in the deformed logarithm domain. In the top panel the value of the deformation parameter ${\alpha }^{*}=0$, resulting a multiplicative noise that distorts more strongly signals $q_{it}^{*}$ with larger values. For the middle panel ${\alpha }^{*}=1$, resulting in an additive Gaussian noise that equally affects all $q_{it}^{*}$ independently of their values. For the bottom panel, ${\alpha }^{*}=3$, distorting more strongly small values of $q_{it}^{*}$.

Figure 8. Illustration of the effect of the parameter ${\alpha }^{*}$ on the noisy observations (${\mathbf{p}}_1$ denotes the first row of the matrix $\mathbf{P}$). Dashed lines corresponds to noiseless mixtures and solid lines to the noisy mixtures that obtained when adding noise in the deformed logarithm (${ln}_{1-{\alpha }^{*}}(·)$) domain. The noise distribution was Gaussian of zero mean and with a variance chosen so as to obtain an SNR of 20 dB in the deformed logarithm domain. In the top panel the value of the deformation parameter ${\alpha }^{*}=0$, resulting a multiplicative noise that distorts more strongly signals $q_{it}^{*}$ with larger values. For the middle panel ${\alpha }^{*}=1$, resulting in an additive Gaussian noise that equally affects all $q_{it}^{*}$ independently of their values. For the bottom panel, ${\alpha }^{*}=3$, distorting more strongly small values of $q_{it}^{*}$.

Interestingly, the log-likelihood of the matrix of observations for mutually independent components, distributed according to (108), is

$$\begin{array}{ccc}\hfill lnf(\mathbf{P}|\mathbf{Q})& =& -\sum _{i,t}ln\left(\sqrt{2\pi }\sigma p_{it}^{1-{\alpha }^{*}}\right)-\frac{1}{{\sigma }^2}{D}_{AB}^{({\alpha }^{*},{\alpha }^{*})}(\mathbf{P}\parallel \mathbf{Q}).\hfill \end{array}$$

(109)

Therefore, provided that (108) is sufficiently accurate, the maximization of the likelihood of the observations is equivalent to the minimization of the AB-divergence for $({\alpha }^{*},{\alpha }^{*})$, that is,

$$\begin{array}{ccc}\hfill arg\underset{\mathbf{Q}}{max}lnf(\mathbf{P}|\mathbf{Q})& \equiv & arg\underset{\mathbf{Q}}{min}{D}_{AB}^{({\alpha }^{*},{\alpha }^{*})}(\mathbf{P}\parallel \mathbf{Q}).\hfill \end{array}$$

(110)

Figure 9 presents the simulation results for mixtures with multiplicative noise. Usually, performances with SIR > 15 dB are considered successful. Observe that the domain where the performance is satisfactory is restricted to $\alpha +\beta \ge -0.5$, otherwise some terms could be extremely small due the inversion of the arguments of the AB-divergence. In other words, for $\alpha +\beta <-0.5$ there is too strong enhancement of the observations that correspond to the values of the model close to zero, deteriorating the performance. In our simulations we restricted the minimum value of entries of the true factorization model to a very small value of ${10}^{-7}$.

Figure 9. Performance of the AB-multiplicative NMF algorithm in the presence of multiplicative noise (${\alpha }^{*}=0$). The distribution of the noise in the transformed domain ${\overline{z}}_{it}$ is Gaussian of zero mean and with variance set to obtain an SNR of 20 dB in the $ln(·)$ domain. The rows of the observation matrix are shown in the top panel, the equivalent additive noise $\mathbf{E}=\mathbf{P}-{\mathbf{Q}}^{*}$ is displayed at the middle panel and the performance results are presented at the bottom panels. As theoretically expected, the best SIR of the model ($26.7$ dB) was achieved in the neighborhood of $(0,0)$, the parameters for which the likelihood of these observations is maximized. On the other hand, the best mean SIR of the sources ($18.0$ dB) and of the mixture ($21.1$ dB) are both obtained for $(\alpha ,\beta )$ close to $(-1.0,1.0)$.

Figure 9. Performance of the AB-multiplicative NMF algorithm in the presence of multiplicative noise (${\alpha }^{*}=0$). The distribution of the noise in the transformed domain ${\overline{z}}_{it}$ is Gaussian of zero mean and with variance set to obtain an SNR of 20 dB in the $ln(·)$ domain. The rows of the observation matrix are shown in the top panel, the equivalent additive noise $\mathbf{E}=\mathbf{P}-{\mathbf{Q}}^{*}$ is displayed at the middle panel and the performance results are presented at the bottom panels. As theoretically expected, the best SIR of the model ($26.7$ dB) was achieved in the neighborhood of $(0,0)$, the parameters for which the likelihood of these observations is maximized. On the other hand, the best mean SIR of the sources ($18.0$ dB) and of the mixture ($21.1$ dB) are both obtained for $(\alpha ,\beta )$ close to $(-1.0,1.0)$.

We confirmed this by extensive computer simulations. As theoretically predicted, for a Gaussian distribution of the noise ${\overline{z}}_{it}$ with ${\alpha }^{*}=0$, the best performance in the reconstruction of desired components was obtained in the neighborhood of the origin of the $(\alpha ,\beta )$ plane. The generalized Itakura-Saito divergence of Equation (46) gives usually a reasonable or best performance for the estimation of $\mathbf{X}$ and $\mathbf{A}$. The result of the simulation can be interpreted in light of the discussion presented in Section 2.3. As the multiplicative noise increases the distortion at large values of the model, the best performance was obtained for those parameters that prevent the inversion of the arguments of the divergence and, at the same time, suppress the observations that correspond with larger values of the model. This leads to the close to optimal choice of the parameters that satisfy equation $\alpha +\beta =0$.

The performance for the case of additive Gaussian noise (${\alpha }^{*}=1$) is presented in Figure 10. For an SNR of 20 dB, the distribution of the noisy observations was approximately Gaussian, since only $1\%$ of the coordinates of the matrix $\mathbf{P}$ were rectified to enforce positivity. This justifies the best performance for the NMF model in the neighborhood of the pair $(\alpha ,\beta )=(1,1)$, since according to (110) minimizing this divergence approximately maximizes the likelihood of these observations. Additionally, we observed poor performance for negative values of α, explained by the large ratios $p_{it}/q_{it}$ being much more unreliable, due to the distortion of the noise for the small values of $q_{it}^{*}$, together with the rectifying of the negative observations.

Figure 10. Performance of the AB-multiplicative NMF algorithm for 25 mixtures with additive Gaussian noise and SNR of 20 dB. The best performance for $\mathbf{A}\mathbf{X}$ was for an SIR of $31.1$ dB, obtained for $(\alpha ,\beta )=(0.8,0.7)$, that is, close to the pair $(1,1)$ that approximately maximizes the likelihood of the observations. The best performance for $\mathbf{X}$ and $\mathbf{A}$ was obtained in the vicinity of $(-0.2,0.8)$, with respective mean SIRs of $17.7$ dB and $20.5$ dB.

Figure 10. Performance of the AB-multiplicative NMF algorithm for 25 mixtures with additive Gaussian noise and SNR of 20 dB. The best performance for $\mathbf{A}\mathbf{X}$ was for an SIR of $31.1$ dB, obtained for $(\alpha ,\beta )=(0.8,0.7)$, that is, close to the pair $(1,1)$ that approximately maximizes the likelihood of the observations. The best performance for $\mathbf{X}$ and $\mathbf{A}$ was obtained in the vicinity of $(-0.2,0.8)$, with respective mean SIRs of $17.7$ dB and $20.5$ dB.

Figure 11 shows simulation results with ${\alpha }^{*}=3$, the case for which the effect of the noise was more pronounced for the small values of $q_{it}^{*}$. The noise in the transformed domain was again Gaussian and set to an SNR (in this domain) of 20 dB. In this case, the distribution of the observation was no longer sufficiently close to Gaussian because $11\%$ of the entries of the matrix $\mathbf{P}$ were rectified to enforce their positivity. The graphs reveal that the best performance was obtained for the pair $(\alpha ,\beta )=(0.9,4.0)$ that is quite close to a Beta-divergence. As a consequence of the strong distortion by the noise the small values of the factorization model the large and small ratios $p_{it}/q_{it}$ were not reliable, leaving as preferable the values of α close to unity. On the other hand, the ratios associated with small $q_{it}$ should be penalized, what leads to a larger parameter β as a robust choice against the contamination of the small values of the model.

Figure 11. Performance of the AB-multiplicative NMF algorithm when the observations are contaminated with Gaussian noise in the ${ln}_{1-{\alpha }^{*}}(·)$ domain, for ${\alpha }^{*}=3$. The best performance for $\mathbf{A}\mathbf{X}$ was for an SIR of $22.6$ dB obtained for $(\alpha ,\beta )=(0.9,4.0)$. A best SIR of $16.1$ dB for $\mathbf{X}$ was obtained for $(\alpha ,\beta )=(0.5,1.7)$, which gave an SIR for $\mathbf{A}$ of $19.1$ dB.

Figure 11. Performance of the AB-multiplicative NMF algorithm when the observations are contaminated with Gaussian noise in the ${ln}_{1-{\alpha }^{*}}(·)$ domain, for ${\alpha }^{*}=3$. The best performance for $\mathbf{A}\mathbf{X}$ was for an SIR of $22.6$ dB obtained for $(\alpha ,\beta )=(0.9,4.0)$. A best SIR of $16.1$ dB for $\mathbf{X}$ was obtained for $(\alpha ,\beta )=(0.5,1.7)$, which gave an SIR for $\mathbf{A}$ of $19.1$ dB.

The last two simulations, shown in Figure 12 and Figure 13, illustrate the effect of uniform spiky noise which was activated with a probability of $0.1$ and contained a bias in its mean. When the observations are biased downwards by the noise the performance improves for a positive α since, as Figure 3 illustrates, in this case the α suppresses the smaller ratios $p_{it}/q_{it}$. On the other hand, for a positive bias we have the opposite effect, resulting in the negative values of α being preferable. With these changes in α, the value of β should be modified accordingly to be in the vicinity of $-\alpha$ plus a given offset, so as to avoid an excessive penalty for observations that correspond to the large or small values of the factorization model $q_{it}$.

Figure 12. Performance for biased (non-zero mean) and spiky, additive noise. For ${\alpha }^{*}=1$, we have uniform noise with support in the negative unit interval, which is a spiky or sparse in the sense that it is only activated with a probability of $0.1$, i.e., it corrupts only $10\%$ of observed samples. The best SIR results were obtained around the line $(\alpha ,1-\alpha )$ for both positive and large values of α.

Figure 12. Performance for biased (non-zero mean) and spiky, additive noise. For ${\alpha }^{*}=1$, we have uniform noise with support in the negative unit interval, which is a spiky or sparse in the sense that it is only activated with a probability of $0.1$, i.e., it corrupts only $10\%$ of observed samples. The best SIR results were obtained around the line $(\alpha ,1-\alpha )$ for both positive and large values of α.

Figure 13. Performance for multiplicative noise that is positively biased and spiky (activated with a probability of $0.1$). For ${\alpha }^{*}=0$, the noise in the $ln(·)$ domain (${\overline{z}}_{it}$) followed a uniform distribution with support in the unit interval. The best SIR results were obtained along the line $(\alpha ,-\alpha )$ for negative values of α.

Figure 13. Performance for multiplicative noise that is positively biased and spiky (activated with a probability of $0.1$). For ${\alpha }^{*}=0$, the noise in the $ln(·)$ domain (${\overline{z}}_{it}$) followed a uniform distribution with support in the unit interval. The best SIR results were obtained along the line $(\alpha ,-\alpha )$ for negative values of α.

5. Conclusions

We have introduced the generalized Alpha-Beta divergence which serves as a flexible and robust cost function and forms a basis for the development of a new class of generalized multiplicative algorithms for NMF. Natural extensions of the Lee-Seung, ISRA, EMML and other NMF algorithms have been presented, in order to obtain more flexible and robust solutions with respect to different contaminations of the data by noise. This class of algorithms allows us to reconstruct (recover) the original signals and to estimate the mixing matrices, even when the observed data are imprecise and/or corrupted by noise. The optimal choice of the tuning parameters α and β depends strongly both on the distribution of noise and the way it contaminates the data. However, it should be emphasized that we are not expecting that the AB-multiplicative NMF algorithms proposed in this paper will work well for any set of parameters.

In summary, we have rigorously derived a new family of robust AB-multiplicative NMF algorithms using the generalized Alpha-Beta Divergence which unifies and extends a number of the existing divergences. The proposed AB-multiplicative NMF algorithms have been shown to work for wide sets of parameters and combine smoothly many existing NMF algorithms. Moreover, they are also adopted for large-scale low-rank approximation problems. Extensive simulation results have confirmed that the developed algorithms are efficient, robust and stable for a wide range of parameters. The proposed algorithms can also be extended to Nonnegative Tensor Factorizations and Nonnegative Tucker Decompositions in a straightforward manner.