Machine Learning with Squared-Loss Mutual Information

Abstract

Mutual information (MI) is useful for detecting statistical independence between random variables, and it has been successfully applied to solving various machine learning problems. Recently, an alternative to MI called squared-loss MI (SMI) was introduced. While ordinary MI is the Kullback–Leibler divergence from the joint distribution to the product of the marginal distributions, SMI is its Pearson divergence variant. Because both the divergences belong to the f-divergence family, they share similar theoretical properties. However, a notable advantage of SMI is that it can be approximated from data in a computationally more efficient and numerically more stable way than ordinary MI. In this article, we review recent development in SMI approximation based on direct density-ratio estimation and SMI-based machine learning techniques such as independence testing, dimensionality reduction, canonical dependency analysis, independent component analysis, object matching, clustering, and causal inference.

1. Introduction

Mutual information (MI) [1,2] for random variables X and Y is defined as:

$$\begin{array}{c}\hfill \mathrm{MI}(X,Y):=\int \int p(x,y)log\frac{p(x,y)}{p\left(x\right)p\left(y\right)}\mathrm{d}x\mathrm{d}y\end{array}$$

where $p(x,y)$ is the joint probability density of X and Y, and $p\left(x\right)$ and $p\left(y\right)$ are the marginal probability densities of X and Y, respectively (Precisely, $p(x,y)$, $p\left(x\right)$, and $p\left(y\right)$ are different functions and thus should be denoted, e.g., by $p_{\mathbf{X},\mathbf{Y}}(x,y)$, $p_{\mathbf{X}}\left(x\right)$, and $p_{\mathbf{Y}}\left(y\right)$, respectively. However, we use the simplified notations for the sake of brevity). Statistically, MI can be regarded as the Kullback–Leibler divergence [3] from the joint density $p(x,y)$ to the product of the marginals $p\left(x\right)p\left(y\right)$, and thus can be regarded as a measure of statistical dependency between X and Y. Estimation of MI from data samples has been one of the major challenges in information science and various approaches have been developed thus far.

The most naive approach to approximating MI from data would be to use a non-parametric density estimator such as kernel density estimation (KDE) [4], i.e., the densities $p(x,y)$, $p\left(x\right)$, and $p\left(y\right)$ included in MI are separately estimated from samples, and the estimated densities are used for approximating MI. However, density estimation is known to be a hard problem [5] and division by estimated densities tends to magnify the estimation error. Therefore, the KDE-based MI approximator may not be reliable in practice.

Another approach uses histogram-based density estimators with data-dependent partitioning. In the context of estimating the Kullback–Leibler divergence [3], histogram-based methods have been studied thoroughly and their consistency has been established [6,7,8]. However, the rate of convergence has not been elucidated yet, and such histogram-based methods are strongly influenced by the curse of dimensionality. Thus, these methods may not be reliable in high-dimensional problems.

MI can be expressed in terms of the entropies as:

$$\begin{array}{c}\hfill \mathrm{MI}(X,Y)=H\left(X\right)+H\left(Y\right)-H(X,Y)\end{array}$$

where $H\left(X\right)$ denotes the entropy of X:

$$\begin{array}{c}\hfill H\left(X\right):=-\int p\left(x\right)logp\left(x\right)\mathrm{d}x\end{array}$$

Based on this expression, the nearest neighbor distance has been used for approximating MI [9]. Such a nearest neighbor approach was shown to perform better than the naive KDE-based approach [10], given that the number k of nearest neighbors is chosen appropriately—a small (large) k yields an estimator with small (large) bias and large (small) variance. However, appropriately determining the value of k so that the bias-variance trade-off is optimally controlled is not straightforward in the context of MI estimation. A similar nearest-neighbor idea has been applied to Kullback–Leibler divergence estimation [11], whose consistency has been proved for finite k—this is an interesting result since Kullback–Leibler divergence estimation is consistent even when density estimation is not consistent. However, the rate of convergence seems to be still an open research issue.

Approximation of the entropies based on the Edgeworth expansion has also been explored in the context of MI estimation [12]. Such a method works well when the target density is close to Gaussian. However, if the target density is far from Gaussian, the Edgeworth expansion method is no longer reliable.

More recently, an MI approximator via direct estimation of the density ratio $\frac{p(x,y)}{p\left(x\right)p\left(y\right)}$ has been developed [13], which is based on a Kullback–Leibler divergence approximator via direct density-ratio estimation [14,15,16]. The MI approximator is given as the solution of a convex optimization problem, which tends to be sparse [14]. A notable advantage of this density-ratio method is that it does not involve separate estimation of densities $p(x,y)$, $p\left(x\right)$, and $p\left(y\right)$, and it was proved to achieve the optimal non-parametric convergence rate. However, due to the “log” operation included in MI, this MI approximator is computationally rather expensive and susceptible to outliers [17,18].

To cope with these problems, a variant of MI called the squared-loss mutual information (SMI) [19] has been explored recently. SMI for X and Y is defined as:

$$\begin{array}{c}\hfill \mathrm{SMI}(X,Y):=\frac{1}{2}\int \int p\left(x\right)p\left(y\right){\left(\frac{p(x,y)}{p\left(x\right)p\left(y\right)}-1\right)}^2\mathrm{d}x\mathrm{d}y\end{array}$$

SMI is the Pearson divergence [20] from the joint density $p(x,y)$ to the product of the marginals $p\left(x\right)p\left(y\right)$. It is always non-negative and it vanishes if and only if X and Y are statistically independent. Note that both the Pearson divergence and the Kullback–Leibler divergence belong to the class of Ali–Silvey–Csiszár divergences (which is also known as f-divergences) [21,22], meaning that they share similar properties.

In a similar way to ordinary MI, SMI can be approximated accurately via direct estimation of the density ratio $\frac{p(x,y)}{p\left(x\right)p\left(y\right)}$ [19], which is based on a Pearson divergence approximator via direct density-ratio estimation [16,23]. This SMI approximator has various desirable properties: For example, it was proved to achieve the optimal non-parametric convergence rate [24], its solution can be obtained analytically just by solving a system of linear equations, it has superior numerical properties [25], and it is robust against outliers [17,18].

In particular, the property of the SMI approximator that an analytic solution is available is highly useful in machine learning, because this allows explicit computation of the derivative of the SMI approximator with respect to another parameter. For example, in supervised dimensionality reduction, linear transformation U for input x is optimized so that the transformed input $Ux$ has the highest dependency on output y. In this context, the derivative of the SMI estimator between $Ux$ and y with respect to transformation U can be exploited for optimizing transformation U. On the other hand, such derivative computation is not straightforward for the MI estimator whose solution is obtained via numerical optimization.

The purpose of this article is to review recent development in SMI approximation based on direct density-ratio estimation and SMI-based machine learning techniques. The remainder of this paper is structured as follows. After reviewing the SMI approximator based on direct density-ratio estimation in Section 2, we illustrate in Section 3 how the SMI approximator can be utilized for solving various machine learning tasks such as: independence testing [26], feature selection [19,27], feature extraction [28,29], canonical dependency analysis [30], independent component analysis [31], object matching [32], clustering [33,34], and causality learning [35].

2. Definition and Estimation of SMI

In this section, we review the definition of SMI and its approximator based on direct density-ratio estimation.

2.1. Definition of SMI

Let us consider two random variables $X\in \mathcal{X}$ and $Y\in \mathcal{Y}$, where $\mathcal{X}$ and $\mathcal{Y}$ are domains of X and Y, respectively. Let $p(x,y)$ be the joint probability density of X and Y, and $p\left(x\right)$ and $p\left(y\right)$ be the marginal probability densities of X and Y, respectively. The squared-loss mutual information (SMI) [19] for X and Y is defined as:

$$\begin{array}{c}\hfill \mathrm{SMI}(X,Y):=\frac{1}{2}\int \int p\left(x\right)p\left(y\right){\left(\frac{p(x,y)}{p\left(x\right)p\left(y\right)}-1\right)}^2\mathrm{d}x\mathrm{d}y\end{array}$$

(1)

SMI is always non-negative and it takes zero if and only if X and Y are statistically independent. Hence, SMI can be used for detecting statistical independence between X and Y.

Below, we consider the problem of estimating SMI from paired samples ${\left\{(x_i,y_i)\right\}}_{i=1}^n$ drawn independently from the joint distribution with density $p(x,y)$.

2.2. Least-Squares Estimation of SMI

Here, we review the basic idea and theoretical properties of the SMI approximator called least-squares mutual information (LSMI) [19].

2.2.1. SMI Approximation via Direct Density-Ratio Estimation

The basic idea of LSMI is to directly estimate the following density-ratio function without going through density estimation of $p(x,y)$, $p\left(x\right)$, and $p\left(y\right)$:

$$\begin{array}{c}\hfill r(x,y):=\frac{p(x,y)}{p\left(x\right)p\left(y\right)}\end{array}$$

(2)

Let $g(x,y)$ be a model of the density ratio $r(x,y)$. In LSMI, the model is learned so that the following squared-error J is minimized:

$$\begin{array}{cc}\hfill J\left(g\right)& :=\frac{1}{2}\int \int {\left(g(x,y)-r(x,y)\right)}^{2}p\left(x\right)p\left(y\right)\mathrm{d}x\mathrm{d}y\hfill \\ \text{(3)}\hfill & & \phantom{:}=\frac{1}{2}\int \int g{(x,y)}^{2}p\left(x\right)p\left(y\right)\mathrm{d}x\mathrm{d}y-\int \int g(x,y)p(x,y)\mathrm{d}x\mathrm{d}y+C\hfill \end{array}$$

where C is a constant defined by:

$$\begin{array}{c}\hfill C:=\frac{1}{2}\int \int r(x,y)p(x,y)\mathrm{d}x\mathrm{d}y\end{array}$$

Since J contains the expectations over unknown densities $p\left(x\right)p\left(y\right)$ and $p(x,y)$, the expectations are approximated by empirical averages. Then the LSMI optimization problem is given as follows:

$$\begin{array}{c}\hfill \widehat{g}:=\underset{g\in \mathcal{G}}{\mathrm{argmin}}\left[\frac{1}{2n^2}\sum _{i,j=1}^{n}g{(x_i,y_j)}^2-\frac{1}{n}\sum _{i=1}^{n}g(x_i,y_i)\right]\end{array}$$

(4)

where $\mathcal{G}$ is a function space from which g is searched.

Finally, the SMI approximator called LSMI is given as:

$$\begin{array}{cc}\hfill \mathrm{LSMI}\left({\left\{(x_i,y_i)\right\}}_{i=1}^n\right)& :=\frac{1}{2n}\sum _{i=1}^n\widehat{g}(x_i,y_i)-\frac{1}{2}\hfill \end{array}$$

(5)

or

$$\begin{array}{cc}\hfill {\mathrm{LSMI}}^{\prime }\left({\left\{(x_i,y_i)\right\}}_{i=1}^n\right)& :=-\frac{1}{2n^2}\sum _{i,j=1}^n\widehat{g}{(x_i,y_j)}^2+\frac{1}{n}\sum _{i=1}^n\widehat{g}(x_i,y_i)-\frac{1}{2}\hfill \end{array}$$

(6)

Equation (5) would be the simplest SMI approximator, while Equation (6) is suitable for theoretical analysis because this corresponds to the negative of the objective function (4) up to the constant $1/2$. These estimators are derived based on the following equivalent expressions of SMI:

$$\begin{array}{cc}\hfill \mathrm{SMI}(X,Y)& =\frac{1}{2}\int \int r(x,y)p(x,y)\mathrm{d}x\mathrm{d}y-\frac{1}{2}\hfill \\ \text{(8)}\hfill & & =-\frac{1}{2}\int \int r{(x,y)}^{2}p\left(x\right)p\left(y\right)\mathrm{d}x\mathrm{d}y+\int \int r(x,y)p(x,y)\mathrm{d}x\mathrm{d}y-\frac{1}{2}\hfill \end{array}$$

(7)

Equation (7) is obtained by expanding the squared term in Equation (1), applying Equation (2) to the squared density-ratio term once, and showing that the cross-term and the remaining terms are $-1$ and $1/2$, respectively. Equivalence between Equations (7) and (8) can be confirmed by applying Equation (2) to the first term in Equation (8) once. Note that Equation (8) can also be obtained via the Legendre–Fenchel duality of Equation (1), implying that the optimization problem (4) corresponds to approximately maximizing the Legendre–Fenchel lower-bound [15].

2.2.2. Convergence Analysis

Here we briefly review theoretical convergence properties of LSMI.

First, let us consider the case where the function class $\mathcal{G}$ from which the function g is searched is a parametric model:

$$\mathcal{G}=\{g_{\theta }(x,y)\mid \theta \in \Theta \subset {\mathbb{R}}^b\}$$

Suppose that the true density-ratio r is contained in the model $\mathcal{G}$, i.e., there exists ${\theta }^{*}$ ($\in \Theta$) such that: $r=g_{{\theta }^{*}}$. Then, it was shown [28] that, under the standard regularity conditions for consistency [36], it holds that:

$$\begin{array}{c}\hfill {\mathrm{LSMI}}^{\prime }\left({\left\{(x_i,y_i)\right\}}_{i=1}^n\right)-\mathrm{SMI}(X,Y)={\mathcal{O}}_p\left(n^{-1/2}\right)\end{array}$$

where ${\mathcal{O}}_p$ denotes the asymptotic order in probability. This shows that ${\mathrm{LSMI}}^{\prime }$ retains the optimality in terms of the order of convergence in n, because ${\mathcal{O}}_p\left(n^{-1/2}\right)$ is the optimal convergence rate in the parametric setup [37].

Next, we consider non-parametric cases where the function class $\mathcal{G}$ is a reproducing kernel Hilbert space [38] on $\mathcal{X}\times \mathcal{Y}$. Let us consider a non-parametric version of the LSMI optimization problem:

$$\begin{array}{c}\hfill \widehat{g}:=\underset{g\in \mathcal{G}}{\mathrm{argmin}}\left[\frac{1}{2n^2}\sum _{i,j=1}^{n}g{(x_i,y_j)}^2-\frac{1}{n}\sum _{i=1}^{n}g(x_i,y_i)+\frac{{\lambda }_n}{2}{\parallel g\parallel }_{\mathcal{G}}^2\right]\end{array}$$

where ${\parallel ·\parallel }_{\mathcal{G}}^2$ denotes the norm in the reproducing kernel Hilbert space $\mathcal{G}$. In the above optimization problem, a regularizer ${\parallel g\parallel }_{\mathcal{G}}^2$ is included to avoid overfitting and ${\lambda }_n\ge 0$ is the regularization parameter.

Suppose that the true density-ratio function r is contained in the function space $\mathcal{G}$ and is bounded from above. Then, it was shown [28] that, if ${\lambda }_n\to 0$ and ${\lambda }_n^{-1}=o\left(n^{2/(2+\gamma )}\right)$ where γ$(0<\gamma <2)$ denotes a complexity measure of the function space $\mathcal{G}$ based on the bracketing entropy (The larger the value of γ is, the more complex the function space $\mathcal{G}$ is) [see p.83 of 36], it holds that:

$${\mathrm{LSMI}}^{\prime }\left({\left\{(x_i,y_i)\right\}}_{i=1}^n\right)-\mathrm{SMI}(X,Y)={\mathcal{O}}_p\left(max({\lambda }_n,n^{-1/2})\right)$$

(9)

The conditions ${\lambda }_n\to 0$ and ${\lambda }_n^{-1}=o\left(n^{2/(2+\gamma )}\right)$ roughly mean that the regularization parameter ${\lambda }_n$ should be sufficiently small but not too small. Equation (9) means that the convergence rate of the non-parametric version can also be ${\mathcal{O}}_p\left(n^{-1/2}\right)$ for an appropriate choice of ${\lambda }_n$, but the non-parametric method requires a milder model assumption. According to [15], the above convergence rate is the minimax optimal rate under some setup. Thus, the convergence property of the above non-parametric method would also be optimal in the same sense.

2.3. Practical Implementation of LSMI

We have seen that LSMI has desirable convergence properties. Here we review practical implementation of LSMI. A MATLAB${}^{\text{®}}$ implementation of LSMI is publicly available [39].

2.3.1. LSMI for Linear-in-Parameter Models

Let us approximate the density ratio Equation (2) using the following linear-in-parameter model:

$$\begin{array}{cc}\hfill g_{\theta }(x,y)& =\sum _{\ell =1}^b{\theta }_{\ell }{\varphi }_{\ell }(x,y)={\theta }^{\top }\varphi (x,y)\hfill \end{array}$$

(10)

where $\theta ={({\theta }_1,\dots ,{\theta }_b)}^{\top }$ are parameters, $\varphi (x,y)={({\varphi }_1(x,y),\dots ,{\varphi }_b(x,y))}^{\top }$ are fixed basis functions, and ${}^{\top }$ denotes the transpose. Practical choices of the basis functions will be explained in Section 2.3.2. For this model, the LSMI optimization problem with an ${\ell }_2$-regularizer is expressed as:

$$\begin{array}{c}\hfill \widehat{\theta }:=\underset{\theta \in {\mathbb{R}}^b}{\mathrm{argmin}}\left[\frac{1}{2}{\theta }^{\top }\widehat{H}\theta -{\theta }^{\top }\widehat{h}+\frac{\lambda }{2}{\theta }^{\top }\theta \right]\end{array}$$

where $\lambda \ge 0$ is the regularization parameter that controls the strength of regularization, $\widehat{H}$ is the $b\times b$ matrix defined by:

$$\begin{array}{cc}\hfill \widehat{H}& :=\frac{1}{n^2}\sum _{i,j=1}^n\varphi (x_i,y_j)\varphi {(x_i,y_j)}^{\top }\hfill \end{array}$$

and $\widehat{h}$ is the b-dimensional vector defined by:

$$\begin{array}{cc}\hfill \widehat{h}& :=\frac{1}{n}\sum _{i=1}^n\varphi (x_i,y_i)\hfill \end{array}$$

The solution $\widehat{\theta }$ can be analytically obtained as:

$$\begin{array}{c}\hfill \widehat{\theta }={(\widehat{H}+\lambda I_b)}^{-1}\widehat{h}\end{array}$$

(11)

where $I_b$ is the b-dimensional identity matrix. Finally, LSMI is also given analytically as:

$$\begin{array}{cc}\hfill \mathrm{LSMI}\left({\left\{(x_i,y_i)\right\}}_{i=1}^n\right)& =\frac{1}{2}{\widehat{h}}^{\top }\widehat{\theta }-\frac{1}{2}\hfill \end{array}$$

(12)

or

$$\begin{array}{cc}\hfill {\mathrm{LSMI}}^{\prime }\left({\left\{(x_i,y_i)\right\}}_{i=1}^n\right)& =-\frac{1}{2}{\widehat{\theta }}^{\top }\widehat{H}\widehat{\theta }+{\widehat{h}}^{\top }\widehat{\theta }-\frac{1}{2}\hfill \end{array}$$

(13)

Some elements of $\widehat{\theta }$ may take negative values in the above formulation, which can lead to negative density-ratio values and negative LSMI values. Such negative values may be rounded up to zero if necessary, although this does not happen for sufficiently large n. Another option is to explicitly impose the non-negativity constraint ${\theta }_1,\dots ,{\theta }_b\ge 0$ on the optimization problem. However, by this modification, the solution can no longer be obtained analytically, but only numerically using a quadratic program solver. (In this case, if the ${\ell }_2$-regularizer is replaced with the ${\ell }_1$-regularizer, the regularization path [40,41]—i.e., solutions for all different regularization parameter values—can be computed efficiently without a quadratic program solver just by solving systems of linear equation [23].)

2.3.2. Design of Basis Functions

The practical accuracy of LSMI depends on the choice of basis functions in the model Equation (10). A typical choice is a non-parametric kernel model, i.e., setting the number of basis function to $b=n$ and the ℓ-th basis function to ${\varphi }_{\ell }(x,y)=K(x,x_{\ell })L(y,y_{\ell })$:

$$\begin{array}{cc}\hfill g_{\theta }(x,y)& =\sum _{\ell =1}^n{\theta }_{\ell }K(x,x_{\ell })L(y,y_{\ell })\hfill \end{array}$$

(14)

where $K(x,x^{\prime })$ and $L(y,y^{\prime })$ are kernel functions for x and y, respectively. If n is too large, b may be set to be smaller than n and choose a subset of data points ${\left\{(x_i,y_i)\right\}}_{i=1}^n$ as kernel centers.

For real vector $x\in {\mathbb{R}}^d$, we may practically use the Gaussian kernel for $K(x,x^{\prime })$ after element-wise variance normalization:

$$\begin{array}{c}\hfill K(x,x^{\prime })=exp\left(-\frac{\parallel x-x^{\prime }{\parallel }^2}{2{\sigma }_{\mathrm{x}}^2}\right)\end{array}$$

where ${\sigma }_{\mathrm{x}}>0$ is the Gaussian width. When x is a non-vectorial structured object such as a string, a tree, and a graph, we may employ a kernel function defined for such structured data [42].

In the (multi-output) regression scenario where y is a real vector, the Gaussian kernel may also be used for $L(y,y^{\prime })$ after element-wise variance normalization:

$$\begin{array}{c}\hfill L(y,y^{\prime })=exp\left(-\frac{\parallel y-y^{\prime }{\parallel }^2}{2{\sigma }_{\mathrm{y}}^2}\right)\end{array}$$

where ${\sigma }_{\mathrm{y}}>0$ is the Gaussian width. In the multi-class classification scenario where $y\in \{1,\dots ,c\}$ and c denotes the number of classes, we may use the delta kernel for $L(y,y^{\prime })$:

$$\begin{array}{c}\hfill L(y,y^{\prime })=\left\{\begin{array}{cc}1\hfill & ify=y^{\prime }\hfill \\ 0\hfill & ify\ne y^{\prime }\hfill \end{array}\right.\end{array}$$

Note that, in the classification case with the delta kernel, the LSMI solution can be computed efficiently in a class-wise manner [33]. In the multi-label classification scenario where $y\in {\{0,1\}}^c$ and c denotes the number of labels, we may use the normalized linear kernel function [43] for y:

$$\begin{array}{c}\hfill L(y,y^{\prime })=\frac{{(y-\overline{y})}^{\top }(y^{\prime }-\overline{y})}{\parallel y-\overline{y}\parallel \parallel y^{\prime }-{\overline{y}}^{\prime }\parallel }\end{array}$$

where $\overline{y}=\frac{1}{n}{\sum }_{i=1}^n{y}_i$ is the sample mean.

2.3.3. Model Selection by Cross-Validation

Most of the above kernels include tuning parameters such as the Gaussian width, and the practical performance of LSMI depends on the choice of such kernel parameters and the regularization parameter λ. Model selection of LSMI is possible based on cross-validation with respect to the criterion J defined by Equation (3).

More specifically, the sample set $\mathcal{D}={\left\{(x_i,y_i)\right\}}_{i=1}^n$ is divided into M disjoint subsets ${\left\{{\mathcal{D}}_m\right\}}_{m=1}^M$. Then the LSMI solution ${\widehat{g}}_m\left(x\right)$ is obtained using $\mathcal{D}\backslash {\mathcal{D}}_m$ (i.e., all samples without ${\mathcal{D}}_m$), and its J-score for the hold-out samples ${\mathcal{D}}_m$ is computed as:

$$\begin{array}{c}\hfill {\widehat{J}}_m^{\mathrm{CV}}:=\frac{1}{2|{\mathcal{D}}_m{|}^2}\sum _{x,y\in {\mathcal{D}}_m}{\widehat{g}}_m{(x,y)}^2-\frac{1}{|{\mathcal{D}}_m|}\sum _{(x,y)\in {\mathcal{D}}_m}{\widehat{g}}_m(x,y)\end{array}$$

where $|{\mathcal{D}}_m|$ denotes the number of elements in the set ${\mathcal{D}}_m$. ${\sum }_{x,y\in {\mathcal{D}}_m}$ denotes the summation over all combinations of x and y in ${\mathcal{D}}_m$ (and thus $|{\mathcal{D}}_m{|}^2$ terms), while ${\sum }_{(x,y)\in {\mathcal{D}}_m}$ denotes the summation over all pairs $(x,y)$ in ${\mathcal{D}}_m$ (and thus $|{\mathcal{D}}_m|$ terms). This procedure is repeated for $m=1,\dots ,M$, and the average score,

$$\begin{array}{c}\hfill {\widehat{J}}^{\mathrm{CV}}:=\frac{1}{M}\sum _{m=1}^M{\widehat{J}}_m^{\mathrm{CV}}\end{array}$$

is computed. Finally, the model (the kernel parameter and the regularization parameter in the current setup) that minimizes ${\widehat{J}}^{\mathrm{CV}}$ is chosen as the most suitable one.

3. SMI-Based Machine Learning

In this section, we show how the SMI estimator, LSMI, can be used for solving various machine learning tasks.

3.1. Independence Testing

First, we show how the SMI estimator can be used for independence testing.

3.1.1. Introduction

Identifying the statistical independence between random variables is one of the fundamental challenges in statistical data analysis. A traditional independence measure between random variables is the Pearson correlation coefficient, which can be used for detecting linear dependency. Recently, kernel-based independence measures have been studied to detect non-linear dependency. The Hilbert–Schmidt independence criterion (HSIC) [44] utilizes cross-covariance operators on universal reproducing kernel Hilbert spaces (RKHSs) [45], which is an infinite-dimensional generalization of covariance matrices. HSIC allows efficient detection of non-linear dependency by making use of the reproducing property of RKHSs [38]. However, HSIC has a weakness that its performance depends on the choice of RKHSs and there is no theoretically justified way to determine the RKHS properly thus far. In practice, using the Gaussian RKHS with width set to the median distance between samples is a popular heuristic [46], but this does not always work well.

To overcome the above limitations, an SMI-based independence test called least-squares independence test (LSIT) was proposed [26]. Below, we review LSIT.

3.1.2. Independence Testing with SMI

Let $x\in \mathcal{X}$ be an input feature and $y\in \mathcal{Y}$ be an output feature, which follow a joint probability distribution with density $p(x,y)$. Suppose that we are given a set of independent and identically distributed (i.i.d.) paired samples ${\left\{(x_i,y_i)\right\}}_{i=1}^n$. The objective of independence testing is to conclude whether x and y are statistically independent or not, based on the samples ${\left\{(x_i,y_i)\right\}}_{i=1}^n$.

The SMI-based independence test, where the null hypothesis is that x and y are statistically independent, is based on the permutation test procedure [47]. More specifically, LSMI is first run using the original dataset $\mathcal{D}={\left\{(x_i,y_i)\right\}}_{i=1}^n$, and an SMI estimate, $\mathrm{LSMI}\left(\mathcal{D}\right)$, is obtained. Next, ${\left\{y_i\right\}}_{i=1}^n$ are randomly permuted and a shuffled dataset $\tilde{\mathcal{D}}={\left\{(x_i,{\tilde{y}}_i)\right\}}_{i=1}^n$ is formed, where ${\left\{{\tilde{y}}_i\right\}}_{i=1}^n$ denote permuted samples. Then LSMI is run again using the shuffled dataset $\tilde{\mathcal{D}}$, and an SMI estimate $\mathrm{LSMI}\left(\tilde{\mathcal{D}}\right)$ is obtained. Note that the random permutation eliminates the dependency between x and y (if it exists), and therefore $\mathrm{LSMI}\left(\tilde{\mathcal{D}}\right)$ would take a value close to zero. This random permutation procedure is repeated many times, and the distribution of $\mathrm{LSMI}\left(\tilde{\mathcal{D}}\right)$ under the null-hypothesis that x and y are statistically independent is constructed. Finally, the p-value is approximated by evaluating the relative ranking of $\mathrm{LSMI}\left(\mathcal{D}\right)$ in the distribution of $\mathrm{LSMI}\left(\tilde{\mathcal{D}}\right)$.

This procedure is called the least-squares independence test (LSIT) [26]. A MATLAB${}^{\text{®}}$ implementation of LSIT is publicly available [48].

3.2. Supervised Feature Selection

Next, we show how the SMI estimator can be used for supervised feature selection.

3.2.1. Introduction

The objective of supervised learning is to learn an input-output relation from input-output paired samples. However, when the dimensionality of input vectors is large, using all input elements could lead to a model interpretability problem. Feature selection is aimed at finding a subset of input elements that is useful for predicting output values [49].

Feature ranking is a simple implementation of feature selection that ranks each feature according to its relevance. In this feature ranking scenario, SMI between a single input variable and an output was shown to be useful [19]. However, feature ranking does not take feature interaction into account, and thus it is not useful when each single feature is not capable of predicting outputs, but multiple features are necessary for a valid prediction of outputs (e.g., an XOR problem). Two criteria, relevancy and redundancy, are often used to select multiple features simultaneously: A feature is said to be relevant if it can explain outputs, and features are said to be redundant if they are similar. Ideally, we want to find a subset of features that has high relevance and low redundancy.

Another important issue in feature selection is the computational cost: Naively selecting multiple features causes computational infeasibility because the number of possible feature combinations is exponential with respect to the number of input features. To cope with this problem, a computationally efficient method to handle multiple features called the least absolute shrinkage and selection operator (LASSO) [50] was proposed. In LASSO, a predictor consisting of a weighted sum of each feature is fitted to output values using the least-squares method, while the weight vector is confined in an ${\ell }_1$-ball. The ${\ell }_1$-ball restriction actually provides a notable property that the solution is sparsified, meaning that some of the weight parameters become exactly zero. Thus, LASSO automatically removes irrelevant features from its predictor, which can be achieved through convex optimization in a computationally efficient way [51,52].

However, LASSO can only handle linear predictors and its feature selection characteristic explicitly depends on the squared-loss function used in the least-squares method. To go beyond these limitations, an SMI-based feature selection method called ${\ell }_1$-LSMI was proposed [27]. Below, we review ${\ell }_1$-LSMI.

3.2.2. Feature Selection with SMI

The objective of feature selection is, from input feature vector $x={(x^{\left(1\right)},\cdots ,x^{\left(d\right)})}^{\top }\in {\mathbb{R}}^d$, to choose a subset of its elements that are useful for the prediction of output $y\in \mathcal{Y}$. Suppose that we are given n i.i.d. paired samples ${\left\{(x_i,y_i)\right\}}_{i=1}^n$ drawn from a joint distribution with density $p(x,y)$. Let $w_1,\dots ,w_d$ be feature weights for $x^{\left(1\right)},\cdots ,x^{\left(d\right)}$, and we learn the weights as:

$$\begin{array}{cc}\hfill \underset{w_1,\dots ,w_d}{max}& \mathrm{LSMI}\left({\left\{\left({(w_1{x}_i^{\left(1\right)},\dots ,w_d{x}_i^{\left(d\right)})}^{\top },y_i\right)\right\}}_{i=1}^n\right)\hfill \\ \hfill \mathrm{subject}\mathrm{to}& \sum _{i=1}^d{w}_i\le \eta \text{ and }{w}_1,\dots ,w_d\ge 0\hfill \end{array}$$

where $\eta \ge 0$ is the regularization parameter that controls the number of features. Because the sign of feature weights is not relevant in feature selection, they are restricted to be non-negative. For non-negative weights, ${\sum }_{i=1}^d{w}_i$ is reduced to the ${\ell }_1$-norm of the feature weight vector ${(w_1,\dots ,w_d)}^{\top }$. The features having zero weights are regarded as irrelevant in this formulation.

To compute the solution, a simple gradient ascent may be used, where the weight vector is projected onto the positive orthant of the ${\ell }_1$-ball in each iteration to guarantee the feasibility. This can be performed by first projecting the weight vector onto the positive orthant by rounding up negative elements to zero, and then projecting it onto the ${\ell }_1$-ball [54].

This SMI-based feature selection algorithm is called the ${\ell }_1$-LSMI [27]. A MATLAB${}^{\text{®}}$ implementation of ${\ell }_1$-LSMI is publicly available [53].

3.3. Supervised Feature Extraction

While feature selection chooses a subset of features for enhancing model interpretability, feature extraction finds a low-dimensional representation of features that can depend on all features (e.g., through linear combination) for improving the prediction accuracy. Here, we show how the SMI estimator can be used for supervised feature extraction.

3.3.1. Introduction

The goal of sufficient dimension reduction (SDR) is to map input features to low-dimensional expressions while “sufficient” information for predicting output values is maintained [55]. Earlier SDR methods developed in the statistics community, such as sliced inverse regression [56], principal Hessian direction [57], and sliced average variance estimation [58], rely on the ellipticity of the data (e.g., Gaussian), but such an assumption may not be fulfilled in practice. To overcome the limitations of these approaches, kernel dimension reduction (KDR) was proposed [59]. KDR employs a kernel-based dependence measure that is distribution-free, and thus KDR is flexible. However, it lacks systematic model selection strategies for kernel and regularization parameters. Furthermore, KDR scales poorly to massive datasets and there is no good way to set an initial solution for its gradient-based optimization. In practice, many restarts from different initial solutions may be needed for finding a good local optimum, which makes the entire procedure even slower and the performance of dimension reduction unreliable.

To overcome the above limitations, an SMI-based SDR method called least-squares dimension reduction (LSDR) was proposed [28]. An advantage of LSDR is that its tuning parameters can be systematically optimized based on cross-validation. To further address the computational and initialization issues, a heuristic search strategy for LSDR called sufficient component analysis (SCA) was proposed [29]. Below, we review LSDR and SCA.

3.3.2. Sufficient Dimension Reduction with SMI

First, we formulate the problem of SDR [55]. Let $x\in {\mathbb{R}}^{d_{\mathrm{x}}}$ be an input vector and $y\in \mathcal{Y}$ be an output. The goal of SDR is to find a subspace of input domain ${\mathbb{R}}^{d_{\mathrm{x}}}$ that contains “sufficient” information about output y. We assume that there exists an orthogonal matrix $U^{*}\in {\mathbb{R}}^{d_{\mathrm{u}}\times d_{\mathrm{x}}}$ for $d_{\mathrm{u}}\le d_{\mathrm{x}}$ such that

$$\begin{array}{c}\hfill y\perp \perp x|U^{*}x\end{array}$$

(15)

That is, given the projected feature $U^{*}x$, the (remaining) feature x is conditionally independent of output y and thus can be discarded without sacrificing the predictability of y. The objective of SDR is to find such $U^{*}$ from n i.i.d. paired samples, ${\left\{(x_i,y_i)\right\}}_{i=1}^n$, drawn from a joint distribution with density $p(x,y)$. We assume that the projection dimensionality $d_{\mathrm{u}}$ is known.

SMI can be used for characterizing the optimal projection matrix $U^{*}$ [28]. Indeed, it was shown that inequality,

$$\begin{array}{c}\hfill \mathrm{SMI}(X,Y)\ge \mathrm{SMI}(UX,Y)\end{array}$$

holds, and the equality holds if and only if Equation (15) holds. Thus, maximizing $\mathrm{SMI}(UX,Y)$ with respect to U leads to $U^{*}$. In practice, the following optimization problem is solved:

$$\begin{array}{cc}\hfill \underset{U\in {\mathbb{R}}^{d_{\mathrm{u}}\times d_{\mathrm{x}}}}{max}& \mathrm{LSMI}\left({\left\{(U{x}_i,y_i)\right\}}_{i=1}^n\right)\hfill \\ \hfill \mathrm{subject}\mathrm{to}& U{U}^{\top }=I_{d_{\mathrm{u}}}\hfill \end{array}$$

This formulation is called least-squares dimension reduction (LSDR) [28].

3.3.3. Gradient-Based Subspace Search

A simple approach to solving the above LSDR optimization problem is the following iterative procedure:

• U is updated to ascend the gradient of $\mathrm{LSMI}\left({\left\{(U{x}_i,y_i)\right\}}_{i=1}^n\right)$ with respect to U.
• U is projected onto the feasible region specified by $U{U}^{\top }=I_{d_{\mathrm{u}}}$.

The gradient of $\mathrm{LSMI}\left({\left\{(U{x}_i,y_i)\right\}}_{i=1}^n\right)$ with respect to U is given by:

$$\begin{array}{c}\hfill \frac{\partial \mathrm{LSMI}}{\partial U}=\sum _{\ell =1}^b{\widehat{\theta }}_{\ell }\frac{\partial {\widehat{h}}_{\ell }}{\partial U}-\frac{1}{2}\sum _{\ell ,{\ell }^{\prime }=1}^b{\widehat{\theta }}_{\ell }{\widehat{\theta }}_{{\ell }^{\prime }}\frac{\partial {\widehat{H}}_{\ell ,{\ell }^{\prime }}}{\partial U}\end{array}$$

If the kernel model Equation (14) with the Gaussian kernel,

$$\begin{array}{c}\hfill K(Ux,U{x}^{\prime })=exp\left(-\frac{\parallel Ux-U{x}^{\prime }{\parallel }^2}{2{\sigma }^2}\right)\end{array}$$

is used, $\frac{\partial {\widehat{h}}_{\ell }}{\partial U}$ and $\frac{\partial {\widehat{H}}_{\ell ,{\ell }^{\prime }}}{\partial U}$ (for $\ell ,{\ell }^{\prime }=1,\dots ,n$) are given by:

$$\begin{array}{cc}\hfill \frac{\partial {\widehat{h}}_{\ell }}{\partial U}& =-\frac{1}{n{\sigma }^2}\sum _{i=1}^n(U{x}_i-U{x}_{\ell }){(x_i-x_{\ell })}^{\top }exp\left(-\frac{\parallel U{x}_i-U{x}_{\ell }{\parallel }^2}{2{\sigma }^2}\right)L(y_i,y_{\ell }),\hfill \\ \hfill \frac{\partial {\widehat{H}}_{\ell ,{\ell }^{\prime }}}{\partial U}& =[-\frac{1}{n{\sigma }^2}\sum _{i=1}^n\left((U{x}_i-U{x}_{\ell }){(x_i-x_{\ell })}^{\top }+(U{x}_i-U{x}_{{\ell }^{\prime }}){(x_i-x_{{\ell }^{\prime }})}^{\top }\right)\hfill \\ & \phantom{=}\times exp\left(-\frac{\parallel U{x}_i-U{x}_{\ell }{\parallel }^2+{\parallel U{x}_i-U{x}_{{\ell }^{\prime }}\parallel }^2}{2{\sigma }^2}\right)]\times \left[\frac{1}{n}\sum _{i=1}^{n}L(y_i,y_{\ell })L(y_i,y_{{\ell }^{\prime }})\right]\hfill \end{array}$$

The projection of U onto the feasible region specified by $U{U}^{\top }=I_{d_{\mathrm{u}}}$ may be carried out by the Gram–Schmidt process [60], although this is time-consuming.

An alternative way to solve the LSDR optimization problem is to perform gradient ascent on the Grassmann manifold [61]. In the Euclidean space, the ordinary gradient gives the steepest direction. However, on a manifold, the natural gradient [62] gives the steepest direction. The natural gradient $\nabla \mathrm{LSMI}\left(U\right)$ at U is given as follows [63]:

$$\begin{array}{c}\hfill \nabla \mathrm{LSMI}\left(U\right)=\frac{\partial \mathrm{LSMI}}{\partial U}-\frac{\partial \mathrm{LSMI}}{\partial U}{U}^{\top }U=\frac{\partial \mathrm{LSMI}}{\partial U}{U}_{\perp }^{\top }{U}_{\perp }\end{array}$$

where $U_{\perp }$ is any $(d-m)\times d$ matrix such that $\left[U^{\top }{U}_{\perp }^{\top }\right]$ is orthogonal. Then the geodesic from U to the direction of the natural gradient $\nabla \mathrm{LSMI}\left(U\right)$ over the Grassmann manifold can be expressed using $t\in \mathbb{R}$ as:

$$U_t:=\left[\begin{array}{cc}{I}_{d_{\mathrm{x}}}& O_{d_{\mathrm{x}}-d_{\mathrm{u}}}\end{array}\right]exp\left(t\left[\begin{array}{cc}{O}_{d_{\mathrm{u}}}& \frac{\partial \mathrm{LSMI}}{\partial U}{U}_{\perp }^{\top }\\ -U_{\perp }{\frac{\partial \mathrm{LSMI}}{\partial U}}^{\top }& O_{d_{\mathrm{x}}-d_{\mathrm{u}}}\end{array}\right]\right)\left[\begin{array}{c}U\\ U_{\perp }\end{array}\right]$$

where “exp” for a matrix denotes the matrix exponential, and $O_{d_{\mathrm{x}}}$ is the $d_{\mathrm{x}}\times d_{\mathrm{x}}$ zero matrix. Thus, line search along the geodesic in the natural gradient direction is equivalent to finding the maximizer from $\{U_t\mid t\ge 0\}$. For choosing the step size of each gradient update, some approximate line search method such as Armijo’s rule [64] or backtracking line search [51] may be used.

A ${\text{MATLAB}}^{\text{®}}$ implementation of LSDR is publicly available [65].

3.3.4. Heuristic Subspace Search

Although the natural gradient method is computationally more efficient than the plain gradient method, it is still time consuming. Moreover, many restarts from different initial solutions may be needed for finding a good local optimum. Here, we introduce a heuristic method that can address these issues [29].

A key idea is to use a truncated negative quadratic function called the Epanechnikov kernel [66] as a kernel function for $Ux$:

$$\begin{array}{cc}\hfill K(Ux,U{x}^{\prime })& =max\left(0,1-\frac{\parallel Ux-U{x}^{\prime }{\parallel }^2}{2{\sigma }_{\mathrm{z}}^2}\right)\hfill \end{array}$$

Let $I\left(c\right)$ be the indicator function, i.e., $I\left(c\right)=1$ if c is true and zero otherwise. Then, for the above kernel function, $\mathrm{LSMI}$ can be expressed as:

$$\begin{array}{c}\hfill \mathrm{LSMI}=\frac{1}{2}\mathrm{tr}\left(U{D}_U{U}^{\top }\right)-\frac{1}{2}\end{array}$$

where $\mathrm{tr}(·)$ denotes the trace of a matrix and $D_U$ is the $d_{\mathrm{x}}\times d_{\mathrm{x}}$ matrix defined by:

$$\begin{array}{cc}\hfill D_U& =\frac{1}{n}\sum _{i=1}^n\sum _{\ell =1}^n{\widehat{\theta }}_{\ell }\left(U\right)I\left(\frac{\parallel U{x}_i-U{x}_{\ell }{\parallel }^2}{2{\sigma }_{\mathrm{z}}^2}<1\right)L(y_i,y_{\ell })\left[\frac{1}{d_{\mathrm{u}}}{I}_{d_{\mathrm{x}}}-\frac{1}{2{\sigma }_{\mathrm{z}}^2}(x_i-x_{\ell }){(x_i-x_{\ell })}^{\top }\right]\hfill \end{array}$$

Here, the fact that ${\widehat{\theta }}_{\ell }$ depends on U is explicitly indicated by ${\widehat{\theta }}_{\ell }\left(U\right)$.

If U in $D_U$ is replaced by $U^{\prime }$, where $U^{\prime }$ is a transformation matrix obtained in the previous iteration, the SMI estimator is simplified as:

$$\begin{array}{c}\hfill \frac{1}{2}\mathrm{tr}\left(U{D}_{U^{\prime }}{U}^{\top }\right)-\frac{1}{2}\end{array}$$

(16)

Because $D_{U^{\prime }}$ is independent of U, a maximizer of Equation (16) with respect to U can be analytically obtained by $(u_1|\cdots |u_{d_{\mathrm{u}}}{)}^{\top }$, where ${\left\{u_i\right\}}_{i=1}^{d_{\mathrm{u}}}$ are the $d_{\mathrm{u}}$ principal components of $D^{\prime }$. The same technique can also be utilized for determining an initial transformation matrix, by computing the above solution for $U^{\prime }=I_{d_{\mathrm{x}}}$ (i.e., no dimensionality reduction).

The above heuristic search method for LSDR is called sufficient component analysis (SCA) [29]. A MATLAB${}^{\text{®}}$ implementation of SCA is publicly available [67].

3.4. Canonical Dependency Analysis

Next, we show how the SMI estimator can be used for feature extraction from two sets of data.

3.4.1. Introduction

Canonical correlation analysis (CCA) [68] is a classical dimensionality reduction technique for two data sources, and it iteratively finds projection directions with maximum correlation. However, because CCA only captures correlations under linear projections, it is often insufficient to analyze complex real-world data that contain higher-order correlations. To be more flexible, non-linear CCA methods have been explored. A simple approach uses neural networks to handle non-linear projections [69,70], but neural networks are prone to local optima. Another approach first non-linearly transforms data samples into feature spaces and then apply linear CCA [71,72]. Given that the non-linear transformation is fixed, this two-step approach allows analytic computation of the global optimal solution via a generalized eigenvalue problem in the same way as linear CCA. This non-linear approach is called kernel CCA (KCCA) because reproducing kernels [38] are used as non-linear transforms. Alternating regression such as the alternating conditional expectation [73] is another possible way to find dependency in a flexible manner, which estimates transformations for two variables alternately by minimizing the squared error between transformed variables. These non-linear variants of CCA are highly flexible, although obtained results are often difficult to interpret due to the non-linearity.

The above non-linear CCA approaches can be regarded as capturing correlations along non-linear projection directions. Another extension of CCA called canonical dependency analysis (CDA) [30] captures higher-order correlations under linear projections. It was shown that KCCA with a universal kernel [45] such as the Gaussian kernel allows efficient detection of higher-order correlations [74]. However, the choice of universal kernels affects the practical performance, and there is no systematic method to choose a suitable kernel function. Another approach to higher-order CCA called informational CCA (ICCA) [75] uses mutual information (MI) as a dependency measure, where MI is estimated via kernel density estimation (KDE). Because systematic model selection strategies are available for KDE [76], ICCA could be more practical than the KCCA-based CDA method. In the ICCA method, one-dimensional projection directions are found in an iterative manner. Thus, it would be more powerful if multi-dimensional projection directions (i.e., a subspace) could be directly found in CDA [30]. However, ICCA may not be reliable in such a subspace search scenario because it involves the ratio of estimated densities, which tends to produce large estimation error if the dimensionality is not small.

To overcome the above limitation, an SMI-based CDA method called least-squares CDA (LSCDA) was proposed [30]. Below, we review LSCDA.

3.4.2. Canonical Dependency Analysis with SMI

Suppose that we are given n i.i.d. paired samples ${\left\{(x_i,y_i)\right|x_i\in {\mathbb{R}}^{d_{\mathrm{x}}},y_i\in {\mathbb{R}}^{d_{\mathrm{y}}}\}}_{i=1}^n$ drawn from a joint distribution with density $p(x,y)$. CDA is aimed at finding the low-dimensional expressions of x and y that are maximally dependent on each other. Here, we focus on linear dimension reduction, i.e., x and y are transformed as $Ux$ and $Vy$, where $U\in {\mathbb{R}}^{d_{\mathrm{u}}\times d_{\mathrm{x}}}$ and $V\in {\mathbb{R}}^{d_{\mathrm{v}}\times d_{\mathrm{y}}}$ are orthogonal matrices with known dimensionalities $d_{\mathrm{u}}$ and $d_{\mathrm{v}}$. The objective of CDA is to find the transformation matrices U and V such that the statistical dependency between $Ux$ and $Vy$ is maximized. Let us use the SMI estimator, $\mathrm{LSMI}\left({\left\{(U{x}_i,V{y}_i)\right\}}_{i=1}^n\right)$, as the dependency measure, i.e., we solve,

$$\begin{array}{cc}\hfill \underset{U\in {\mathbb{R}}^{d_{\mathrm{u}}\times d_{\mathrm{x}}},V\in {\mathbb{R}}^{d_{\mathrm{v}}\times d_{\mathrm{y}}}}{\mathrm{argmax}}& \mathrm{LSMI}\left({\left\{(U{x}_i,V{y}_i)\right\}}_{i=1}^n\right)\hfill \\ \hfill \mathrm{subject}\mathrm{to}& U{U}^{\top }=I_{d_{\mathrm{u}}}\mathrm{and}V{V}^{\top }=I_{d_{\mathrm{v}}}\hfill \end{array}$$

This formulation is called least-squares CDA (LSCDA) [30].

The above optimization problem can be solved in the same way as LSDR presented in Section 3.3.3 A MATLAB${}^{\text{®}}$ implementation of LSCDA is publicly available [77].

3.5. Independent Component Analysis

Here, we show how the SMI estimator can be used for independent component analysis.

3.5.1. Introduction

Suppose that there exist statistically independent sources of signals, and we observe their mixtures. The purpose of independent component analysis (ICA) [78] is to separate the mixed signals into the original source signals. An approach to ICA is to separate the mixed signals such that statistical independence among separated signals is maximized under some independence measure.

Various methods for evaluating the statistical independence among random variables from samples have been explored so far. A naive approach is to estimate probability densities based on parametric or non-parametric density estimation methods. However, finding an appropriate parametric model is not straightforward without strong prior knowledge and non-parametric estimation is not generally accurate in high-dimensional problems. Thus, this naive approach is not reliable in practice. Another approach is to approximate the entropy based on the Gram–Charlier expansion [79] or the Edgeworth expansion [80]. An advantage of this entropy-based approach is that a hard task of density estimation is not directly involved. However, these expansion techniques are based on the assumption that the target density is close to Gaussian, and violation of this assumption can cause large approximation error.

The above approaches are based on the probability densities of signals. Another line of research that does not explicitly involve probability densities employs non-linear correlation—signals are statistically independent if and only if all non-linear correlations among signals vanish. Following this line, computationally efficient algorithms have been developed based on a contrast function [81,82], which is an approximation of the entropy or mutual information. However, non-linearities in the contrast function need to be pre-specified in these methods, and thus they could be inaccurate if the predetermined non-linearities do not match the target distribution. To cope with this problem, the kernel trick has been applied in ICA, which allows computationally efficient evaluation of all non-linear correlations citeJMLR:Bach+Jordan:2002. However, its practical performance depends on the choice of kernels (more specifically, the Gaussian kernel width) and there seems no theoretically justified method to determine the kernel width. This is a critical problem in unsupervised learning tasks such as ICA.

To cope with this problem, an SMI-based ICA algorithm called least-squares independent component analysis (LICA) has been developed [31]. Below, we review LICA.

3.5.2. Independent Component Analysis with SMI

Suppose there are d signal sources and let: ${\left\{x_i\right|x_i={(x_i^{\left(1\right)},\dots ,x_i^{\left(d\right)})}^{\top }\in {\mathbb{R}}^d\}}_{i=1}^n$ be i.i.d. samples drawn from a distribution with density $p\left(x\right)$. We assume that elements $x^{\left(1\right)},\dots ,x^{\left(d\right)}$ are statistically independent of each other, i.e., $p\left(x\right)$ is factorized as:

$$\begin{array}{c}\hfill p\left(x\right)=p\left(x^{\left(1\right)}\right)\cdots p\left(x^{\left(d\right)}\right)\end{array}$$

We cannot directly observe ${\left\{x_i\right\}}_{i=1}^n$, but only their linearly mixed samples ${\left\{y_i\right\}}_{i=1}^n$:

$$\begin{array}{c}\hfill y_i:=U{x}_i\end{array}$$

where U is a $d\times d$ invertible matrix called the mixing matrix.

The goal of ICA is, from the mixed samples ${\left\{y_i\right\}}_{i=1}^n$, to obtain a demixing matrix V that recovers the original source samples ${\left\{x_i\right\}}_{i=1}^n$. We denote the demixed samples by ${\left\{z_i\right\}}_{i=1}^n$:

$$\begin{array}{c}\hfill z_i=V{y}_i\end{array}$$

The ideal solution is $V=U^{-1}$, but we can only recover the source signals up to permutation and scaling of components of x due to non-identifiability of the ICA setup [78]. Let us denote the demixed samples by:

$$\begin{array}{c}\hfill z_i={(z_i^{\left(1\right)},\dots ,z_i^{\left(d\right)})}^{\top }:=V{y}_i\end{array}$$

for $i=1,\dots ,n$.

A direct approach to ICA is to determine V so that elements of z are as statistically independent as possible. Here, we adopt SMI as the independence measure:

$$\begin{array}{cc}\hfill \mathrm{SMI}(Z^{\left(1\right)},\dots ,Z^{\left(d\right)})& :=\frac{1}{2}\int \cdots \int p\left(z^{\left(1\right)}\right)\cdots p\left(z^{\left(d\right)}\right){\left(\frac{p(z^{\left(1\right)},\dots ,z^{\left(d\right)})}{p\left(z^{\left(1\right)}\right)\cdots p\left(z^{\left(d\right)}\right)}-1\right)}^2\mathrm{d}{z}^{\left(1\right)}\cdots \mathrm{d}{z}^{\left(d\right)}\hfill \end{array}$$

We try to find the demixing matrix V that minimizes SMI. In practice, the following optimization problem is solved:

$$\begin{array}{cc}\hfill \underset{V\in {\mathbb{R}}^{d\times d}}{min}& \mathrm{LSMI}\left({\left\{V{y}_i\right\}}_{i=1}^n\right)\hfill \end{array}$$

where $\mathrm{LSMI}\left({\left\{V{y}_i\right\}}_{i=1}^n\right)$ is given by the same form as Equation (12) (or Equation (13)), but the matrix $\widehat{H}$ and the vector $\widehat{h}$ are defined in a slightly different way. For the Gaussian kernel,

$$\begin{array}{c}\hfill K(Vy,V{y}^{\prime })=exp\left(-\frac{\parallel Vy-V{y}^{\prime }{\parallel }^2}{2{\sigma }^2}\right)\end{array}$$

$\widehat{H}$ and $\widehat{h}$ are given by:

$$\begin{array}{cc}\hfill {\widehat{H}}_{\ell ,{\ell }^{\prime }}& =\frac{1}{n^d}\prod _{m=1}^d\left[\sum _{i=1}^{n}exp\left(-\frac{{(z_{\ell }^{\left(m\right)}-z_i^{\left(m\right)})}^2+{(z_{{\ell }^{\prime }}^{\left(m\right)}-z_i^{\left(m\right)})}^2}{2{\sigma }^2}\right)\right]\hfill \\ \hfill {\widehat{h}}_{\ell }& =\frac{1}{n}\sum _{i=1}^{n}exp\left(-\frac{\parallel z_i-z_{\ell }{\parallel }^2}{2{\sigma }^2}\right)\hfill \end{array}$$

This formulation is called least-squares independent component analysis (LICA) [31].

3.5.3. Gradient-Based Demixing Matrix Search

Based on the plain gradient technique, an update rule of V is given by:

$$V⟵V-t\frac{\partial \mathrm{LSMI}}{\partial V}$$

(17)

where t$(>0)$ is the step size. The gradient $\frac{\partial \mathrm{LSMI}}{\partial V}$ is given by:

$$\begin{array}{cc}\hfill \frac{\partial \mathrm{LSMI}}{\partial V}& =\sum _{\ell =1}^n{\widehat{\theta }}_{\ell }\frac{\partial {\widehat{h}}_{\ell }}{\partial V}-\frac{1}{2}\sum _{\ell ,{\ell }^{\prime }=1}^n{\widehat{\theta }}_{\ell }{\widehat{\theta }}_{{\ell }^{\prime }}\frac{\partial {\widehat{H}}_{\ell ,{\ell }^{\prime }}}{\partial V}\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill \frac{\partial {\widehat{h}}_{\ell }}{\partial V_{k,k^{\prime }}}& =-\frac{1}{n{\sigma }^2}\sum _{i=1}^n(z_i^{\left(k\right)}-z_{\ell }^{\left(k\right)}){(y_i^{\left(k^{\prime }\right)}-y_{\ell }^{\left(k^{\prime }\right)})}^{\top }exp\left(-\frac{\parallel z_i-z_k{\parallel }^2}{2{\sigma }^2}\right)\hfill \\ \hfill \frac{\partial {\widehat{H}}_{\ell ,{\ell }^{\prime }}}{\partial V_{k,k^{\prime }}}& =\frac{1}{n^{d-1}}\prod _{m\ne k}\left[\sum _{i=1}^{n}exp\left(-\frac{{(z_i^{\left(m\right)}-z_{\ell }^{\left(m\right)})}^2+{(z_i^{\left(m\right)}-z_{{\ell }^{\prime }}^{\left(m\right)})}^2}{2{\sigma }^2}\right)\right]\hfill \\ & \phantom{=}\times [-\frac{1}{n{\sigma }^2}\sum _{i=1}^n\left((z_i^{\left(k\right)}-z_{\ell }^{\left(k\right)})(y_i^{\left(k^{\prime }\right)}-y_{\ell }^{\left(k^{\prime }\right)})+(z_i^{\left(k\right)}-z_{{\ell }^{\prime }}^{\left(k\right)})(y_i^{\left(k^{\prime }\right)}-y_{{\ell }^{\prime }}^{\left(k^{\prime }\right)})\right)\hfill \\ & \times exp\left(-\frac{{(z_i^{\left(k\right)}-v_{\ell }^{\left(k\right)})}^2+{(z_i^{\left(k\right)}-z_{{\ell }^{\prime }}^{\left(k\right)})}^2}{2{\sigma }^2}\right)]\hfill \end{array}$$

In ICA, scaling of components of z can be arbitrary. This implies that the above gradient updating rule can lead to a solution with poor scaling, which is not preferable from a numerical viewpoint. To avoid possible numerical instability, V is normalized at each gradient iteration as:

$$\begin{array}{c}\hfill V_{k,k^{\prime }}⟵\frac{V_{k,k^{\prime }}}{\sqrt{{\sum }_{m=1}^d{V}_{k,m}^2}}\end{array}$$

3.5.4. Natural Gradient Demixing Matrix Search

Suppose that data samples are whitened, i.e., samples ${\left\{y_i\right\}}_{i=1}^n$ are pre-transformed as:

$$\begin{array}{c}\hfill y_i⟵{\widehat{\Sigma }}^{-\frac{1}{2}}{y}_i\end{array}$$

where $\widehat{\Sigma }$ is the sample covariance matrix:

$$\begin{array}{c}\hfill \widehat{\Sigma }:=\frac{1}{n}\sum _{i=1}^n\left(y_i-\frac{1}{n}\sum _{j=1}^n{y}_j\right){\left(y_i-\frac{1}{n}\sum _{j=1}^n{y}_j\right)}^{\top }\end{array}$$

Then it can be shown that any demixing matrix that eliminates the second order correlation is an orthogonal matrix [78]. Thus, for whitened data, the search space of V can be restricted to the orthogonal group without loss of generality. The natural gradient [62] update rule on the orthogonal group is given by:

$$\begin{array}{c}\hfill V⟵Vexp\left(-t\left(V^{\top }\frac{\partial \mathrm{LSMI}}{\partial V}-{\frac{\partial \mathrm{LSMI}}{\partial V}}^{\top }V\right)\right)\end{array}$$

where “exp” for a matrix denotes the matrix exponential and t $(>0)$ is the step size.

A MATLAB${}^{\text{®}}$ implementation of LICA is publicly available [83].

3.6. Cross-Domain Object Matching

Next, we show how the SMI estimator can be used for cross-domain object matching.

3.6.1. Introduction

The objective of cross-domain object matching is to match two sets of unpaired objects in different domains. For example, in photo album summarization, we are given a set of photos and a designed photo frame expressed as a set of photo slots in the Cartesian coordinate system, and we want to automatically assign the photos into the designed photo frame. A typical approach of cross-domain object matching is to find a mapping from objects in one domain (photos) to objects in the other domain (frame) so that the pairwise dependency is maximized. In this scenario, accurately evaluating the dependence between objects is a key issue.

Kernelized sorting [84] tries to find the mapping between two domains that maximizes mutual information under the Gaussian assumption. However, because the Gaussian assumption may not be fulfilled in practice, this method tends to perform poorly. To overcome the above limitation, the kernel-based dependence measure called the Hilbert–Schmidt independence criterion (HSIC) [85] was proposed to use in kernelized sorting [86]. Because HSIC is distribution-free, HSIC-based kernelized sorting is more flexible than the original method based on the Gaussian assumption. However, HSIC includes a tuning parameter (more specifically, the Gaussian kernel width), and its choice is crucial to obtain better performance [87].

To cope with this problem, an SMI-based cross-domain object matching method called least-squares object matching (LSOM) was developed [32]. Below, we review LSOM.

3.6.2. Cross-Domain Object Matching with SMI

The goal of cross-domain object matching is, given two sets of unpaired samples of the same size, ${\left\{x_i\right|x_i\in \mathcal{X}\}}_{i=1}^n$ and ${\left\{y_i\right|y_i\in \mathcal{Y}\}}_{i=1}^n$, to find a mapping that well “matches” them. Let π be a permutation function over $\{1,\dots ,n\}$. The optimal permutation, denoted by ${\pi }^{*}$, can be obtained as the maximizer of the dependency between the two sets ${\left\{x_i\right\}}_{i=1}^n$ and ${\left\{y_{\pi \left(i\right)}\right\}}_{i=1}^n$. Here, we use the SMI approximator, $\mathrm{LSMI}\left({\left\{(x_i,y_{\pi \left(i\right)})\right\}}_{i=1}^n\right)$, as the dependency measure, i.e., we solve,

$$\begin{array}{c}\hfill \underset{\pi }{max}\mathrm{LSMI}\left({\left\{(x_i,y_{\pi \left(i\right)})\right\}}_{i=1}^n\right)\end{array}$$

Let K and L be the $n\times n$ kernel matrices defined by $K_{i,j}=K(x_i,x_j)$ and $L_{i,j}=L(y_i,y_j)$. Then LSMI for ${\left\{(x_i,y_{\pi \left(i\right)})\right\}}_{i=1}^n$ can be expressed as:

$$\begin{array}{c}\hfill \mathrm{LSMI}\left({\left\{(x_i,y_{\pi \left(i\right)})\right\}}_{i=1}^n\right)=\frac{1}{2n}\mathrm{tr}\left({\Pi }^{\top }L\Pi {\widehat{\Theta }}_{\Pi }K\right)-\frac{1}{2}\end{array}$$

(18)

where Π is the permutation matrix corresponding to π, i.e., Π is the $n\times n$ zero-one matrix such that ${\Pi }_{i,j}=1$ if $i=\pi \left(j\right)$ for $j=1,\dots ,n$ and ${\Pi }_{i,j}=0$ otherwise. ${\widehat{\Theta }}_{\Pi }$ is the diagonal matrix with diagonal elements given by the LSMI solution ${\widehat{\theta }}_{\pi }$ obtained by paired data ${\left\{(x_i,y_{\pi \left(i\right)})\right\}}_{i=1}^n$ (see Equation (11)).

Because maximizing Equation (18) with respect to Π is computationally infeasible, greedy update from previous solution ${\Pi }^{\prime }$ is used in practice:

$$\begin{array}{c}\hfill {\Pi }^{\mathrm{new}}=(1-t){\Pi }^{\prime }+t·\underset{\Pi }{\mathrm{argmax}}\mathrm{tr}\left({\Pi }^{\top }L{\Pi }^{\prime }{\widehat{\Theta }}_{{\Pi }^{\prime }}K\right)\end{array}$$

where $0<t\le 1$ is the step size. Maximization of the second term is called a linear assignment problem, which can be solved efficiently by the Hungarian method [88].

The above method is called least-squares object matching (LSOM) [32]. A MATLAB${}^{\text{®}}$ implementation of LSOM is publicly available [89].

3.7. Clustering

Here, we show how SMI can be effectively used for clustering.

3.7.1. Introduction

The objective of clustering is to classify data samples into disjoint groups in an unsupervised manner. K-means [90] is a classic but still popular clustering algorithm. However, k-means only produces linearly separated clusters, and thus its usefulness is rather limited in practice. To cope with this problem, various non-linear clustering methods have been developed. Kernel k-means [91] performs k-means in a feature space induced by a reproducing kernel function [46]. Spectral clustering [92,93] first unfolds non-linear data manifolds by a spectral embedding method, and then performs k-means in the embedded space. Blurring mean-shift [94,95] uses a non-parametric kernel density estimator for modeling the data-generating probability density, and finds clusters based on the modes of the estimated density. Discriminative clustering learns a discriminative classifier for separating clusters, where class labels are also treated as parameters to be optimized [96,97]. Dependence-maximization clustering determines cluster assignments so that their dependence on input data is maximized [34,98,99].

Information-maximization clustering exhibited the state-of-the-art performance [100,101], where probabilistic classifiers such as a kernelized Gaussian classifier [100] and a kernel logistic regression classifier [101] are learned so that mutual information between feature vectors and cluster assignments is maximized in an unsupervised manner. A notable advantage of information-maximization clustering is that classifier training is formulated as continuous optimization, which is substantially simpler than discrete optimization of cluster assignments. Indeed, classifier training can be carried out in computationally efficient manners by a gradient method [100] or a quasi-Newton method [101]. Furthermore, a model selection strategy based on the information-maximization principle is also provided [100]. Thus, kernel parameters can be systematically optimized in an unsupervised way. However, the optimization problems of these clustering methods are non-convex and finding a good local optimal solution is not straightforward in practice.

To overcome the above limitation, an SMI-based clustering method called SMI clustering (SMIC) was proposed [33]. Below, we review SMIC.

3.7.2. Clustering with SMI

Suppose that we are given d-dimensional i.i.d. feature vectors of size n, ${\left\{x_i\right|x_i\in {\mathbb{R}}^d\}}_{i=1}^n$, which are drawn independently from a distribution with density $p\left(x\right)$. The goal of clustering is to give cluster assignments, ${\left\{y_i\right|y_i\in \{1,\dots ,c\}\}}_{i=1}^n$, to the feature vectors ${\left\{x_i\right\}}_{i=1}^n$, where c denotes the number of clusters. c is assumed to be pre-fixed below. To solve the clustering problem, the information-maximization approach is taken [100,101]. That is, clustering is regarded as an unsupervised classification problem, and the class-posterior probability $p\left(y\right|x)$ is learned so that “information” between feature vector x and cluster label y is maximized.

As an information measure, SMI Equation (1) is adopted, which can expressed as:

$$\begin{array}{cc}\hfill \mathrm{SMI}& =\frac{1}{2}\int \sum _{y=1}^{c}p\left(y\right|x)p\left(x\right)\frac{p\left(y\right|x)}{p\left(y\right)}\mathrm{d}x-\frac{1}{2}\hfill \end{array}$$

(19)

Suppose that the class-prior probability $p\left(y\right)$ is set to a user-specified value ${\pi }_y$ for $y=1,\dots ,c$, where ${\pi }_y>0$ and ${\sum }_{y=1}^c{\pi }_y=1$. Without loss of generality, ${\left\{{\pi }_y\right\}}_{y=1}^c$ are assumed to be sorted in the ascending order:

$$\begin{array}{c}\hfill {\pi }_1\le \cdots \le {\pi }_c\end{array}$$

If ${\left\{{\pi }_y\right\}}_{y=1}^c$ is unknown, the uniform class-prior distribution may be adopted:

$$\begin{array}{c}\hfill p\left(y\right)=\frac{1}{c}fory=1,\dots ,c\end{array}$$

Substituting ${\pi }_y$ into $p\left(y\right)$, we can express Equation (19) as:

$$\begin{array}{c}\hfill \frac{1}{2}\int \sum _{y=1}^c\frac{1}{{\pi }_y}p\left(y\right|x)p\left(x\right)p\left(y\right|x)\mathrm{d}x-\frac{1}{2}\end{array}$$

(20)

Let us approximate the class-posterior probability $p\left(y\right|x)$ by the following kernel model:

$$\begin{array}{cc}\hfill q_{\alpha }\left(y\right|x)& :=\sum _{i=1}^n{\alpha }_{y,i}{K}_{}(x,x_i),\hfill \end{array}$$

(21)

where $\alpha ={({\alpha }_{1,1},\dots ,{\alpha }_{c,n})}^{\top }$ is the parameter vector and $K_{}(x,x^{\prime })$ denotes a kernel function. A useful example of kernel functions is the local-scaling kernel [102] defined as:

$$\begin{array}{c}\hfill K_{}(x_i,x_j)=\left\{\begin{array}{cc}{\displaystyle exp\left(-\frac{\parallel x_i-x_j{\parallel }^2}{2{\sigma }_i{\sigma }_j}\right)}\hfill & \text{if }{x}_i\in {\mathcal{N}}_k\left(x_j\right)\text{ or }{x}_j\in {\mathcal{N}}_k\left(x_i\right)\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

where ${\mathcal{N}}_k\left(x\right)$ denotes the set of k nearest neighbors for x (k is the kernel parameter), ${\sigma }_i$ is a local scaling factor defined as ${\sigma }_i=\parallel x_i-x_i^{\left(k\right)}\parallel$, and $x_i^{\left(k\right)}$ is the k-th nearest neighbor of $x_i$. Note that we did not include the normalization term in Equation (21) because model outputs will be normalized later (see Equation (22)).

Further approximating the expectation with respect to $p\left(x\right)$ included in Equation (20) by the empirical average of samples ${\left\{x_i\right\}}_{i=1}^n$, we arrive at the following SMI approximator:

$$\begin{array}{cc}\hfill \widehat{\mathrm{SMI}}& :=\frac{1}{2n}\sum _{y=1}^c\frac{1}{{\pi }_y}{\alpha }_y^{\top }{K}^2{\alpha }_y-\frac{1}{2}\hfill \end{array}$$

where ${\alpha }_y:={({\alpha }_{y,1},\dots ,{\alpha }_{y,n})}^{\top }$ and $K_{i,j}:=K_{}(x_i,x_j)$.

For each cluster y, ${\alpha }_y^{\top }{K}^2{\alpha }_y$ is maximized under $\parallel {\alpha }_y\parallel =1$. Since this is the Rayleigh quotient, the maximizer is given by the normalized principal eigenvector of K [104]. To avoid all the solutions ${\left\{{\alpha }_y\right\}}_{y=1}^c$ to be reduced to the same principal eigenvector, their mutual orthogonality is imposed:

$$\begin{array}{c}\hfill {\alpha }_y^{\top }{\alpha }_{y^{\prime }}=0\text{for }y\ne y^{\prime }\end{array}$$

Then the solutions are given by the normalized eigenvectors ${\psi }_1,\dots ,{\psi }_c$ associated with the eigenvalues ${\lambda }_1\ge \cdots \ge {\lambda }_n\ge 0$ of K. Since the sign of ${\psi }_y$ is arbitrary, the sign is set as:

$$\begin{array}{c}\hfill {\tilde{\psi }}_y={\psi }_y\times \mathrm{sign}\left({\psi }_y^{\top }{1}_n\right)\end{array}$$

where $\mathrm{sign}(·)$ denotes the sign of a scalar and $1_n$ denotes the n-dimensional vector with all ones.

On the other hand, because

$$\begin{array}{c}\hfill p\left(y\right)=\int p\left(y\right|x)p\left(x\right)\mathrm{d}x\approx \frac{1}{n}\sum _{i=1}^n{q}_{\alpha }\left(y\right|x_i)={\alpha }_y^{\top }K{1}_n\end{array}$$

and the class-prior probability $p\left(y\right)$ was set to ${\pi }_y$ for $y=1,\dots ,c$, the following normalization condition is obtained:

$$\begin{array}{c}\hfill {\alpha }_y^{\top }K{1}_n={\pi }_y\end{array}$$

(22)

Furthermore, probability estimates should be non-negative, which can be achieved by rounding up negative outputs to zero.

By taking these normalization and non-negativity issues into account, cluster assignment $y_i$ for $x_i$ is determined as the maximizer of the approximation of $p\left(y\right|x_i)$:

$$\begin{array}{cc}\hfill y_i& =\underset{y}{\mathrm{argmax}}\frac{{[max(0_n,K{\tilde{\psi }}_y)]}_i}{{\pi }_y^{-1}max{(0_n,K{\tilde{\psi }}_y)}^{\top }{1}_n}=\underset{y}{\mathrm{argmax}}\frac{{\pi }_y{[max(0_n,{\tilde{\psi }}_y)]}_i}{max{(0_n,{\tilde{\psi }}_y)}^{\top }{1}_n}\hfill \end{array}$$

where the “max” operation for vectors is applied in the element-wise manner and ${[·]}_i$ denotes the i-th element of a vector. Note that $K{\tilde{\psi }}_y={\lambda }_y{\tilde{\psi }}_y$ was used in the above derivation. For out-of-sample prediction, cluster assignment $y^{\prime }$ for new sample $x^{\prime }$ may be obtained as:

$$\begin{array}{cc}\hfill y^{\prime }& :=\underset{y}{\mathrm{argmax}}\frac{{\pi }_{y}max\left(0,{\sum }_{i=1}^{n}K(x^{\prime },x_i){\left[{\tilde{\psi }}_y\right]}_i\right)}{{\lambda }_{y}max{(0_n,{\tilde{\psi }}_y)}^{\top }{1}_n}\hfill \end{array}$$

The above method is called SMI-based clustering (SMIC) [33]. LSMI can be used for model selection of SMIC, i.e., LSMI is computed as a function of the kernel parameter included in $K_{}(x,x^{\prime })$ and the maximizer of LSMI is chosen as the most promising one. A MATLAB${}^{\text{®}}$ implementation of SMIC is publicly available [103].

3.8. Causal Direction Estimation

Finally, we show how the SMI estimator can be used for causal direction estimation.

3.8.1. Introduction

Learning causality from data is one of the important challenges in the artificial intelligence, statistics, and machine learning communities [105]. A traditional method of learning causal relationship from observational data is based on the linear-dependence Gaussian-noise model [106]. However, the linear-Gaussian assumption is too restrictive and may not be fulfilled in practice. Recently, non-Gaussianity and non-linearity have been shown to be beneficial in causal inference, because it can break symmetry between observed variables [107,108]. Since then, much attention has been paid to the discovery of non-linear causal relationship through non-Gaussian noise models [109].

In the framework of non-linear non-Gaussian causal inference, the relation between a cause X and an effect Y is assumed to be described by $Y=f\left(X\right)+E$, where f is a non-linear function and E is non-Gaussian additive noise that is independent of the cause X. Under this additive noise assumption, it was shown [108] that the causal direction between X and Y can be identified based on a hypothesis test of whether the causal model $Y=f\left(X\right)+E$ or the alternative model $X=f^{\prime }\left(Y\right)+E^{\prime }$ fits the data well—here, the goodness of fit is measured by independence between inputs and residuals (i.e., estimated noise). In [108], the functions f and $f^{\prime }$ were learned by the Gaussian process (GP) regression [110], and the independence between inputs and residuals was evaluated by the Hilbert–Schmidt independence criterion (HSIC) [85].

However, standard regression methods such as GP are designed to handle Gaussian noise, and thus they may not be suited for discovering causality in the non-Gaussian additive noise formulation. To cope with this problem, an alternative regression method called HSIC regression was proposed [109], which learns a function so that the dependence between inputs and residuals is directly minimized based on HSIC. Through experiments, HSIC regression was shown to outperform the GP-based method [109]. However, the choice of the kernel width in HSIC regression heavily affects the sensitivity of the independence measure, and systematic model selection strategies are not available. Another weakness of HSIC regression is that the kernel width of the regression model is fixed to the same value as HSIC. This crucially limits the flexibility of function approximation in HSIC regression.

To overcome the above weaknesses, an SMI-based regression method for causal inference called least-squares independence regression (LSIR) was developed [35]. Below, we review LSIR.

3.8.2. Dependence Minimizing Regression with SMI

Suppose random variables $X\in \mathbb{R}$ and $Y\in \mathbb{R}$ are connected by the following additive noise model [108]:

$$\begin{array}{c}\hfill Y=f\left(X\right)+E\end{array}$$

where $f:\mathbb{R}\to \mathbb{R}$ is some non-linear function and $E\in \mathbb{R}$ is a zero-mean random variable that is independent of X. The goal of dependence minimizing regression is, from i.i.d. paired samples ${\left\{(x_i,y_i)\right\}}_{i=1}^n$, to obtain a function $\widehat{f}$ such that input X and estimated additive noise $\widehat{E}=Y-\widehat{f}\left(X\right)$ are independent.

Let us employ a linear model for dependence minimizing regression:

$$\begin{array}{c}\hfill f_{\beta }\left(x\right)=\sum _{l=1}^m{\beta }_l{\psi }_l\left(x\right)={\beta }^{\top }\psi \left(x\right)\end{array}$$

where m is the number of basis functions, $\beta ={({\beta }_1,\dots ,{\beta }_m)}^{\top }$ are regression parameters, and $\psi \left(x\right)={({\psi }_1\left(x\right),\dots ,{\psi }_m\left(x\right))}^{\top }$ are basis functions. In LSMI-based dependence minimization regression, the regression parameters β are learned as:

$$\begin{array}{c}\hfill \underset{\beta }{min}\left[\mathrm{LSMI}\left({\left\{(x_i,e_i)\right\}}_{i=1}^n\right)+\frac{\gamma }{2}{\beta }^{\top }\beta \right]\end{array}$$

where $e_i=y_i-f_{\beta }\left(x_i\right)$ is the residual and $\gamma >0$ is the regularization parameter to avoid overfitting.

For regression parameter learning, a gradient descent method may be used:

$$\begin{array}{c}\hfill \beta ⟵\beta -t\left(\frac{\partial \mathrm{LSMI}}{\partial \beta }+\gamma \beta \right)\end{array}$$

where t is the step size. The gradient $\frac{\partial \mathrm{LSMI}}{\partial \beta }$ can be approximately expressed as:

$$\begin{array}{c}\hfill \frac{\partial \mathrm{LSMI}}{\partial \beta }=\sum _{\ell =1}^n{\widehat{\theta }}_{\ell }\frac{\partial {\widehat{h}}_{\ell }}{\partial \beta }-\frac{1}{2}\sum _{\ell ,{\ell }^{\prime }=1}^n{\widehat{\theta }}_{\ell }{\widehat{\theta }}_{{\ell }^{\prime }}\frac{\partial {\widehat{H}}_{\ell ,{\ell }^{\prime }}}{\partial \beta }\end{array}$$

where

$$\begin{array}{cc}\hfill \frac{\partial {\widehat{h}}_{\ell }}{\partial \beta }& \approx -\frac{1}{2n{\sigma }^2}\sum _{j=1}^{n}exp\left(-\frac{{(x_i-x_{\ell })}^2+{(e_i-e_{\ell })}^2}{2{\sigma }^2}\right)(e_i-e_{\ell })\psi \left(x_i\right),\hfill \\ \hfill \frac{\partial {\widehat{H}}_{\ell ,{\ell }^{\prime }}}{\partial \beta }& \approx -\frac{1}{2n^2{\sigma }^2}\sum _{i,j=1}^{n}exp\left(-\frac{{(x_i-x_{\ell })}^2+{(e_j-e_{\ell })}^2+{(x_i-x_{{\ell }^{\prime }})}^2+{(e_j-e_{{\ell }^{\prime }})}^2}{2{\sigma }^2}\right)\hfill \\ & \times \left((e_j-e_{\ell })\psi \left(x_i\right)+(e_i-e_{\ell })\psi \left(x_j\right)\right)\hfill \end{array}$$

Note that, in the above derivation, the dependence of β on $e_i$ is ignored for simplicity. Although it is possible to exactly compute the derivative in principle, this approximated expression is computationally more efficient with good performance in practice.

By taking into account the assumption that the mean of noise E is zero, the final regressor is obtained as:

$$\begin{array}{c}\hfill \widehat{f}\left(x\right)=f_{\widehat{\beta }}\left(x\right)+\frac{1}{n}\sum _{i=1}^n\left(y_i-f_{\widehat{\beta }}\left(x_i\right)\right)\end{array}$$

This method is called least-squares independence regression (LSIR) [35]. A MATLAB^® implementation of LSIR is publicly available [111].

3.8.3. Causal Direction Inference by LSIR

Our final goal is, given i.i.d. paired samples ${\left\{(x_i,y_i)\right\}}_{i=1}^n$, to determine whether X causes Y or vice versa under the additive noise assumption. To this end, we test whether the causal model $Y=f_Y\left(X\right)+E_Y$ or the alternative model $X=f_X\left(Y\right)+E_X$ fits the data well, where the goodness of fit is measured by independence between inputs and residuals (i.e., estimated noise). Independence of inputs and residuals may be decided in practice based on the permutation test procedure [47].

More specifically, LSIR is first run for ${\left\{(x_i,y_i)\right\}}_{i=1}^n$ as usual, and obtain a regression function $\widehat{f}$. This procedure also provides an SMI estimate, $\mathrm{LSMI}\left({\left\{(x_i,{\widehat{e}}_i)\right\}}_{i=1}^n\right)$, where ${\widehat{e}}_i=y_i-\widehat{f}\left(x_i\right)$. Next, pairs of input and residual ${\left\{(x_i,{\widehat{e}}_i)\right\}}_{i=1}^n$ are randomly permuted as ${\left\{(x_i,{\widehat{e}}_{\pi \left(i\right)})\right\}}_{i=1}^n$, where $\pi (·)$ is a randomly generated permutation function. Note that the permuted pairs of samples are independent of each other because the random permutation breaks the dependency between X and $\widehat{E}$ (if it exists). Then, an SMI estimate for the permuted data, $\mathrm{LSMI}\left({\left\{(x_i,{\widehat{e}}_{\pi \left(i\right)})\right\}}_{i=1}^n\right)$, is computed. This random permutation process is repeated many times, and the distribution of LSMI values under the null-hypothesis that X and $\widehat{E}$ are independent is constructed. Finally, the p-value is approximated by evaluating the relative ranking of LSMI computed from the original input-residual data, $\mathrm{LSMI}\left({\left\{(x_i,{\widehat{e}}_i)\right\}}_{i=1}^n\right)$, over the distribution of LSMI values for randomly permuted data.

In order to decide the causal direction, the p-values $p_{X\to Y}$ and $p_{X\leftarrow Y}$ for both directions $X\to Y$ (i.e., X causes Y) and $X\leftarrow Y$ (i.e., Y causes X) are computed. Then, for a given significance level δ, the causal direction is determined as follows:

• If $p_{X\to Y}>\delta$ and $p_{X\leftarrow Y}\le \delta$, the causal model $X\to Y$ is chosen.
• If $p_{X\leftarrow Y}>\delta$ and $p_{X\to Y}\le \delta$, the causal model $X\leftarrow Y$ is selected.
• If $p_{X\to Y},p_{X\leftarrow Y}\le \delta$, perhaps there is no causal relation between X and Y or our modeling assumption is not correct (e.g., an unobserved confounding variable exists).
• If $p_{X\to Y},p_{X\leftarrow Y}>\delta$, perhaps our modeling assumption is not correct or it is not possible to identify a causal direction (i.e., X, Y, and E are Gaussian random variables).

When we have prior knowledge that there exists a causal relation between X and Y but the causal direction is unknown, the values of $p_{X\to Y}$ and $p_{X\leftarrow Y}$ may be simply compared for determining the causal direction as follows:

• If $p_{X\to Y}>p_{X\leftarrow Y}$, we conclude that X causes Y.
• Otherwise, we conclude that Y causes X.

This simplified procedure does not include the computational expensive permutation process and thus it is computationally very efficient.

4. Conclusions

In this article, we reviewed recent development in the estimation of squared-loss mutual information (SMI) and its application to machine learning. The key idea for accurately estimating SMI is to directly estimate the ratio of probability densities without separately estimating each density. A notable advantage of the SMI estimator called least-squares mutual information (LSMI) [19] is that it can be computed analytically in a computationally more efficient and numerically more stable way than ordinary MI.

We have introduced SMI as a measure of statistical independence between random variables. On the other hand, ordinary MI has a rich information-theoretic interpretation via entropies. Thus, it is important to investigate an information-theoretic meaning of SMI, which remains to be an open question currently.

Various methods of direct density-ratio estimation have been explored so far [16,18], and such density ratio estimators were shown to be applicable to an even wider class of machine learning tasks beyond SMI estimation, such as non-stationarity adaptation [112], outlier detection [113], change detection [114,115], class-balance estimation [116], two-sample homogeneity testing [117,118], probabilistic classification [119,120], and conditional density estimation [121].

Improving the accuracy of density ratio estimation contributes to enhancing the performance of the above machine learning solutions. Recent advances in this line of research include dimensionality reduction for density ratio estimation [122,123,124], a unified statistical framework of density ratio estimation [18], and extensions to relative density ratios [125] and density differences [126]. Further improving the accuracy and computational efficiency and exploring new application areas are important future directions to pursue.

More program codes are publicly available [127].