A Gaussian Process Decoder with Spectral Mixtures and a Locally Estimated Manifold for Data Visualization

Abstract

Dimensionality reduction plays an important role in interpreting and visualizing high-dimensional data. Previous methods for data visualization overestimate the local structure and lack the consideration of global preservation. In this study, we develop a Gaussian process latent variable model (GP-LVM) for data visualization. GP-LVMs are one of the frameworks of principal component analysis and preserve the global structure effectively. The drawbacks of GP-LVMs are the absence of local structure preservation and the use of low-expressive kernel functions. Therefore, we introduce regularization for local preservation and an expressive kernel function into GP-LVMs to overcome these limitations. As a result, we reflect the global and local structures in low-dimensional representations, improving the reliability and visibility of embeddings. We conduct qualitative and quantitative experiments comparing baselines and state-of-the-art methods on image and text datasets.

1. Introduction

Real-world data generally contain high-dimensional structures with nonlinear correlation. Under this condition, it is required to identify a low-dimensional representation and to present a visible expression of data beneficial for understanding data structures. Dimensionality reduction [1] is one of the techniques for achieving this and it searches low-dimensional manifolds that preserve the original structures as much as possible. Principal component analysis (PCA) [2,3] is a classical approach to reduce dimensionality; however, PCA is a linear approach and has limitations in expressing the derived low-dimensional representation.

There are several nonlinear techniques for dimensionality reduction. Deep generative models (DGMs) [4], such as variational autoencoders [5,6] and diffusion models [7,8], have achieved remarkable success. DGMs are based on probabilistic modeling and are parameterized by deep neural networks [9], enabling them to obtain highly expressive low-dimensional structures. Although DGMs provide a strong representation, they necessitate the selection of many hyperparameters and network architectures and are typically overkill for data preprocessing and visualization. Graph-based approaches, such as t-distributed stochastic embedding (t-SNE) [10,11] or uniform manifold approximation and projection (UMAP) [12], are visualization-aided dimensionality reduction techniques frequently used for biological data visualization [13,14]. Graph-based approaches use few hyperparameters and have a high scalability; however, they overestimate local structures and tend to induce a misleading visualization overlooking global structures. Gaussian process latent variable models (GP-LVMs) [15,16] are another framework for dimensionality reduction and the extension of the classical PCA. The Gaussian process has similar properties to deep neural networks [17,18] and has fewer hyperparameters than DGMs. From the above, GP-LVMs are suitable for data preprocessing and visualization and have gained significant attention for dimensionality reduction as a counterpart to graph-based approaches [19,20,21,22,23].

The kernel function and prior distribution used in GP-LVMs influence the quality of low-dimensional representations. The kernel function determines the properties of the embedding space, and the prior distribution corresponds to the regularization term of low-dimensional embedding. Previously, researchers have proposed prior distributions for various purposes, such as time-series analyses [24,25], WiFi localization [26,27], and face verification [28]. Furthermore, many kernel functions have been developed in the regression literature [29,30,31] and have achieved many successful results. However, in GP-LVMs, the prior distribution is used to express supervised information [28,32,33] and is not designed for unsupervised cases typical in data visualization. Furthermore, the kernel functions used in GP-LVMs are limited to the class of homogeneous kernels, such as the squared exponential (SE) kernel and Matérn families, which have limited expressions. Therefore, we require a comprehensive approach to GP-LVMs for data visualization purposes.

In this study, we develop a GP-LVM for data visualization. Our main contribution is the development of a stationary kernel for GP-LVMs and a prior distribution for unsupervised cases. The detail of each component is as follows.

• Kernel function. We focus on the spectral mixture (SM) kernel [30] and introduce it into a GP-LVM. Generally, the SE kernel and Matérn kernel are applied to GP-LVMs, which are homogeneous kernels that only depend on the distance between latent features. In contrast, the SM kernel is a broader class kernel that depends on not only the distance but the periodicity of input features. Although the SM kernel has strong expressiveness compared to the SE and Matérn kernels, it has been applied to regression models and not to GP-LVMs. In this study, we develop the SM kernel for GP-LVMs and reveal its high ability for pattern discovery and visualization.
• Prior distribution. We focus on the use of a locally estimated manifold in the prior distribution. Local structures, such as the cluster structure, are crucial to improving the visibility of a low-dimensional representation. Therefore, we regularize low-dimensional embedding with the local estimation of the data manifold. Specifically, we locally approximate it with a neighborhood graph and regularize the low-dimensional representation with it. Through this strategy, we consider local structures and improve the visibility of low-dimensional representation. The regularization based on the graph Laplacian in supervised cases has been previously studied as the Gaussian process latent random field (GPLRF) [33], and we apply the GPLRF scheme in an unsupervised manner.

The remainder is as follows. In Section 2 and Section 3, we present related works and the preliminaries to our method. Section 4 presents our method, and we validate our method in Section 5.

2. Related Works

This section presents previous studies on dimensionality reduction approaches aiming at data visualization and preprocessing. We explain the graph-based method in Section 2.1 and PCA-based methods in Section 2.2.

2.1. Graph-Based Approaches

Graph-based approaches first encode high-dimensional data into a neighborhood graph to preserve the local structures of data and embed a neighborhood graph into a low-dimensional space, generally a two-dimensional space. Laplacian eigenmaps (LEs) [34] are one of the classical methods in graph-based approaches and they derive a low-dimensional representation as the eigenvectors of the graph Laplacian of the neighborhood graph. An LE is a linear method with rich theoretical properties for its convergence and loss function [35,36]. t-Distributed stochastic embedding (t-SNE) [10] is a long-standing nonlinear graph embedding approach that provides a low-dimensional representation by matching the neighborhood structure. Specifically, t-SNE minimizes the Kullback–Leibler (KL) divergence between neighborhood graphs in high- and low-dimensional space. For its strong visibility, t-SNE is frequently used to visualize complex structures, such as biological data and the middle layers of deep neural networks, and researchers have developed a computationally efficient version of t-SNE introducing the Burnes–Hut algorithm [11] or hierarchical optimization [37]. UMAP [12] is a state-of-the-art method in graph-based approaches and is a more efficient algorithm than t-SNE. UMAP derives a k-nearest neighborhood graph as a fuzzy topological representation of data. Then, UMAP minimizes the cross-entropy of neighborhood graphs in high- and low-dimensional space, which does not necessitate the normalization of the graph, in contrast to t-SNE which necessitates this normalization for the calculation of KL divergence. Furthermore, the cross-entropy objective is considered as a combination of the attraction force of neighbors and the repulsion force of non-neighbors, allowing negative-sampling-based optimization to boost the scalability of the UMAP algorithm. LE, t-SNE, and UMAP have similar properties [38] in their loss functions and produce visible two-dimensional representations compared with the other dimensionality reduction approaches. However, their representations do not preserve the global structures of data because of the encoding process of data into neighborhood graphs. Researchers have developed approaches for global preservation by introducing a graph construction scheme based on the triplet [39,40] or diffusion process [41]. However, they estimate global structures by connecting local information, which inevitably causes failure to preserve the global structure. In this study, we focus on PCA-based approaches, which are suited for global preservation, and introduce them in the following subsection.

2.2. PCA-Based Approaches

PCA [2,3] is a classical dimensionality reduction technique and has been applied in various situations, such as data visualization and preprocessing. PCA aims to derive a linear projection onto a subspace that maximizes the variance of projected high-dimensional data. This formulation yields a closed-form expression of the embedding, improving the applicability and scalability of PCA. Probabilistic PCA (PPCA) [42] introduces probabilistic formulation into PCA. Specifically, PPCA defines latent variables in a low-dimensional space and derives the projection by maximizing the likelihood after marginalizing these latent variables. Although PCA and PPCA have produced successful results, they are both linear methods, and data generally contain nonlinear correlations. GP-LVMs [15] are a nonlinear extension of the PCA framework and realize nonlinear embedding with two differences: optimization of latent variables instead of projection and introduction of the kernel method [43] for nonlinear expression. These differences are achieved by the Gaussian-process-based formulation, and GP-LVMs assume that high-dimensional data are generated from the Gaussian process [44] input using latent variables. GP-LVMs can derive nonlinear latent variables according to the nonlinearity of the kernel, and several extensions introduce prior distributions, including discriminative [28,32], dynamic [24,25], and hierarchical GP-LVMs [45]. Recently, GP-LVMs have been developed for scalable training by inducing methods [46,47,48], for the Bayesian inference of latent variables [16,23], and for deep modeling [49,50]. Although GP-LVMs and PCA-based approaches preserve the global structures of data, they do not consider the local structures of data beneficial for visualization-aided dimensionality reduction. Furthermore, the kernel functions used in GP-LVMs are selected from homogeneous kernels, which have limited expression. In this study, we design a GP-LVM for visualization with a stationary kernel and local estimation of data.

3. Preliminary

In this section, we present the preliminaries of our method. Specifically, we introduce the basic idea of a GP-LVM, the kernel function, and the UMAP algorithm for the local estimation of the data manifold.

3.1. Gaussian Process Latent Variable Model

Let $\mathbf{Y}={[{\mathbf{y}}_1,{\mathbf{y}}_2,\dots ,{\mathbf{y}}_N]}^{\top }\in {\mathbb{R}}^{N\times D}$ be D-dimensional observed variables with N samples. A GP-LVM assumes the existence of a projection from the Q-dimensional latent space to the D-dimensional observed space. Specifically, a GP-LVM assumes that the observed variables are generated from the Gaussian process [44] input by Q-dimensional latent variables $\mathbf{X}={[{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_N]}^{\top }\in {\mathbb{R}}^{N\times Q}$ as follows:

$$\begin{array}{cc}\hfill {\mathbf{y}}_{:,d}& =\mathbf{f}\left(\mathbf{X}\right)+\mathbf{ϵ},\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill \mathbf{f}& \sim \mathcal{GP}(\mathbf{0},k(·,·\left)\right),\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill \mathbf{ϵ}& \sim \mathcal{N}(\mathbf{0},{\beta }^{-1}\mathbf{I}),\hfill \end{array}$$

(3)

where ${\mathbf{y}}_{:,d}\in {\mathbb{R}}^N$ is the d-th column of $\mathbf{Y}$, $\mathbf{f}\sim \mathcal{GP}(\mathbf{0},k(·,·\left)\right)$ is a zero-mean Gaussian process prior with a kernel function $k(·,·)$, and $\mathbf{ϵ}$ is a Gaussian noise with a precision $\beta$. The GP-LVM estimates the latent variables $\mathbf{X}$ by maximum likelihood estimation after marginalizing the Gaussian process prior $\mathbf{f}$. The log-likelihood function of the GP-LVM is derived from Equations (1)–(3) as

$$\begin{array}{cc}\hfill logp\left(\mathbf{Y}\right|\mathbf{X})& =\sum _{d=1}^{D}log\mathcal{N}\left({\mathbf{y}}_{:,d}\right|\mathbf{0},{\mathbf{K}}_{NN}+{\beta }^{-1}\mathbf{I})\hfill \\ \hfill & =-\frac{ND}{2}log2\pi -\frac{D}{2}log|{\mathbf{K}}_{NN}+{\beta }^{-1}\mathbf{I}|-\frac{1}{2}\mathrm{tr}\left[{({\mathbf{K}}_{NN}+{\beta }^{-1}\mathbf{I})}^{-1}\mathbf{Y}{\mathbf{Y}}^{\top }\right],\hfill \end{array}$$

(4)

where ${\mathbf{K}}_{NN}\in {\mathbb{R}}^{N\times N}$ denotes a gram matrix whose $ij$-th entry is $k({\mathbf{x}}_i,{\mathbf{x}}_j)$. The extensions of a GP-LVM generally introduce a prior distribution of the latent variables $p\left(\mathbf{X}\right)$ and estimate them by maximum a posteriori (MAP) estimation. The posterior distribution is given as the following equation:

$$\begin{array}{c}\hfill logp\left(\mathbf{X}\right|\mathbf{Y})=logp(\mathbf{Y}\left|\mathbf{X}\right)+logp\left(\mathbf{X}\right)+C,\end{array}$$

(5)

where C represents a constant term. From these equations, the selection of the kernel function $k(·,·)$ and the prior $p\left(\mathbf{X}\right)$ significantly influences the quality of the derived latent variables. Furthermore, the evaluation of Equation (4) contains the inversion of an $N\times N$ matrix ${\mathbf{K}}_{NN}$ whose evaluation necessitates cubic time complexity. This complexity limits the scalability of the model, and the original GP-LVM cannot handle datasets with thousands of samples. Therefore, we use the inducing method [47] for the evaluation of the likelihood function and describe it in Section 4.

3.2. Kernel Function

The kernel function can be considered as a similarity measure of two inputs $\mathbf{x},{\mathbf{x}}^{\prime }\in {\mathbb{R}}^Q$ and should be a positive definite function. The most well-known kernel function is the SE or radius basis function (RBF) kernel, defined as follows:

$$\begin{array}{c}\hfill k_{SE}(\mathbf{x},{\mathbf{x}}^{\prime })={\sigma }^{2}exp\left(-\frac{1}{2l^2}\left|\right|\mathbf{x}-{\mathbf{x}}^{\prime }{\left|\right|}^2\right),\end{array}$$

(6)

where $\sigma$ indicates a variance and l indicates a lengthscale parameter. Another popular kernel is the Matérn kernel, which is the sparse version of the SE kernel. The Matérn kernel is given as

$$\begin{array}{c}\hfill k_{Mat\acute{e}rn}(\mathbf{x},{\mathbf{x}}^{\prime })={\sigma }^2\frac{2^{1-\nu }}{\Gamma \left(\nu \right)}{\left(\sqrt{2\nu }\frac{\left|\right|\mathbf{x}-{\mathbf{x}}^{\prime }\left|\right|}{\kappa }\right)}^{\nu }{K}_{\nu }\left(\sqrt{2\nu }\frac{\left|\right|\mathbf{x}-{\mathbf{x}}^{\prime }\left|\right|}{\kappa }\right),\end{array}$$

(7)

where $\nu$ and $\kappa$ denote kernel parameters, $\Gamma (·)$ denotes the gamma function, and $K_v(·)$ denotes the modified Bessel function of the second kind. $\nu$ is generally set to $\frac{3}{2},\frac{5}{2}$, and they can be easily defined as follows:

$$\begin{array}{cc}\hfill k_{Mat\acute{e}rn3/2}(\mathbf{x},{\mathbf{x}}^{\prime })& ={\sigma }^2\left(1+\frac{\sqrt{3}\left|\right|\mathbf{x}-{\mathbf{x}}^{\prime }\left|\right|}{\kappa }\right)exp\left(-\frac{\sqrt{3}\left|\right|\mathbf{x}-{\mathbf{x}}^{\prime }\left|\right|}{\kappa }\right),\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill k_{Mat\acute{e}rn5/2}(\mathbf{x},{\mathbf{x}}^{\prime })& ={\sigma }^2\left(1+\frac{\sqrt{5}\left|\right|\mathbf{x}-{\mathbf{x}}^{\prime }\left|\right|}{\kappa }+\frac{5\left|\right|\mathbf{x}-{\mathbf{x}}^{\prime }{\left|\right|}^2}{3{\kappa }^2}\right)exp\left(-\frac{\sqrt{5}\left|\right|\mathbf{x}-{\mathbf{x}}^{\prime }\left|\right|}{\kappa }\right)\hfill \end{array}$$

(9)

Furthermore, if $\nu \to \infty$, the Matérn kernel converges to the SE kernel in Equation (6). Although the SE and Matérn kernels have different expressions, they are both homogeneous kernels as $k(\mathbf{x},{\mathbf{x}}^{\prime })=k\left(\left|\right|\mathbf{x}-{\mathbf{x}}^{\prime }\left|\right|\right)$. The stationary kernel is in a broader class of homogeneous kernels as $k(\mathbf{x},{\mathbf{x}}^{\prime })=k\left(\mathbf{x}-{\mathbf{x}}^{\prime }\right)$. The SM kernel [30] is a stationary kernel and can measure the periodicity of input signals. The SM kernel is defined by the following equation:

$$\begin{array}{c}\hfill k_{SM}(\mathbf{x},{\mathbf{x}}^{\prime })=\sum _{p=1}^{N_w}{w}_p\prod _{q=1}^{Q}exp\left[-2{\pi }^2{(x_q-x_q^{\prime })}^2{v}_q^{\left(p\right)}\right]cos\left[2\pi (x_q-x_q^{\prime }){\mu }_q^{\left(p\right)}\right],\end{array}$$

(10)

where $N_w$ denotes the number of mixtures, $w_p$ denotes a weight parameter of each mixture, $v_q^{\left(p\right)}$ denotes a lengthscale parameter of the dimension q of a mixture p, and ${\mu }_q^{\left(p\right)}$ denotes a mean parameter of the dimension q of a mixture p. We collectively denote ${\mathit{\mu }}_p={[{\mu }_1^{\left(p\right)},{\mu }_2^{\left(p\right)},\dots ,{\mu }_Q^{\left(p\right)}]}^{\top }\in {\mathbb{R}}^Q$. The SM kernel is derived from Bochner’s theorem, which determines the spectral density of any stationary kernel. On this basis, ${\mu }_q^{\left(p\right)}$ indicates the center in the spectral domain, and the SE kernel is the special case of the SM kernel as ${\mu }_q^{\left(p\right)}=0$. The SM kernel measures the similarity between two inputs with the distance $\left|\right|\mathbf{x}-{\mathbf{x}}^{\prime }\left|\right|$ and the periodicity represented by the cosine function of the spectral ${\mathit{\mu }}_p$. By optimizing these parameters, we can find the optimal center in the spectral domain from the input signals and derive a data-specific expression of the kernel. For intuitive comparison of these kernels, we present the sampling results from $\mathcal{GP}(\mathbf{0},k(·,·\left)\right)$ for each kernel in Figure 1.

3.3. Uniform Manifold Approximation and Projection

The approximation of data manifolds is crucial for graph-based dimensionality reduction approaches. UMAP is a state-of-the-art method and incorporates the idea of algebraic topology and categorical theory. As a result, the approximation of data manifolds is simplified to a k-nearest neighborhood graph computation, which is computationally efficient due to its very sparse structure. Specifically, UMAP first calculates a weight matrix ${\mathbf{W}}_{n|m}$ as follows:

$$\begin{array}{c}\hfill {\left[{\mathbf{W}}_{n|m}\right]}_{ij}=\left\{\begin{array}{cc}exp\left[-\frac{\mathrm{d}({\mathbf{y}}_i,{\mathbf{y}}_j)-{\rho }_i}{{\sigma }_i}\right]\hfill & (j\in \{i_1,i_2,\dots ,i_k\})\hfill \\ 0\hfill & \left(\mathrm{otherwise}\right)\hfill \end{array}\right.,\end{array}$$

(11)

where ${\left[{\mathbf{W}}_{n|m}\right]}_{ij}$ denotes an $(i,j)$ element of ${\mathbf{W}}_{n|m}$, $i_l(l=1,2,\dots ,k)$ denotes the l-nearest neighbor of ${\mathbf{y}}_i$, $\mathrm{d}(·,·)$ denotes an arbitrary distance function, ${\rho }_i=\mathrm{d}({\mathbf{y}}_i,{\mathbf{y}}_{i_1})$ denotes a nearest neighbor distance, and ${\sigma }_i$ denotes local connectivity around ${\mathbf{y}}_i$. The weight matrix represents the similarities between samples, and UMAP approximates the data manifold as follows:

$$\begin{array}{c}\hfill \mathbf{W}={\mathbf{W}}_{n|m}+{\mathbf{W}}_{m|n}-{\mathbf{W}}_{n|m}\circ {\mathbf{W}}_{m|n},\end{array}$$

(12)

where ∘ denotes the Hadamard product. Equation (12) corresponds to the symmetrization of ${\mathbf{W}}_{n|m}$, and $\mathbf{W}$ is called a fuzzy topological representation of observed variables $\mathbf{Y}$. $\mathbf{W}$ denotes a locally estimated data manifold that contains local information about data. Then, UMAP minimizes the cross-entropy between the fuzzy topological representation and the local structure of the low-dimensional space. In this study, we focus on the estimated manifold $\mathbf{W}$ and regularize the embedding with $\mathbf{W}$ to improve the visibility of the representation.

4. Proposed Method

In Section 4.1, we define our model based on GP-LVMs and introduce the regularization with the graph Laplacian based on GPLRF. In Section 4.2, we derive the scalable lower bound of the log-posterior distribution, which enables our model to deal with a larger dataset than the original GP-LVM. In Section 4.3, we present the optimization method of our model and extend the SM kernel for the use of latent variable models.

4.1. Model Formulation

As in GP-LVMs, we define the generation process from the latent space to the observed space through the Gaussian process. Furthermore, we establish a prior distribution of latent variables by applying the GPLRF scheme in an unsupervised manner. The model of our method is as follows:

$$\begin{array}{cc}\hfill {\mathbf{y}}_{:,d}& =\mathbf{f}\left(\mathbf{X}\right)+\mathbf{ϵ},\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {\mathbf{x}}_{:,q}& \sim \mathcal{N}(\mathbf{0},{\alpha }^{-1}{\mathbf{L}}^{†}),\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill \mathbf{f}& \sim \mathcal{GP}(\mathbf{0},k_{SM}(·,·)),\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill \mathbf{ϵ}& \sim \mathcal{N}(\mathbf{0},{\beta }^{-1}\mathbf{I}),\hfill \end{array}$$

(16)

where ${\mathbf{x}}_{:,q}\in {\mathbb{R}}^N$ denotes the q-th column of $\mathbf{X}$, $\alpha$ denotes a hyperparameter to control the strength of the regularization, $\mathbf{L}\in {\mathbb{R}}^{N\times N}$ is a graph Laplacian of a fuzzy topological representation $\mathbf{W}$ in Equation (12), and † denotes the pseudoinverse. From these definitions, we estimate the latent variables and kernel parameters by MAP estimation, and the objective function is given as follows:

$$\begin{array}{cc}\hfill logp\left(\mathbf{X}\right|\mathbf{Y})& =logp\left(\mathbf{Y}\right|\mathbf{X})+logp(\mathbf{X})+C\hfill \\ \hfill & =\sum _{d=1}^{D}logp\left({\mathbf{y}}_{:,d}\right|\mathbf{X})+\sum _{q=1}^{Q}logp\left({\mathbf{x}}_{:,q}\right)+C,\hfill \end{array}$$

(17)

where

$$\begin{array}{cc}\hfill p\left({\mathbf{x}}_{:,q}\right)& =\mathcal{N}\left({\mathbf{x}}_{:,q}\right|0,{\alpha }^{-1}{\mathbf{L}}^{†}).\hfill \end{array}$$

(18)

The novelty of our method is the use of the expressive kernel in Equation (15) and the prior distribution utilizing the locally estimated manifold W in Equation (14). We further explain the effects of the prior distribution in Equation (18). The log-prior term in Equation (17) can be written as follows:

$$\begin{array}{cc}\hfill logp\left(\mathbf{X}\right)& =-\frac{\alpha }{2}\sum _{q=1}^Q{\mathbf{x}}_{:,q}^{\top }\mathbf{L}{\mathbf{x}}_{:,q}+C\hfill \\ \hfill & =-\frac{\alpha }{4}\sum _{i=1}^N\sum _{j=1}^N{w}_{ij}\left|\right|{\mathbf{x}}_i-{\mathbf{x}}_j{\left|\right|}^2+C,\hfill \end{array}$$

(19)

where C denotes a constant. From Equation (19), the log-prior term is equal to the weighted sum of the distances between connected samples on the neighborhood graph. Latent variables are coordinated to minimize them through optimization; thus, the connected samples become closer, and the local information $\mathbf{W}$ is reflected in the latent representation. In Equation (17), the first term is equal to Equation (4) and necessitates $O\left(N^3\right)$ time complexity. In the following section, we introduce the inducing method [47] into Equation (17) and derive a scalable objective function.

4.2. Lower Bound of Log-Posterior Distribution

The inducing method [47] uses the property of the conditional distribution of a Gaussian and replaces the matrix inversion of a large matrix with that of a small matrix. Specifically, we define a few inducing points $\mathbf{u}\in {\mathbb{R}}^M$ and their positions $\mathbf{Z}={[{\mathbf{z}}_1,{\mathbf{z}}_2,\dots ,{\mathbf{z}}_M]}^{\top }\in {\mathbb{R}}^{M\times Q}$ (M being the number of inducing points) in the latent space and evaluate the log-likelihood function after conditioning by these inducing positions $\mathbf{Z}$ as $logp\left({\mathbf{y}}_{:,d}\right|\mathbf{X},\mathbf{Z})$. First, the joint distribution of ${\mathbf{y}}_{:,d}$, $\mathbf{f}$, and $\mathbf{u}$ is given as follows:

$$\begin{array}{c}\hfill p({\mathbf{y}}_{:,d},\mathbf{f},\mathbf{u}|\mathbf{X},\mathbf{Z})=p\left({\mathbf{y}}_{:,d}\right|\mathbf{f})p\left(\mathbf{f}\right|\mathbf{u},\mathbf{X},\mathbf{Z})p\left(\mathbf{u}\right|\mathbf{Z}).\end{array}$$

(20)

Each distribution is given as the following equation:

$$\begin{array}{cc}\hfill p\left({\mathbf{y}}_{:,d}\right|\mathbf{f})& =\mathcal{N}\left({\mathbf{y}}_{:,d}\right|\mathbf{f},{\beta }^{-1}\mathbf{I}),\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill p\left(\mathbf{f}\right|\mathbf{u},\mathbf{X},\mathbf{Z})& =\mathcal{N}\left(\mathbf{f}\right|{\mathbf{K}}_{NM}{\mathbf{K}}_{MM}\mathbf{u},{\mathbf{K}}_{NN}-{\mathbf{K}}_{NM}{\mathbf{K}}_{MM}^{-1}{\mathbf{K}}_{MN}),\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill p\left(\mathbf{u}\right|\mathbf{Z})& =\mathcal{N}\left(\mathbf{u}\right|\mathbf{0},{\mathbf{K}}_{MM}),\hfill \end{array}$$

(23)

where ${\mathbf{K}}_{MN}={\mathbf{K}}_{NM}^{\top }\in {\mathbb{R}}^{M\times N}$ denotes a covariance matrix whose $ij$-th entry is $k_{SM}({\mathbf{x}}_i,{\mathbf{z}}_j)$ and ${\mathbf{K}}_{MM}\in {\mathbb{R}}^{M\times M}$ is a gram matrix whose $ij$-th entry denotes $k_{SM}({\mathbf{z}}_i,{\mathbf{z}}_j)$. For the evaluation of $logp\left({\mathbf{y}}_{:,d}\right|\mathbf{X},\mathbf{Z})$, we need to marginalize the Gaussian process prior $\mathbf{f}$ and inducing points $\mathbf{u}$. The variational method [47] introduces the Jensen inequality to evaluate the log-likelihood as follows:

$$\begin{array}{cc}\hfill logp\left({\mathbf{y}}_{:,d}\right|\mathbf{X},\mathbf{Z})& =log\int \left\{\int p\left({\mathbf{y}}_{:,d}\right|\mathbf{f})p\left(\mathbf{f}\right|\mathbf{u},\mathbf{X},\mathbf{Z})d\mathbf{f}\right\}p\left(\mathbf{u}\right|\mathbf{Z})d\mathbf{u}\hfill \\ \hfill & \ge log\int exp\left\{\int logp\left({\mathbf{y}}_{:,d}\right|\mathbf{f})p\left(\mathbf{f}\right|\mathbf{u},\mathbf{X},\mathbf{Z})d\mathbf{f}\right\}p\left(\mathbf{u}\right|\mathbf{Z})d\mathbf{u}\hfill \\ \hfill & \triangleq {\mathcal{L}}_d.\hfill \end{array}$$

(24)

We evaluate the lower bound ${\mathcal{L}}_d$ of the log-posterior in Equation (24) instead of the original log-likelihood. ${\mathcal{L}}_d$ can be computed in the closed-form expression as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_d=-\frac{N}{2}log2\pi -\frac{1}{2}log|{\mathbf{Q}}_{NN}+{\beta }^{-1}\mathbf{I}|-\frac{1}{2}{\mathbf{y}}_{:,d}^{\top }{({\mathbf{Q}}_{NN}+{\beta }^{-1}\mathbf{I})}^{-1}{\mathbf{y}}_{:,d}-\frac{\beta }{2}\mathrm{tr}({\mathbf{K}}_{NN}-{\mathbf{Q}}_{NN}),\end{array}$$

(25)

where ${\mathbf{Q}}_{NN}={\mathbf{K}}_{NM}{\mathbf{K}}_{MM}^{-1}{\mathbf{K}}_{MN}$. By comparing Equation (4) with Equation (25), the $N\times N$ matrix ${\mathbf{K}}_{NN}$ is replaced by the matrix ${\mathbf{Q}}_{NN}$, which is considered a low-rank approximated form of the original matrix ${\mathbf{K}}_{NN}$. Therefore, by applying the matrix formula to Equation (25), we can evaluate the lower bound of the likelihood in $O\left(N{M}^2\right)$ time complexity more efficiently than the likelihood function in Equation (4). By substituting (25) into (17) and neglecting the constants, we obtain the following objective function $\mathcal{L}$:

$$\begin{array}{cc}\hfill \mathcal{L}& =-\frac{D}{2}log|{\mathbf{Q}}_{NN}+{\beta }^{-1}\mathbf{I}|-\frac{1}{2}\mathrm{tr}\left[{({\mathbf{Q}}_{NN}+{\beta }^{-1}\mathbf{I})}^{-1}\mathbf{Y}{\mathbf{Y}}^{\top }\right]\hfill \\ \hfill & -\frac{\beta D}{2}\mathrm{tr}({\mathbf{K}}_{NN}-{\mathbf{Q}}_{NN})-\frac{\alpha Q}{2}\mathrm{tr}\left(\mathbf{L}\mathbf{X}{\mathbf{X}}^{\top }\right).\hfill \end{array}$$

(26)

We maximize the lower bound of the likelihood function in Equation (26) and explain the methodology of the optimization in the following section.

4.3. Optimization

The objective function in Equation (26) is a continuous function with respect to the latent variables and kernel parameters and can be generally maximized by gradient-based optimization, such as conjugate gradient descent [51] and the quasi-Newton method [52]. Generally, gradients are derived from a chain role of differentiation, and, thus, we need to compute the differentiation of the SE kernel with respect to the latent variables. This differentiation is given as follows:

$$\begin{array}{cc}\hfill \frac{\partial }{\partial x_q}k(\mathbf{x},{\mathbf{x}}^{\prime })& =\sum _{p=1}^{N_w}{w}_p\left[\phantom{\prod _{q=1}^Q}\left[-4{\pi }^2(x_q-x_q^{\prime })v_q^{\left(p\right)}-2\pi {\mu }_q^{\left(p\right)}tan\left\{2\pi (x_q-x_q^{\prime }){\mu }_q^{\left(p\right)}\right\}\right]\right.\hfill \\ \hfill & \left.\times \prod _{q=1}^{Q}exp\left\{-2{\pi }^2{(x_q-x_q^{\prime })}^2{v}_q^{\left(p\right)}\right\}cos\left\{2\pi (x_q-x_q^{\prime }){\mu }_q^{\left(p\right)}\right\}\right].\hfill \end{array}$$

(27)

Furthermore, the differentiations with respect to the kernel parameters $w_p$, $v_q^{\left(p\right)}$, and ${\mu }_q^{\left(p\right)}$ are given as follows:

$$\begin{array}{cc}\hfill \frac{\partial }{\partial w_p}{k}_{SM}(\mathbf{x},{\mathbf{x}}^{\prime })& =\prod _{q=1}^{Q}exp\left\{-2{\pi }^2{(x_q-x_q^{\prime })}^2{v}_q^{\left(p\right)}\right\}cos\left\{2\pi (x_q-x_q^{\prime }){\mu }_q^{\left(p\right)}\right\},\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill \frac{\partial }{\partial v_q^{\left(p\right)}}{k}_{SM}(\mathbf{x},{\mathbf{x}}^{\prime })& =-2{\pi }^2{(x_q-x_q^{\prime })}^{2}exp\left\{-2{\pi }^2{(x_q-x_q^{\prime })}^2{v}_q^{\left(p\right)}\right\}cos\left\{2\pi (x_q-x_q^{\prime }){\mu }_q^{\left(p\right)}\right\},\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill \frac{\partial }{\partial {\mu }_q^{\left(p\right)}}{k}_{SM}(\mathbf{x},{\mathbf{x}}^{\prime })& =-2\pi (x_q-x_q^{\prime })tan\left\{2\pi (x_q-x_q^{\prime }){\mu }_q^{\left(p\right)}\right\}\hfill \\ \hfill & \times exp\left\{-2{\pi }^2{(x_q-x_q^{\prime })}^2{v}_q^{\left(p\right)}\right\}cos\left\{2\pi (x_q-x_q^{\prime }){\mu }_q^{\left(p\right)}\right\}.\hfill \end{array}$$

(30)

We optimize the parameters with Equations (27)–(30) as well as $\frac{\partial \mathcal{L}}{\partial {\mathbf{K}}_{MN}}$ and $\frac{\partial \mathcal{L}}{\partial {\mathbf{K}}_{MM}}$, which have been derived in previous research [48]. After optimization, we obtain the optimal latent variables $\mathbf{X}$, spectral ${\mu }_p^{\left(q\right)}$, weights $w_p$, and lengthscales $v_q^{\left(p\right)}$. As a result, we can reduce the number of hyperparameters and improve the tractability of our model.

5. Experiment

In this section, we validate our method through qualitative and quantitative experiments. We implemented our method with GPy [53], an open library for the Gaussian process, and ran it on an Intel Core i7-10700 CPU and 16 GB random access memory.

5.1. Experimental Setup

5.1.1. Dataset

We used four real-world datasets.

Table 1. Details of the datasets used in this study.

Dataset	Samples	Dimensions	Categories	Type	Features
MNIST	20,000	784	10	Image	Pixels
COIL-20	720	16,384	10	Image	Pixels
DBPedia	20,000	100	14	Text	FastText
FMNIST	20,000	784	10	Image	Pixels

shows summary of the datasets used in this study.

• MNIST (http://yann.lecun.com/exdb/mnist/ (accessed on 25 July 2022)) contains 70K images of hand-written digits and labels corresponding to each digit. We randomly selected 20K images and colored embeddings along the corresponding labels. MNIST contains a cluster structure of each digit, and the embedding should preserve the cluster structure.
• COIL-20 [54] contains 1440 grayscale images of rotated objects. We used the first ten objects and colored embeddings along each object. Images of COIL-20 have a rotating structure, and the embedding should preserve it.
• DBPedia (https://wiki.dbpedia.org/ (accessed on 29 July 2022)) contains 530K Wikipedia articles classified into 14 categories. We used 20K random articles and extracted feature vectors by FastText following [37]. We colored the embedding following the category information, and the low-dimensional representation should preserve the cluster structure.
• Fashion MNIST (FMNIST) [55] contains 70K images of 10 kinds of fashion items and labels corresponding to each item. We randomly selected 20K images and colored embeddings to match the corresponding labels. Although FMNIST has a cluster structure similar to MNIST, the categories of FMNIST are more correlated than those of MNIST, and separating clusters in a low-dimensional representation is more difficult.

5.1.2. Comparative Methods

To validate the novelty of our method, we compared the proposed method (PM) with its version without the SM kernel in Equation (10) (PM w/o SM kernel) and without the prior in Equation (18) (PM w/o prior). Furthermore, we compared our method with the following comparative methods.

• PCA [3] is a classical method for dimensionality reduction and linearly derives its embedding. We compared our method with PCA as a benchmark method. Note that GP-LVM-based methods typically use PCA as the initial values of their latent variables.
• LE [34] is a classical graph-based approach and derives its low-dimensional embedding as the eigenvectors of the graph Laplacian. We used LE as the benchmark in the graph-based approaches.
• t-SNE [10,11] is a long-standing graph-based data visualization technique that derives the low-dimensional representation by minimizing the KL divergence between the observed and low-dimensional spaces. We used t-SNE as the baseline method of the graph-based dimensionality reduction approaches.
• Bayesian GP-LVM (BGP-LVM) [16,25] is a GP-LVM that introduces the Bayesian inference into the latent variables. We used BGP-LVM as the baseline method and visualized the mean vectors as a low-dimensional representation. We used the RBF kernel in Equation (6) and standard normal density as the prior distribution following the original work [16].
• Potential of heat diffusion of affinity-based transition embedding (PHATE) [41] is a recently proposed graph-based approach and derives the neighborhood graph on the basis of the diffusion operation [56], enabling the global preservation of data. We used PHATE as a state-of-the-art method among the methods aimed at preserving global structures.

5.1.3. Evaluation

We used average-based and correlation-based metrics to evaluate the local and global preservation quality. The average-based metric is calculated by averaging the preservation quality around each data point and evaluates the local preservation around them. The correlation-based metric is a correlation value between the observed and embedding spaces and quantifies the global similarity between two spaces. We used (1) trustworthiness [57] as an average-based metric and (2) Shepard goodness [58] as a correlation-based one, which have also been used in previous studies [58,59]. Trustworthiness $T\left(k\right)\in [0,1]$ is defined as follows:

$$\begin{array}{c}\hfill T\left(k\right)=1-\frac{2}{Nk(2N-3k-1)}\sum _{i=1}^N\sum _{j\in N_i^k}\max (0,r(i,j)-k),\end{array}$$

(31)

where $N_i^k$ denotes an index list of the k-nearest neighbors in the low-dimensional space, and $r(i,j)$ indicates that samples i and j are the $r(i,j)$-th nearest neighbors in the observed space. Therefore, trustworthiness penalizes unexpected neighbors in the low-dimensional space and can evaluate the preservation quality of the local structures of data. We set the number of neighbors as $k=5$, but we observed that the results are robust to this setting. Shepard goodness is a global metric and is calculated with the Shepard diagram [60]. The Shepard diagram is a scatterplot of the point-wise distances of all points in the observed and embedding spaces. The Shepard goodness is the Spearman rank correlation of the Shepard diagram and can evaluate the global similarity between the original data and embedding. The Shepard goodness is computed as follows:

$$\begin{array}{c}\hfill \rho =1-\frac{6{\sum }_{n=1}^{N_s}{D}_n^2}{N_s^3-N_s},\end{array}$$

(32)

where $N_s=\frac{N(N-1)}{2}$ is the number of the points on the Shepard diagram and $D_n$ is the distance between the ranks of two features of the n-th point on the Shepard diagram. Both metrics are within $[0,1]$, and a higher value indicates better results. Importantly, the global and local metrics are tradeoffs of each other and should be balanced to realize reliable visualization.

5.1.4. Training Procedure

Our method optimizes the latent variables and kernel parameters by maximizing the lower bound in Equation (26). The hyperparameters in our method include the strength of the regularization $\alpha$ and the number of inducing points M. The number of dimensions D enlarges the log-likelihood function in Equation (26); thus, we chose $\alpha$ as the proportional number to the dimensions $\alpha =0.1D$. The number of inducing points significantly impacts the quality of the low-dimensional representation and is easily chosen in the small natural number (e.g., 10∼50); thus, we adaptively selected M according to the dataset. We optimized the objective function in Equation (26) using L-BFGS-B [52] of SciPy implementation [61].

5.2. Ablation Study

Figure 2 shows the ablation results on MNIST. We observed that the low-dimensional representation with PM w/o SM kernel divided the independent clusters into different parts in contrast to PM and PM w/o prior that properly connected each cluster. PM and PM w/o prior used the SM kernel considering the spectral density of the latent variables, which contributed to the proper embeddings of the cluster structure. Although the effect of the prior distribution was unclear from the MNIST results, it was demonstrated in the COIL-20 results in Figure 3. PM w/o prior does not preserve the object variation and rotation structure, due to the absence of a local structure consideration. From these comparisons, we confirm the effectiveness of our novelty.

5.3. Qualitative Results

Figure 4 shows the visualization results with the comparative methods. On MNIST, PCA and LE do not reflect the cluster structure corresponding to each digit, which is difficult to capture by linear dimensionality reduction. In contrast, embeddings of the nonlinear methods, such as t-SNE, BGP-LVM, PHATE, and PM, contain several cluster structures and successfully preserve the local structure of data. In the COIL-20 results, PCA and LE do not preserve the object variation. t-SNE, BGP-LVM, and PM retain the object variation, and PM retains the rotation structure more correctly than t-SNE and BGP-LVM, which have several linear and separated clusters. PHATE can embed COIL-20 with a complete object variation and rotation structure, due to the graph embedding optimization. On DBPedia, PCA and LE also fail to preserve local clusters. t-SNE, BGP-LVM, PHATE, and PM preserve them, but their shapes are different, and their correctness needs to be evaluated by quantitative metrics. On FMNIST, although all embeddings do not preserve the cluster structure, t-SNE, PHATE, and PM split them into several independent clusters, beneficial for gaining insight into the data structure.

5.4. Quantitative Results

Table 2. Quantitative results of trustworthiness. We boldface the best results and underline the second-best results on each dataset.

Dataset	PCA	LE	t-SNE	BGP-LVM	PHATE	PM
MNIST	0.738	0.756	0.994	0.823	0.871	0.881
COIL-20	0.898	0.882	0.997	0.984	0.931	0.974
DBPedia	0.883	0.956	0.998	0.989	0.986	0.990
FMNIST	0.912	0.927	0.994	0.909	0.958	0.953

and

Table 3. Quantitative results of Shepard goodness. We boldface the best results and underline the second-best results on each dataset.

Dataset	PCA	LE	t-SNE	BGP-LVM	PHATE	PM
MNIST	0.503	0.431	0.349	0.464	0.368	0.512
COIL-20	0.818	0.633	0.611	0.525	0.355	0.687
DBPedia	0.778	0.490	0.339	0.594	0.361	0.694
FMNIST	0.876	0.692	0.579	0.883	0.615	0.862

show the quantitative results of trustworthiness and Shepard goodness on each dataset, respectively. We first evaluate these results by separating them into graph-based methods (LE, t-SNE, and PHATE) and PCA-based methods (PCA and BGP-LVM). LE, t-SNE, and PHATE exhibit high values on trustworthiness (local metric) but low values on Shepard goodness (global metric). They embedded a neighborhood graph, the local estimation of given data, and thus over-reflected the local structures in low-dimensional representations. This drawback resulted in misleading visualizations due to the lack of global preservation, such as the shape of clusters on DBPedia by t-SNE and PHATE in Figure 4. In contrast, PCA and BGP-LVM, especially PCA, performed a reliable reduction in global preservation with a high Shepard goodness value, but they did not show good local preservation, resulting in a non-visible representation. Our method exhibited high values for both trustworthiness and Shepard goodness, indicating that our method balances global and local structure preservation, i.e., ensuring the reliability and visibility of low-dimensional embedding. Our method is based on GP-LVMs and regularizes latent variables with a neighborhood graph. This approach contributed to retaining local and global structures and achieving the high reliability and visibility of low-dimensional representations. From the above, we confirm the effectiveness of our method quantitatively.

6. Conclusions

In this study, we proposed a novel visualization-aided dimensionality reduction technique based on the Gaussian process. Our proposed method derives the latent representation using an expressive kernel function and by regularization based on a graph Laplacian retaining the local structure of data. Furthermore, we introduced a sparse method and realized scalable optimization with the lower bound of the log-posterior distribution. Our method preserves the global structure of data with a Gaussian-process-based formulation and the local structure with a locally estimated data manifold. In the experiment, we validated our method on multiple datasets and confirmed the effectiveness qualitatively and quantitatively. However, the visibility of low-dimensional representations is better with graph embedding methods, and the PCA-based methods have low scalability. In future work, we will introduce the local information and scalable optimization in a more efficient way to improve its applicability to visualization-aided dimensionality reduction.