Ensuring Topological Data-Structure Preservation under Autoencoder Compression Due to Latent Space Regularization in Gauss–Legendre Nodes

Abstract

We formulate a data-independent latent space regularization constraint for general unsupervised autoencoders. The regularization relies on sampling the autoencoder Jacobian at Legendre nodes, which are the centers of the Gauss–Legendre quadrature. Revisiting this classic allows us to prove that regularized autoencoders ensure a one-to-one re-embedding of the initial data manifold into its latent representation. Demonstrations show that previously proposed regularization strategies, such as contractive autoencoding, cause topological defects even in simple examples, as do convolutional-based (variational) autoencoders. In contrast, topological preservation is ensured by standard multilayer perceptron neural networks when regularized using our approach. This observation extends from the classic FashionMNIST dataset to (low-resolution) MRI brain scans, suggesting that reliable low-dimensional representations of complex high-dimensional datasets can be achieved using this regularization technique.

1. Introduction

Systematic analysis and post-processing of high-dimensional and high-throughput datasets [1,2] is a current computational challenge across disciplines such as neuroscience [3,4,5], plasma physics [6,7,8], and cell biology and medicine [9,10,11,12]. In the machine learning (ML) community, autoencoders (AEs) are commonly considered the central tool for learning a low-dimensional one-to-one representation of high-dimensional datasets. These representations serve as a baseline for feature selection and classification tasks, which are prevalent in bio-medicine [13,14,15,16,17].

AEs can be considered as a non-linear extension of classic principal component analysis (PCA) [18,19,20]. Comparisons for linear problems are provided in [21]. While addressing the non-linear case, AEs face the challenge of preserving the topological data structure under AE compression.

To state the problem: We mathematically formalize AEs as pairs of continuously differentiable maps $(\phi ,\nu )$, $\phi :{\Omega }_{m_2}⟶{\Omega }_{m_1}$, $\nu :{\Omega }_{m_1}⟶{\Omega }_{m_2}$, $0<m_1<m_2\in \mathbb{N}$, defined on bounded domains ${\Omega }_{m_1}\subseteq {\mathbb{R}}^{m_1}$ and ${\Omega }_{m_2}\subseteq {\mathbb{R}}^{m_2}$. Commonly, $\phi$ is termed the encoder, and $\nu$ the decoder. We assume that the data $D\subseteq \mathcal{D}$ is sampled from a regular or even smooth data manifold $\mathcal{D}\subseteq {\Omega }_{m_2}$, with $\dim \mathcal{D}=m_0\le m_1$.

We seek to find proper AEs $(\phi ,\nu )$ yielding homeomorphic latent representations $\phi \left(D\right)={\mathcal{D}}^{\prime }\cong \mathcal{D}$. In other words, the restrictions ${\phi }_{|\mathcal{D}}:\mathcal{D}⟶{\mathcal{D}}^{\prime }$ and ${\nu }_{|{\mathcal{D}}^{\prime }}:{\mathcal{D}}^{\prime }⟶\mathcal{D}$ of the encoder and decoder result in one-to-one maps, being inverse to each other:

$$\mathcal{D}\cong {\mathcal{D}}^{\prime }=\phi \left(\mathcal{D}\right)\subseteq {\Omega }_{m_1},\nu \left(\phi \left(x\right)\right)=x,\mathrm{for}\mathrm{all}x\in \mathcal{D}.$$

(1)

While the second condition in Equation (1) is usually realized by minimization of a reconstruction loss, this is insufficient for guaranteeing the one-to-one representation $\mathcal{D}\cong {\mathcal{D}}^{\prime }$.

To realize AEs matching both requirements in Equation (1), we strengthen the condition by requiring the decoder to be an embedding of the whole latent domain ${\Omega }_{m_1}\supset \mathcal{D}$, including $\nu \left({\Omega }_{m_1}\right)\supset \mathcal{D}$ in its interior. See Figure 1 for an illustration. We mathematically prove and empirically demonstrate this latent regularization strategy to deliver regularized AEs (AR-REG), satisfying Equation (1).

Our investigations are motivated by recent results of Hansen et al. [22,23,24], complementing other contributions [25,26,27] that investigate instabilities of machine learning methods from a general mathematical perspective.

1.1. The Inherent Instability of Inverse Problems

The instability phenomenon of inverse problems states that, in general, one cannot guarantee solutions of inverse problems to be stable. An excellent introduction to the topic is given in [22] with deeper treatments and discussions in [23,24].

In our setup, these insights translate to the fact that, in general, the local Lipschitz constant

$$L_{\epsilon }(\nu ,y)=\underset{0<\parallel y^{\prime }-y\parallel <\epsilon }{\sup }{\displaystyle \frac{\parallel \nu \left(y^{\prime }\right)-\nu \left(y\right)\parallel }{\parallel y^{\prime }-y\parallel }},\epsilon >0$$

of the decoder $\nu :{\Omega }_{m_1}⟶{\Omega }_{m_2}$ at some latent code $y\in {\Omega }_1$ might be unbounded. Consequently, small perturbations $y^{\prime }\approx y$ of the latent code can result in large differences of the reconstruction $\parallel \nu \left(y^{\prime }\right)-\nu \left(y\right)\parallel \gg 0$. This fact generally applies and can only be avoided if an additional control on the null space of the Jacobian of the encoder $\ker J\left(\phi \right(x\left)\right)$ is given. Providing this control is the essence of our contribution.

1.2. Contribution

Avoiding the aforementioned instability, requires the null space of the Jacobian of the encoder $\ker J\left(\phi \right)$ to be perpendicular to the tangent space of $\mathcal{D}$

$$\ker J\left(\phi \right)\perp T\mathcal{D}.$$

(2)

In fact, due to the inverse function theorem, see, e.g., [28,29], the conditions Equations (1) and (2) are equivalent. In Figure 1, $\ker J\left(\phi \right)$ is illustrated to be perpendicular to the image of the whole latent domain ${\Omega }_{m_1}$

$$\ker J\left(\phi \right)\perp T\nu \left({\Omega }_{m_1}\right),\nu \left({\Omega }_{m_1}\right)\supseteq \mathcal{D},$$

being sufficient for guaranteeing Equation (2), and consequently, Equation (1).

While several state-of-the-art AE regularization techniques are commonly established, none of them specifically formulates this necessary mathematical requirement in Equation (2). Consequently, we are not aware of any regularization approach that can theoretically guarantee the regularized AE to preserve the topological data-structure, as we do in Theorem 1. Our computational contributions split into:

(C1) 

For realising a latent space regularized AE (AE-REG) we introduce the $L^2$-regularization loss

$${\mathcal{L}}_{\mathrm{reg}}(\phi ,\nu )={\parallel J(\phi \circ \nu )-I\parallel }_{L^2\left({\Omega }_{m_1}\right)}^2$$

(3)

and mathematically prove the AE-REG to satisfy condition Equation (1), Theorem 1, when being trained due to this additional regularization.

(C2) 

To approximate ${\mathcal{L}}_{\mathrm{reg}}(\phi ,\nu )$ we revisit the classic Gauss–Legendre quadratures (cubatures) [30,31,32,33,34], only requiring sub-sampling of $J(\nu \circ \phi )\left(p_{\alpha }\right)$, $p_{\alpha }\in P_{m,n}$ on a Legendre grid of sufficient high resolution $1\ll |P_{m,n}|$ in order to execute the regularization. While the data-independent latent Legendre nodes $P_{m,n}\subseteq {\Omega }_{m_1}$ are contained in the smaller dimensional latent space, regularization of high resolution can be efficiently realised.

(C3) 

Based on our prior work [35,36,37], and [38,39,40,41], we complement the regularization through a hybridisation approach combining autoencoders with multivariate Chebyshev-polynomial-regression. The resulting Hybrid AE is acting on the polynomial coefficient space, given by pre-encoding the training data due to high-quality regression.

We want to emphasize that the proposed regularization is data-independent in the sense that it does not require any prior knowledge of the data manifold, its embedding, or any parametrization of $\mathcal{D}$. Moreover, while being integrated into the loss function, the regularization is independent of the AE architecture and can be applied to any AE realizations, such as convolutional or variational AEs. Our results show that already regularized MLP-based AEs perform superior to these alternatives.

As we demonstrate, the regularization yields the desired re-embedding, enhances the autoencoder’s reconstruction quality, and increases robustness under noise perturbations.

1.3. Related Work—Regularization of Autoencoders

A multitude of supervised learning schemes, addressing representation learning tasks, are surveyed in [42,43]. Self-supervised autoencoders rest on inductive bias learning techniques [44,45] in combination with vectorized autoencoders [46,47]. However, the mathematical requirements, Equations (1) and (2) were not considered in these strategies at all. Consequently, one-to-one representations might only be realized due to a well-chosen inductive bias regularization for rich datasets [9].

This article focus on regularization techniques of purely unsupervised AEs. We want to mention the following prominent approaches:

(R1) 

Contractive AEs (ContraAE) [48,49] are based on an ambient Jacobian regularization loss

$${\mathcal{L}}_{\mathrm{reg}}^{\ast }(\phi ,\nu )={\parallel J(\nu \circ \phi )-I\parallel }_{L^2\left({\Omega }_{m_2}\right)}^2$$

(4)

formulated in the ambient domain. This makes contraAEs arbitrarily contractive in perpendicular directions ${\left(T\mathcal{D}\right)}^{\perp }$ of $T\mathcal{D}$. However, this is insufficient to guarantee Equation (1). In addition, the regularization is data dependent, resting on the training dataset, and is computationally costly due to the large Jacobian $J\in {\mathbb{R}}^{m_2\times m_2}$, $m_2\gg m_1\ge 1$. Several experiments in Section 5 demonstrate contraAE failing to deliver topologically preserved representations.

(R2) 

Variational AEs (VAE), along with extensions like $\beta$-VAE, consist of stochastic encoders and decoders and are commonly used for density estimation and generative modelling of complex distributions based on minimisation of the Evidence Lower Bound (ELBO) [50,51]. The variational latent space distribution induces an implicit regularization, which is complemented by [52,53] due to a $l_1$-sparsity constraint of the decoder Jacobian.

However, as the contraAE-constraint, this regularization is computationally costly and insufficient for guaranteeing a one-to-one encoding, which is reflected in the degenerated representations appearing in Section 5.

(R3) 

Convolutional AEs (CNN-AE) are known to deliver one-to-one representations for a generic setup theoretically [54]. However, the implicit convolutions seems to prevent clear separation of tangent $T\mathcal{D}$ and perpendicular direction ${\left(T\mathcal{D}\right)}^{\perp }$ of the data manifold $\mathcal{D}$, resulting in topological defects already for simple examples, see Section 5.

2. Mathematical Concepts

We provide the mathematical notions on which our approach rests, starting by fixing the notation.

2.1. Notation

We consider neural networks (NNs) $\nu (·,w)$ of fixed architecture ${\Xi }_{m_1,m_2}$, specifying number and depth of the hidden layers, the choice of piece-wise smooth activation functions $\sigma \left(x\right)$, e.g., ReLU or sin, with input dimension $m_1$ and output dimension $m_2$. Further, ${{\rm Y}}_{{\Xi }_{m_1,m_2}}$ denotes the parameter space of the weights and bias $w=(v,b)\in W=V\times B\subseteq {\mathbb{R}}^K$, $K\in \mathbb{N}$, see, e.g., [55,56].

We denote with ${\Omega }_m={(-1,1)}^m$ the m-dimensional open standard hypercube, with $\parallel ·\parallel$ the standard Euclidean norm on ${\mathbb{R}}^m$ and with ${\parallel ·\parallel }_p$, $1\le p\le \infty$ the $l_p$-norm. ${\Pi }_{m,n}=\mathrm{span}{\left\{x^{\alpha }\right\}}_{{\parallel \alpha \parallel }_{\infty }\le n}$ denotes the $\mathbb{R}$-vector space of all real polynomials in m variables spanned by all monomials $x^{\alpha }={\prod }_{i=1}^m{x}_i^{{\alpha }_i}$ of maximum degree $n\in \mathbb{N}$ and $A_{m,n}=\{\alpha \in {\mathbb{N}}^m:{\parallel \alpha \parallel }_{\infty }={\max }_{i=1,\dots ,m}\left\{\right|{\alpha }_i\left|\right\}\le n\}$ the corresponding multi-index set. For an excellent overview on functional analysis we recommend [57,58,59]. Here, we consider the Hilbert space $L^2({\Omega }_m,\mathbb{R})$ of all Lebesgue measurable functions $f:{\Omega }_m⟶\mathbb{R}$ with finite $L^2$-norm ${\parallel f\parallel }_{L^2\left({\Omega }_m\right)}^2<\infty$ induced by the inner product

$$<f,g{>}_{L^2\left({\Omega }_m\right)}=\underset{{\Omega }_m}{\int }f·gd{\Omega }_m,f,g\in L^2(\Omega ,\mathbb{R}).$$

(5)

Moreover, $C^k({\Omega }_m,\mathbb{R})$, $k\in \mathbb{N}\cup \{\infty \}$ denotes the Banach spaces of continuous functions being k-times continuously differentiable, equipped with the norm

$${\parallel f\parallel }_{C^k\left({\Omega }_m\right)}=\sum _{i=0}^k\underset{x\in {\Omega }_m}{\sup }|D^{\alpha }f(x)|,{\parallel \alpha \parallel }_1\le k.$$

2.2. Orthogonal Polynomials and Gauss–Legendre Cubatures

We follow [30,31,32,33,60] for recapturing: Let $m,n\in \mathbb{N}$ and $P_{m,n}={\oplus }_{i=1}^m{\mathrm{Leg}}_{\mathrm{n}}\subseteq {\Omega }_m$ be the m-dimensional Legendre grids, where ${\mathrm{Leg}}_{\mathrm{n}}=\{p_0,\dots ,p_n\}$ are the $n+1$ Legendre nodes given by the roots of the Legendre polynomials of degree $n+2$. We denote $p_{\alpha }=(p_{{\alpha }_1},\dots ,p_{{\alpha }_m})\in P_{A_{m,n}}$, $\alpha \in A_{m,n}$. The Lagrange polynomials $L_{\alpha }\in {\Pi }_{A_{m,n}}$, defined by $L_{\alpha }\left(p_{\beta }\right)={\delta }_{\alpha ,\beta }$, $\forall \alpha ,\beta \in A_{m,n}$, where ${\delta }_{·,·}$ denotes the Kronecker delta, are given by

$$L_{\alpha }=\prod _{i=1}^m{l}_{{\alpha }_i,i},l_{j,i}=\prod _{j\ne i,j=0}^m{\displaystyle \frac{x_i-p_j}{p_i-p_j}}.$$

(6)

Indeed, the $L_{\alpha }$ are an orthogonal $L^2$-basis of ${\Pi }_{m,n}$,

$${〈L_{\alpha },L_{\beta }〉}_{L^2\left({\Omega }_m\right)}=\underset{{\Omega }_m}{\int }{L}_{\alpha }\left(x\right)L_{\beta }\left(x\right)d{\Omega }_m=w_{\alpha }{\delta }_{\alpha ,\beta },$$

(7)

where the Gauss–Legendre cubature weight $w_{\alpha }={\parallel L_{\alpha }\parallel }_{L^2\left({\Omega }_m\right)}^2$ can be computed numerically. Consequently, for any polynomial $Q\in {\Pi }_{m,2n+1}$ of degree $2n+1$ the following cubature rule applies:

$$\underset{{\Omega }_m}{\int }Q\left(x\right)d{\Omega }_m=\sum _{\alpha \in A_{m,n}}{w}_{\alpha }Q\left(p_{\alpha }\right).$$

(8)

Thanks to $|P_{m,n}{|=(n+1)}^m\ll {(2n+1)}^m$ this makes Gauss–Legendre integration a very powerful scheme, yielding

$${〈Q_1,Q_2〉}_{L^2\left({\Omega }_m\right)}=\sum _{\alpha \in A_{m,n}}{Q}_1\left(p_{\alpha }\right)Q_2\left(p_{\alpha }\right)w_{\alpha },$$

(9)

for all $Q_1,Q_2\in {\Pi }_{m,n}$.

In light of this fact, we propose the following AE regularization method.

3. Legendre-Latent-Space Regularization for Autoencoders

The regularization is formulated from the perspective of classic differential geometry, see, e.g., [28,61,62,63]. As introduced in Equation (1), we assume that the training data $D_{\mathrm{train}}\subseteq \mathcal{D}\subseteq {\mathbb{R}}^{m_2}$ is sampled from a regular data manifold. We formalise the notion of autoencoders:

Definition 1

(autoencoders and data manifolds). Let $1\le m_0\le m_1\le m_2\in \mathbb{N}$, $\mathcal{D}\subseteq {\Omega }_{m_2}$ be a (data) manifold of dimension $\dim \mathcal{D}=m_0$. Given continuously differentiable maps $\phi :{\Omega }_{m_2}⟶{\Omega }_{m_1}$, $\nu :{\Omega }_{m_1}⟶{\Omega }_{m_2}$ such that:

(i) 

ν is a right-inverse of φ on $\mathcal{D}$, i.e, $\nu \left(\phi \right(x\left)\right)=x$ for all $x\in \mathcal{D}$.

(ii) 

φ is a left-inverse of ν, i.e, $\phi \left(\nu \right(y\left)\right)=y$ for all $y\in {\Omega }_{m_1}$

Then we call the pair $(\phi ,\nu )$ a proper autoencoder with respect to $\mathcal{D}$.

Given a proper AE $(\phi ,\nu )$, $\phi$ yields a low dimensional homeomorphic re-embedding of $\mathcal{D}\cong {\mathcal{D}}^{\prime }=\phi \left(\mathcal{D}\right)\subseteq {\mathbb{R}}^{m_1}$ as demanded in Equation (1) and illustrated in Figure 1, fulfilling the stability requirement of Equation (2).

We formulate the following losses for deriving proper AEs:

Definition 2

(regularization loss). Let $\mathcal{D}\subseteq {\Omega }_{m_2}$ be a $C^1$-data manifold of dimension $\dim \mathcal{D}=m_0<m_1<m_2$ and $\varnothing \ne D_{\mathrm{train}}\subseteq \mathcal{D}$ be a finite training dataset. For NNs $\phi (·,u)\in {\Xi }_{m_2,m_1}$, $\nu (·,w)\in {\Xi }_{m_1,m_2}$ with weights $(u,w)\in {{\rm Y}}_{{\Xi }_{m_2,m_1}}\times {{\rm Y}}_{{\Xi }_{m_1,m_2}}$, we define the loss

$$\begin{array}{c}\hfill {\mathcal{L}}_{D_{\mathrm{train}},n}:{{\rm Y}}_{{\Xi }_{m_2,m_1}}\times {{\rm Y}}_{{\Xi }_{m_1,m_2}}⟶{\mathbb{R}}^{+},{\mathcal{L}}_{D_{\mathrm{train}},n}(u,w)={\mathcal{L}}_0(D_{\mathrm{train}},u,w)+\lambda {\mathcal{L}}_1(u,w,n),\end{array}$$

where $\lambda >0$ is a hyper-parameter and

$$\begin{array}{c}\hfill {\mathcal{L}}_0(D_{\mathrm{train}},u,w)=\sum _{x\in D_{\mathrm{train}}}{\parallel x-\nu (\phi (x,u),w)\parallel }^2\end{array}$$

(10)

$$\begin{array}{c}\hfill {\mathcal{L}}_1(u,w,n)=\sum _{\alpha \in A_{m_1,n}}{\parallel I-J\left(\phi (\nu (p_{\alpha },w),u)\right)\parallel }^2,\end{array}$$

(11)

with $I\in {\mathbb{R}}^{m_1\times m_1}$ denoting the identity matrix, $p_{\alpha }\in P_{m_1,n}$ be the Legendre nodes, and $J\left(\phi (\nu (p_{\alpha },w)\right)\in {\mathbb{R}}^{m_1\times m_1}$ the Jacobian.

We show that the AEs with vanishing loss result to be proper AEs, Defintion 1.

Theorem 1

(Main Theorem). Let the assumptions of Definition 2 be satisfied, and $\phi (·,u_n)\in {\Xi }_{m_2,m_1}$, $\nu (·,w_n)\in {\Xi }_{m_1,m_2}$ be sequences of continuously differentiable NNs satisfying:

(i) 

The loss converges ${\mathcal{L}}_{D_{\mathrm{train}},n}(u_n,w_n)\underset{n\to \infty }{\to }0$.

(ii) 

The weight sequences converge

$$\underset{n\to \infty }{\lim }(u_n,w_n)=(u_{\infty },w_{\infty })\in {{\rm Y}}_{{\Xi }_{m_2,m_1}}\times {{\rm Y}}_{{\Xi }_{m_1,m_2}}.$$

(iii) 

The decoder satisfies $\nu ({\Omega }_{m_1},w_n)\supseteq \mathcal{D}$, $\forall n\ge n_0\in \mathbb{N}$ for some $n_0\ge 1$.

Then $(\phi (·,w_n),\nu (·,u_n))\underset{n\to \infty }{\to }(\phi (·,w_{\infty }),\nu (·,u_{\infty }))$ uniformly converges to a proper autoencoder with respect to $\mathcal{D}$.

Proof. 

The proof follows by combining several facts: First, the inverse function theorem [29] implies that any map $\rho \in C^1({\Omega }_m,{\Omega }_m)$ satisfying

$$J\left(\rho \left(x\right)\right)=I,\forall x\in {\Omega }_m,\mathrm{and}\rho \left(x_0\right)=x_0,$$

(12)

for some $x_0\in {\Omega }_m$ is given by the identity, i.e., $\rho \left(x\right)=x$, $\forall x\in {\Omega }_m$.

Secondly, the Stone–Weierstrass theorem [64,65] states that any continuous map $\rho \in C^0({\Omega }_m,{\Omega }_m)$, with coordinate functions $\rho \left(x\right)=({\rho }_1\left(x\right),\dots ,{\rho }_m\left(x\right))$ can be uniformly approximated by a polynomial map $Q_{\rho }^n\left(x\right)=(Q_{\rho ,1}^n\left(x\right),\dots ,Q_{\rho ,m}^n\left(x\right))$, $Q_{\rho ,i}^n\left(x\right)\in {\Pi }_{m,n}$, $1\le i\le m$, i.e, $\parallel \rho -Q_{\rho }^n{\parallel }_{C^0\left({\Omega }_m\right)}\underset{n\to \infty }{\to }0$.

Thirdly, while the NNs $\phi (·,w)$, $\nu (·,u)$ depend continuously on the weights $u,w$, the convergence in $\left(ii\right)$ is uniform. Consequently, the convergence ${\mathcal{L}}_{D_{\mathrm{train}},n}(u_n,w_n)\underset{n\to \infty }{\to }0$ of the loss implies that any sequence of polynomial approximations $Q_{\rho }^n\left(x\right)$ of the map $\rho (·)=\phi (\nu (·,w_{\infty }),u_{\infty })$ satisfies

$$\sum _{\alpha \in A_{m_1,n}}{\parallel I-J\left(Q_{\rho }^n\left(p_{\alpha }\right)\right)\parallel }^2=0$$

in the limit for $n\to \infty$. Hence, Equation (12) holds in the limit for $n\to \infty$ and consequently $\phi (\nu (y,w_{\infty }),u_{\infty })=Q_{\rho }^{\infty }\left(y\right)=y$ for all $y\in {\Omega }_{m_1}$ yielding requirement $\left(ii\right)$ of Definition 1.

Given that assumption $\left(iii\right)$ is satisfied, in completion, requirement $\left(i\right)$ of Definition 1 holds, finishing the proof. □

Apart from ensuring topological maintenance, one seeks for high-quality reconstructions. We propose a novel hybridization approach, delivering both.

4. Hybridization of Autoencoders Due to Polynomial Regression

The hybridisation approach rests on deriving Chebyshev Polynomial Surrogate Models $Q_{\Theta ,d}$ fitting the initial training data $d\in D_{\mathrm{train}}\subseteq {\Omega }_{m_2}$. For the sake of simplicity, we motivate the setup in case of images:

Let $d={\left(d_{ij}\right)}_{1\le i,j\le r}\in {\mathbb{R}}^{r\times r}$ be the intensity values of an image on an equidistant pixel grid $G_{r\times r}={\left(g_{ij}\right)}_{1\le i,j\le r}\subseteq {\Omega }_2$ of resolution $r\times r$, $r\in \mathbb{N}$. We seek for a polynomial

$$Q_{\Theta }:{\Omega }_2⟶\mathbb{R},Q_{\Theta }\in {\Pi }_{2,n},$$

such that evaluating $Q_{\Theta }$, $\Theta ={\left({\theta }_{\alpha }\right)}_{\alpha \in A_{2,n}}\in {\mathbb{R}}^{|A_{2,n}|}$ on $G_{r\times r}$ approximates d, i.e., $Q_{\Theta }\left(g_{ij}\right)\approx d_{ij}$ for all $1\le i,j\le r$. We model $Q_{\Theta }$ in terms of Chebyshev polynomials of first kind well known to provide excellent approximation properties [33,35]:

$$Q_{\Theta }(x_1,x_2)=\sum _{\alpha \in A_{2,n}}{\theta }_{\alpha }{T}_{{\alpha }_1}\left(x_1\right)T_{{\alpha }_2}\left(x_2\right).$$

(13)

The expansion is computed due to standard least-square fits:

$${\Theta }_d={\mathrm{argmin}}_{C\in {\mathbb{R}}^{|A_{2,n}|}}{\parallel RC-d\parallel }^2,$$

(14)

where $R={\left(T_{\alpha }\left(g_{ij}\right)\right)}_{1\le i,j,\le n,\alpha \in A_{2,n}}\in {\mathbb{R}}^{r^2\times \left|A_{2,n}\right|}$, $T_{\alpha }=T_{{\alpha }_1}·T_{{\alpha }_2}$ denotes the regression matrix.

Given that each image (training point) $d\in D_{\mathrm{train}}$ can be approximated with the same polynomial degree $n\in \mathbb{N}$, we posterior train an autoencoder $\phi ,\nu$, only acting on the polynomial coefficient space $\phi :{\mathbb{R}}^{|A_{2,n}|}⟶{\Omega }_{m_1}$, $\nu {\Omega }_{m_1}⟶{\mathbb{R}}^{|A_{2,n}|}$ by exchanging the loss in Equation (10) due to

$${\mathcal{L}}_0^{\ast }(D_{\mathrm{train}},u,w)=\sum _{d\in D_{\mathrm{train}}}{\parallel d-R·\nu (\phi ({\Theta }_d,u),w)\parallel }^2$$

(15)

In contrast to the regularization loss in Definition 2, here, pre-encoding the training data due to polynomial regression decreases the input dimension $m_2\in \mathbb{N}$ of the (NN) encoder $\phi :{\Omega }_{m_2}⟶{\Omega }_{m_1}$. In practice, this enables to reach low dimensional latent dimension by increasing the reconstruction quality, as we demonstrate in the next section.

5. Numerical Experiments

We executed experiments, designed to validate our theoretical results, on hemera a NVIDIA V100 cluster at HZDR. Complete code benchmark sets and supplements are available at https://github.com/casus/autoencoder-regularisation, accessed on 2 June 2024. The following AEs were applied:

(B1) 

Multilayer perceptron autoencoder (MLP-AE): Feed forward NNs with activation functions $\sigma \left(x\right)=\sin \left(x\right)$.

(B2) 

Convolutional autoencoder (CNN-AE): Standard convolutional neural networks (CNNs) with activation functions $\sigma \left(x\right)=\sin \left(x\right)$, as discussed in (R3).

(B3) 

Variational autoencoder: MLP based (MLP-VAE) and CNN based (CNN-VAE) as in [50,51], discussed in (R2).

(B4) 

Contractive autoencoder (ContraAE): MLP based with with activation functions $\sigma \left(x\right)=\sin \left(x\right)$ as in [48,49], discussed in (R1).

(B5) 

regularized autoencoder (AE-REG): MLP based, as in (B1), trained with respect to the regularization loss from Definition 2.

(B6) 

Hybridised AE (Hybrid AE-REG): MLP based, as in (B1), trained with respect to the modified loss in Definition 2 due to Equation (15).

The choice of activation functions $\sigma \left(x\right)=\sin \left(x\right)$ yields a natural way for normalizing the latent encoding to ${\Omega }_m$ and performed best compared to trials with ReLU, ELU or $\sigma \left(x\right)=\tanh \left(x\right)$. The regularization of AE-REG and Hybrid AE-REG is realized due to sub-sampling batches from the Legendre grid $P_{m,n}$ for each iteration and computing the autoencoder Jacobians due to automatic differentiation [66].

5.1. Topological Data-Structure Preservation

Inspired by Figure 1, we start by validating Theorem 1 for known data manifold topologies.

Experiment 1

(Cycle reconstructions in dimension 15). We consider the unit circle $S^1\subseteq {\mathbb{R}}^2$, a uniform random matrix $A\in {\mathbb{R}}^{15,2}$ with entries in $[-2,2]$ and the data manifold $\mathcal{D}=\{Ax:x\in S^1\subseteq {\mathbb{R}}^2\}$, being an ellipse embedded along some 2-dimensional hyperplane $H_A=\{Ax:x\in {\mathbb{R}}^2\}\subseteq {\mathbb{R}}^{15}$. Due to Bezout’s Theorem [67,68], a 3-points sample uniquely determines a circle in the 2-dimensional plane. Therefore, we executed the AEs for this minimal case of a set of random samples $|D_{\mathrm{train}}|=3$, $D_{\mathrm{train}}\subseteq \mathcal{D}$ as training set.

MLP-AE, MLP-VAE, and AE-REG consists of 2 hidden linear layers (in the encoder and decoder), each of length 6. The degree of the Legendre grid $P_{m,n}$ used for the regularization of AE-REG was set to $n=21$, Definition 2. CNN-AE and CNN-VAE consists of 2 hidden convolutional layers with kernel size 3, stride of 2 in the first hidden layer and 1 in the second, and 5 filters per layer. The resulting parameter spaces ${{\rm Y}}_{{\Xi }_{15,2}}$ are all of similar size: $\left|{{\rm Y}}_{{\Xi }_{15,2}}\right|\sim 400$. All AEs were trained with the Adam optimizer [69].

Representative results out of 6 repetitions are shown in Figure 2. Only AE-REG delivers a feasible 2D re-embedding, while all other AEs cause overlappings or cycle-crossings. More examples are given in the supplements; whereas AE-REG delivers similar reconstructions for all other trials while the other AEs fail in most of the cases.

Linking back to our initial discussions of ContraAE (R1): The results show that the ambient domain regularization formulated for the ContraAE, is insufficient for guaranteeing a one-to-one encoding. Similarily, CNN-based AEs cause self-intersecting points. As initially discussed in (R3), CNNs are invertible for a generic setup [54], but seem to fail sharply separating tangent $T\mathcal{D}$ and perpendicular direction $T{\mathcal{D}}^{\perp }$ of the data manifold $\mathcal{D}$.

We demonstrate the impact of the regularization to not belonging to an edge case by considering the following scenario:

Experiment 2

(Torus reconstruction). Following the experimental design of Experiment 1 we generate challenging tori embeddings of a squeezed torus with radii $0<r,R$, $r=0.7$ $R=2.0$ in dimension $m=15$ and dimension $m=1024$ due to multiplication with random matrices $A\in {[-1,1]}^{m\times 3}$. We randomly sample 50 training points $|D_{\mathrm{train}}|=50$, $D_{\mathrm{train}}\subseteq \mathcal{D}$ and seek for their 3D re-embedding due to the AEs. We choose a Legendre grid $P_{m,n}$ of degree $n=21$.

A show-case is given in Figure 3, visualized by a dense set of 2000 test points. As in Experiment 1 only AE-REG is capable for re-embedding the torus in a feasible way. MLP-VAE, CNN-AE and CNN-VAE flatten the torus, ContraAE and MPL-AE cause self-intersections. Similar results occur for the high-dimensional case $m=1024$, see the supplements. Summarizing the results suggests that without regularization AE-compression does not preserve the data topology. We continue our evaluation to give further evidence on this expectation.

5.2. Autoencoder Compression for FashionMNIST

We continue by benchmarking on the the classic FashionMNIST dataset [70].

Experiment 3

(FashionMNIST compression). The 70,000 FashionMNIST images separated into 10 fashion classes (T-shirts, shoes, etc.) being of $32\times 32=1024$-pixel resolution (ambient domain dimension). For providing a challenging competition, we reduced the dataset to 24,000 uniformly sub-sampled images and trained the AEs for $40\%$ training data and complementary test data, respectively. Here, we consider latent dimensions $m=4,10$. Results of further runs for $m=2,4,6,8,10$ are given in the supplements.

MLP-AE, MLP-VAE, AE-REG and Hybrid AE-REG consists of 3 hidden layers, each of length 100. The degree of the Legendre grid $P_{m,n}$ used for the regularization of AE-REG was set to $n=21$, Definition 2. CNN-AE and CNN-VAE consists of 3 convolutional layers with kernel size 3, stride of 2. The resulting parameter spaces ${{\rm Y}}_{{\Xi }_{15,2}}$ of all AEs are of similar size. Further details of the architectures are reported in the supplements.

We evaluated the reconstruction quality with respect to peak signal-to-noise ratios (PSNR) for perturbed test data due $0\%,10\%,20\%,50\%$ of Gaussian noise encoded to latent dimension $m=10$, and plot them in Figure 4. The choice $m=10$, here, reflects the number of FashionMNIST-classes.

While Hybrid AE-REG performs compatible to MLP-AE and worse than the other AEs in the non-perturbed case, its superiority appears already for perturbations with $10\%$ of Gaussian noise and exceeds the reached reconstruction quality of all other AEs for $20\%$ Gaussian noise or more. We want to stress that Hybrid AE-REG maintains its reconstruction quality throughout the noise perturbations (up to $70\%$, see the supplements). This outstanding appearance of robustness gives strong evidence on the impact of the regularization and well-designed pre-encoding technique due to the hybridization with polynomial regression. Analogue results appear when measuring the reconstruction quality with respect to the structural similarity index measure (SSIM), given in the supplements.

In Figure 5, show cases of the reconstructions are illustrated, including additional vertical and horizontal flip perturbations. Apart from AE-REG and Hybrid AE-REG (rows (7) and (8)), all other AEs flip the FashionMNIST label-class for reconstructions of images with $20\%$ or $50\%$ of Gaussian noise. Flipping the label-class is the analogue to topological defects as cycle crossings appeared for the non-regularized AEs in Experiment 1, indicating again that the latent representation of the FashionMNIST dataset given due to the non-regularized AEs does not maintain structural information.

While visualization of the FashionMNIST data manifold is not possible, we decided to investigate its structure by computing geodesics. Figure 6 provides show cases of decoded latent-geodesics $\nu \left(\gamma \right)$ with respect to latent dimension $m=4$, connecting two AE-latent codes of the encoded test data that has been initially perturbed by $50\%$ Gaussian noise before encoding. The latent-geodesics $\gamma$ have to connect the endpoints along the curved encoded data manifold ${\mathcal{D}}^{\prime }=\phi \left(\mathcal{D}\right)$ without forbidden short-cuts through ${\mathcal{D}}^{\prime }$. That is why the geodesics are computed as shortest paths for an early Vietoris–Rips filtration [71] that contains the endpoints in one common connected component. More examples are given in the supplements.

Apart from CNN-AE and AE-REG, all other geodesics contain latent codes of images belonging to another FashionMNIST-class, while for Hybrid AE-REG this happens just once. We interpret these appearances as forbidden short-cuts of $\nu \left(\gamma \right)$ through $\mathcal{D}$, caused by topological artefacts in ${\mathcal{D}}^{\prime }=\phi \left(\mathcal{D}\right)$.

AE-REG delivers a smoother transition between the endpoints than CNN-AE, suggesting that though the CNN-AE geodesic is shorter, the regularized AEs preserve the topology with higher resolution.

5.3. Autoencoder Compression for Low-Resolution MRI Brain Scans

For evaluating the potential impact of the hybridisation and regularization technique to more realistic high-dimensional problems, we conducted the following experiment.

Experiment 4

(MRI compression). We consider the MRI brain scans dataset from Open Access Series of Imaging Studies (OASIS) [72]. We extract two-dimensional slices from the three-dimensional MRI images, resulting in $\mathrm{60,000}$ images of resolution $91\times 91$-pixels. We follow Experiment 3 by splitting the dataset into $40\%$ training images and complementary test images and compare the AE compression for latent dimension $m=40$. Results for latent dimension $m=10,15,20,40,60,70$ and $5\%,20\%,40\%,80\%$ training data are given in the supplements, as well as further details on the specifications.

We keep the architecture setup of the AEs, but increase the NN sizes to 5 hidden layers each consisting of $1000$ neurons. Reconstructions measured by PSNR are evaluated in Figure 7. Analogous results appear for SSIM, see the supplements.

As in Experiment 3, we observe that AE-REG and Hybrid AE-REG perform compatible or slightly worse than the other AEs in the unperturbed scenario, but show their superiority over the other AEs for $10\%$ Gaussian noise, or $20\%$ for CNN-VAE. Hybrid AE-REG specifically maintains its reconstruction quality under noise perturbations up to $20\%$ (maintains stable for $50\%$). The performance increase compared to the un-regularized MLP-AE becomes evident and validates again that a strong robustness is achieved due to the regularization.

A show case is given in Figure 8. Apart from Hybrid AE-REG (row (8)), all AEs show artefacts when reconstructing perturbed images. CNN-VAE (row (4)) and AE-REG (row (7)) perform compatible and maintain stability up to $20\%$ Gaussian noise perturbation.

In Figure 9, examples of geodesics are visualized, being computed analogously as in Experiment 3 for the encoded images once without noise and once by adding $10\%$ Gaussian noise before encoding. The AE-REG geodesic consists of similar slices, including one differing slice for $10\%$ Gaussian noise perturbation. CNN-VAE delivers a shorter path; however, it includes a strongly differing slice, which is kept for $10\%$ of Gaussian noise. CNN-AE provides a feasible geodesic in the unperturbed case; however, it becomes unstable in the perturbed case.

We interpret the difference of the AE-REG to CNN-VAE and CNN-AE as an indicator for delivering consistent latent representations on a higher resolution. While the CNN-AE and AE-REG geodesics indicate that one may trust the encoded latent representations, the CNN-AE encoding may not be suitable for reliable post-processing, such as classification tasks. More showcases are given in the supplements, showing similar unstable behaviour of the other AEs.

Summarizing, the results validate once more regularization and hybridization to deliver reliable AEs that are capable for compressing datasets to low-dimensional latent spaces by preserving their topology. A feasible approach to extend the hybridization technique to images or datasets of high resolution is one of the aspects we discuss in our concluding thoughts.

6. Conclusions

We delivered a mathematical theory for addressing encoding tasks of datasets being sampled from smooth data manifolds. Our insights were condensed in an efficiently realizable regularization constraint, resting on sampling the encoder Jacobian in Legendre nodes, located in the latent space. We have proven the regularization to guarantee a re-embedding of the data manifold under mild assumptions on the dataset.

We want to stress that the regularization is not limited to specific NN architectures, but already strongly impacts the performance of simple MLPs. Combinations with initially discussed vectorised AEs [44,45] might extend and improve high-dimensional data analysis as in [9]. When combined with the proposed polynomial regression, the hybridised AEs increase strongly in reconstruction quality. For addressing images of high resolution or multi-dimensional datasets, $\dim \ge 3$, we propose to apply our recent extension of these regression methods [35].

In summary, the regularized AEs performed far better than the considered alternatives, especially with regard to maintaining the topological structure of the initial dataset. The present computations of geodesics provides a tool for analysing the latent space geometry encoded by the regularized AEs and contributes towards explainability of reliable feature selections, as initially emphasised [13,14,15,16,17].

While structural preservation is substantial for consistent post-analysis, we believe that the proposed regularization technique can deliver new reliable insights across disciplines and may even enable corrections or refinements of prior deduced correlations.