Efficient Hyperbolic Perceptron for Image Classification

Abstract

Deep neural networks, often equipped with powerful auto-optimization tools, find widespread use in diverse domains like NLP and computer vision. However, traditional neural architectures come with specific inductive biases, designed to reduce parameter search space, cut computational costs, or introduce domain expertise into the network design. In contrast, multilayer perceptrons (MLPs) offer greater freedom and lower inductive bias than convolutional neural networks (CNNs), making them versatile for learning complex patterns. Despite their flexibility, most neural architectures operate in a flat Euclidean space, which may not be optimal for various data types, particularly those with hierarchical correlations. In this paper, we move one step further by introducing the hyperbolic Res-MLP (HR-MLP), an architecture extending the attention-free MLP to a non-Euclidean space. HR-MLP leverages fully hyperbolic layers for feature embeddings and end-to-end image classification. Our novel Lorentz cross-patch and cross-channel layers enable direct hyperbolic operations with fewer parameters, facilitating faster training and superior performance compared to Euclidean counterparts. Experimental results on CIFAR10, CIFAR100, and MiniImageNet confirm HR-MLP’s competitive and improved performance.

1. Introduction

In recent years, significant advancements have been made in the development of neural network architectures with reduced hard-coded priors and inductive biases. Traditional approaches, such as manually designing fixed features, e.g., SIFT and LBP [1,2,3], have been replaced by automatic feature-learning strategies powered with more flexible architectures like convolutional neural networks (CNNs) [4]. Very recently, vision transformers (ViTs) [5] have further pushed the boundaries by eliminating hard-coded decisions, like translation invariance and local connectivity commonly found in CNNs. However, such models always have huge computational complexity, are extremely data-hungry, and lack interpretability as the complex interactions and attention patterns within the self-attention mechanism. At the same time, such models require higher expert knowledge for adaptively designing neural architectures [6,7] for different applications. All of these, coupled with improved training schemes, have facilitated the reemergence of purely multi-layer perceptron (MLP) architectures in computer vision tasks [8,9,10,11,12]. Previously, MLPs were largely overlooked or disregarded in computer vision due to their computationally intensive nature. However, recent developments [8,9] have made it feasible to harness the potential of MLPs while achieving promising trade-offs between accuracy and design complexity in image classification tasks; for instance, MLPs [8,9] have obtained very promising results when trained and evaluated on the large-scale dataset, ImageNet [13].

However, it is worth noting that these neural network architectures, including MLPs, predominantly operate within Euclidean space. Euclidean space represents the conventional generalization of our intuitive three-dimensional space but may not be the most suitable representation for all types of data [14]. For instance, complex datasets, such as social networks, human skeletons, sentences, and evolutionary relationships, often exhibit hierarchical structures, leading to big distortion when embedding such data in Euclidean space. Recognizing the limitations of Euclidean representation learning, researchers have explored alternative approaches that leverage hyperbolic spaces, which provide a natural fit for capturing the non-Euclidean nature of such data.

Hyperbolic deep learning [14], based on the negative curvature of hyperbolic spaces [15], has been successfully applied to various domains, including natural language processing (NLP) and graph analysis [16,17]. Hyperbolic embeddings have been employed for tasks such as text classification [18], entity typing [19], word embeddings [20,21,22], and some areas in graphs, such as node classification [17], graph classification [16,23], link prediction [17], and graph embeddings [24]. While it may not be immediately apparent that images possess a hierarchical data structure, several studies [23,25,26,27,28] have demonstrated the effective utilization of hyperbolic spaces for learning image embeddings.

Images, by their nature, often possess inherent hierarchical structures. For example, in an image of a person, there are hierarchies ranging from pixels to body parts (e.g., limbs, face), and finally, to the whole person. These hierarchical relationships are not easily modeled in a flat Euclidean space, which treats all distances equally. When using Euclidean geometry, important contextual information and relative hierarchical importance may be lost, making it challenging to recognize and understand complex image features. Prior research, such as by Khrulkov et al. (2020) [25], has compellingly demonstrated the presence of distinct hierarchical relationships within image features learned by widely adopted neural networks, like VGG. Simultaneously, it is worth noting that hierarchy and tree structures constitute common paradigms for human cognition in comprehending and recognizing the world. However, due to the challenges of extending hyperbolic neural networks to computer vision, most existing works [23,25] related to hyperbolic spaces in the context of computer vision rely on a hybrid framework, in which the Euclidean backbone is first applied to generate embeddings and then a hyperbolic layer is introduced to perform the final prediction. While this approach has shown promise, it still operates within a hard-to-train hybrid setting. Therefore, fully hyperbolic networks that leverage hyperbolic operations and layers to learn feature embeddings need to be explored in the domain.

Therefore, this research is motivated to bridge the gap, as we endeavor to address this deficiency by developing a fully hyperbolic neural network architecture without the usage of the tangent space, specifically tailored for computer vision tasks. By embracing the intrinsic hierarchical structures within images and harnessing the power of hyperbolic spaces, we expect that our proposed architecture will offer novel insights and potentially surpass traditional Euclidean-based models in certain scenarios. The contributions of this paper can be summarized as follows:

• A fully hyperbolic deep neural architecture for image tasks, which is called hyperbolic ResMLP (HR-MLP), is presented to explore the potential of the hyperbolic perceptron to high dimensional data.
• The proposed HR-MLP has a Lorentz cross-patch and cross-channel layer, which is a manifold-preserving neural operator.
• Results on CIFAR10, CIFAR100, and MiniImageNet demonstrate comparable and superior performance with their Euclidean counterpart while having much better interoperability.

2. Related Works

2.1. Image Classification in Euclidean Space

Image classification is the process of categorizing and labeling images, which are commonly represented by groups of pixels or vectors. In terms of the availability of the class label during training, there are generally supervised and unsupervised image classifications [29]. Image classification models, which can be various machine learning methods/architectures, take an image as input and return a prediction about which class the image belongs to.

Before the advent of deep neural networks, especially convolutional neural networks (CNNs), the traditional methods for image classification were heavily based on hand-crafted features, like the histogram of oriented gradients (HOGs), scale-invariant feature transform (SIFT), local binary patterns (LBPs) [1,2,3,30], or bag-of-visual-words (BoVWs) [31], to represent the input images. After that the feature will be further fed into a classifier, which can be a support vector machine (SVM) [32] or random forest [33], to predict the predefined labels. While traditional methods have been widely used for image classification tasks, they often face limitations in capturing the rich and hierarchical representations that cannot handle large-scale and complex datasets well. CNNs and other deep learning models have largely surpassed traditional methods in terms of accuracy and generalization, especially when applied to large-scale datasets like ImageNet [13]. At the deep learning age, numerous neural architectures and techniques are proposed to automatically learn feature representations from a dataset, and in most cases, feature learning and class prediction are combined in an end-to-end fashion. In terms of neural architectures, CNNs [4] are now some of the most commonly used networks for images, such as residual networks (ResNets) [34] (which introduce skip connections to alleviate the vanishing gradient problem), inception networks [35] (which utilize multiple parallel convolutional layers of different sizes to capture features at various scales, promoting the extraction of both fine-grained and high-level contextual information), DenseNet [36], squeeze-and-excitation networks (SENets) [37], and EfficientNet [38] (which proposes a compound scaling method to strike an optimal balance between model size and performance).

Recently, to further improve the representation ability of the neural network, self-attention mechanisms were introduced. Dosovitskiy et al. [5] proposed an attention-based method, termed vision transformers (ViTs). The models process images in a patch-based manner (in which the image is treated as a sequence of tokens) and apply the self-attention mechanism to capture global correlations. This approach has achieved remarkable results in various image classification benchmarks. Unlike this, residual multi-layer perceptrons (ResMLPs) [9] remove self-attention and are expected to provide a much more efficient way to learn rich features. In particular, they combine the strengths of MLPs and residual connections and give more flexibility to the learning process. These models achieve competitive performance in image classification tasks while maintaining the simplicity and interpretability of MLPs.

Parallel to the design of the neural architectures, new techniques are also presented to improve the task. For instance, data augmentation techniques [39], including different transformations, are commonly used to augment the training data and improve the generalization ability of image classifiers. Transfer learning techniques [40], such as pre-trained models, like VGG [41], ResNet [34], and InceptionNet [35], trained on large-scale datasets like ImageNet [13], have been widely adopted for image classification. By leveraging features learned from these models, transfer learning enables effective classification even with limited labeled data. Since designing neural architectures requires high expert knowledge, both for deep neural networks with non-trivial optimizations and knowledge for a specific domain (e.g., segmentation, or different modalities), the neural architecture search (NAS) [42] was introduced to automate the search for optimal network architectures for various datasets. NAS [42] has been utilized to discover novel architectures that achieve state-of-the-art performance in image classification tasks.

It is worth noting that all of the methods here are in Euclidean space. These works highlight the extensive research efforts and advancements in image classification within Euclidean space, leading to the development of highly effective and efficient neural network models for a wide range of applications. However, to give better interpretability, many attempts were made to construct neural networks in non-Euclidean space [23,25], while just turning to its tangent space, which has a huge space to improve.

2.2. Hyperbolic Deep Learning

Hyperbolic deep learning [14] is a new research field of deep learning, which aims to learn compact and rich feature representation, utilizing hyperbolic spaces as the underlying mathematical structure for representing and processing data. Unlike traditional deep learning methods that primarily operate in Euclidean spaces, hyperbolic deep learning leverages the unique properties of hyperbolic geometries to handle data with hierarchical structures, non-Euclidean relationships [16], and complex interconnectedness [14,17]. Hyperbolic deep learning has emerged as a promising paradigm in various domains, including graph analysis [16,17,23,43], natural language processing (NLP) [18,19,20,21], and other complex data structures. Complex hierarchical relationships are prevalent in social networks, biological networks, and other graph data [14]. By leveraging the negative curvature of hyperbolic spaces, in graph analysis, hyperbolic deep learning has shown significant success in tasks such as node classification [17], graph classification [16,23], and link prediction [17], demonstrating significant potential to its Euclidean counterparts. For, instance, in the field of NLP, hyperbolic embeddings could provide extremely compact embedding (even two-dimensional) [22], which has demonstrated its efficacy in tasks like text classification, entity typing, word embeddings, and sentence representations.

In recent years, hyperbolic embeddings have garnered attention in computer vision due to their ability to capture hierarchical relationships among data points. Several works have explored the application of hyperbolic learning in classical computer vision tasks.

One of the early attempts at utilizing hyperbolic embeddings for computer vision tasks comes from Khrulkov et al. [25]. They argue that hierarchical relations between images are common in computer vision tasks, such as image retrieval and classification. In their work, they showed that feature embeddings of popular architectures, including ResNet [34], VGG19 [41], and InceptionV3 [35], exhibit hyperbolicity on various datasets like CIFAR10, CIFAR100 [44], CUB [45], and MiniImageNet. However, their approach utilized hyperbolic layers along with Euclidean backbones, thus only partially leveraging the full potential of hyperbolic geometry.

Similarly, in [23], a Poincaré ST-GCN was introduced for skeleton-based action recognition. They modeled the input sequence (skeletons) as a graph in hyperbolic space, demonstrating the benefits of hyperbolic geometry for capturing spatial relationships. Nevertheless, their work mainly focused on applying hyperbolic transformations to specific components of the architecture without building a complete hyperbolic neural network.

Additionally, work [46] proposed hyperbolic manifolds as an alternative for image segmentation, enabling pixel-level classification with hierarchical formulation. Their work showed how hyperbolic spaces offer natural uncertainty estimation measures and improved zero-label generalization compared to Euclidean counterparts. However, similar to the previous works, they did not fully explore the potential of building a full hyperbolic neural network for computer vision tasks.

While the mentioned works have made significant strides in applying hyperbolic learning to computer vision, they have primarily used hyperbolic layers or transformations in conjunction with traditional Euclidean networks, limiting the realization of the full benefits of hyperbolic geometry. As a result, it is very valuable to develop a complete hyperbolic neural network for computer vision tasks. Building such a full hyperbolic neural network has the potential to enhance feature representations, improve generalization, and enable more efficient computations, opening up new avenues for computer vision research.

3. Preliminary

Hyperbolic geometry [14], or the Lobachevsky–Bolyai–Gauss geometry, is a non-Euclidean geometry having a constant negative sectional curvature. This geometry satisfies all of Euclid’s five postulates, except the last parallel postulate, as illustrated in Figure 1.

Unlike Euclidean geometry, where the sum of the angles in a triangle is always 180 degrees, in hyperbolic geometry, the sum of the angles in a triangle is less than 180 degrees. This property is a consequence of the negative curvature of the hyperbolic plane. Distances in hyperbolic space grow exponentially as one moves away from a fixed point. As a result, objects in hyperbolic space appear to expand rapidly as they move away from an observer. The distance between two points in hyperbolic space is measured along the unique geodesic (the hyperbolic equivalent of a straight line) connecting them. Hyperbolic distance is often used to define similarity measures in hyperbolic embeddings. Hyperbolic geometry has found applications in various scientific fields, including physics, cosmology, computer graphics, and machine learning. In machine learning, hyperbolic geometry has been applied to design novel deep learning models, particularly in hyperbolic deep learning, to handle data with hierarchical or tree-like structures. By utilizing hyperbolic spaces, current studies aim to improve the representation and processing of complex data, such as graphs, hierarchical text data, and relational data, where traditional Euclidean spaces may not be the most suitable choice. Hyperbolic geometry provides a powerful and elegant mathematical framework to understand and analyze such intricate structures in a more efficient and expressive manner.

3.1. Topological Spaces and Manifold

A topological space [14] is a set equipped with a collection of open sets that satisfy certain properties. Open sets are subsets of the space that are considered “open” in the sense that they contain a neighborhood around each of their points. These open sets must fulfill three conditions:

• The entire space and the empty set must both be open.
• The intersection of any finite number of open sets must be open.
• The union of any number of open sets must be open.

Based on this we provide the definition of a manifold. A d-dimensional manifold $M_d$ (which can be embedded in ${\mathbb{R}}^{d+1}$) is a topological space that can be locally approximated by a d-dimensional Euclidean space ${\mathbb{R}}^d$. For any point $x\in M_d$, there is a homeomorphism between the neighborhood of x and Euclidean space ${\mathbb{R}}^d$. Lines and circles are examples of one-dimensional manifolds. Planes and spheres are examples of two-dimensional manifolds, which are called surfaces. The notion of the manifold is a generalization of surfaces in any dimension d. The tangent space $T_x{M}_d$ at point $x\in M_d$ is a d-dimensional hyperplane, which is embedded in ${\mathbb{R}}^{d+1}$ and locally approximates the manifold $M_d$ around the point x.

3.2. Isometric Models in Hyperbolic Space

Hyperbolic space is a homogeneous space with constant negative curvature. It is a smooth Riemannian manifold, and as such, a locally Euclidean space. The hyperbolic space can be modeled using the commonly used five isometric models [47,48], which are the Lorentz (hyperboloid) model, the Poincaré ball model, the Poincaré half-space model, the Klein model, and the hemisphere model. In the following, we will detail these models. Note that we describe the model by fixing the radius of the model to 1 for clarity, without loss of generality.

3.2.1. Lorentz Model

The Lorentz model ${\mathbb{L}}^n$ of an n-dimensional hyperbolic space is a manifold embedded in the $n+1$ dimensional Minkowski space. The Lorentz model is defined as the upper sheet of a two-sheeted n-dimensional hyperbola with the metric ${\mathfrak{g}}^{\mathbb{L}}$, which is

$${\mathbb{L}}^n=\{x=(x^0,...,x^n)\in {\mathbb{R}}^{n+1}:{\langle x,x\rangle }_{\mathbb{L}}=-1,x^0>0\},$$

(1)

in which the ${\langle ,\rangle }_{\mathbb{L}}$ represents the Lorentzian inner product:

$${\langle x,y\rangle }_{\mathbb{L}}=x^T{\mathfrak{g}}^{\mathbb{L}}y=-x^0{y}^0+\sum _{i=1}^n{x}^i{y}^i,x\mathrm{and}y\in {\mathbb{R}}^{n+1},$$

(2)

where ${\mathfrak{g}}^{\mathbb{L}}$ is a diagonal matrix with entries of 1s, except for the first element being −1. For any $x\in {\mathbb{L}}^n$, we can obtain that $x^0=\sqrt{1+{\sum }_{i=1}^n{\left(x^i\right)}^2}$. The distance in the Lorentz Model is defined as

$$d(x,y)=\mathrm{arcosh}(-{\langle x,y\rangle }_{\mathbb{L}}).$$

(3)

The main advantage of this parameterization model is that it provides an efficient space for Riemannian optimization. An additional advantage is that its distance function avoids numerical instability when compared to the Poincaré model, where the instability arises from the fraction.

3.2.2. Klein Model

The Klein model is also known as the Beltrami–Klein model, named after the Italian mathematician Eugenio Beltrami and German mathematician Felix Klein. The Klein model of hyperbolic space is a subset of ${\mathbb{R}}^n$, as illustrated in Figure 2. It is the isometric image of the Lorentz model under the stereographic projection [48]. The Klein model is obtained by mapping $x\in {\mathbb{L}}^{n+1}$ to the hyperplane $x^0=1$, using rays emanating from the origin. Formally, the Klein model is defined as

$${\mathbb{K}}^n=\{x\in {\mathbb{R}}^n:\left|\right|x\left|\right|<1\}.$$

(4)

The distance is

$$d(x,y)=\mathrm{arcosh}\left(1+\frac{1-\langle x,y\rangle }{\sqrt{{(1-|\left|x\right||}^2\left)\right(1-{\left|\right|y\left|\right|}^2)}}\right).$$

(5)

A straight line in the Klein model, e.g., line $\overline{AB}$ in the second figure from the left of Figure 2, is an intersection of a plane with the disk; thus, it is still straight, like in Euclidean space. Therefore, the Klein model is commonly used to compute the middle point. This model does not conform to the Euclidean model, which means that angles and circles are distorted.

3.2.3. Poincaré Model

The Poincaré model, as shown in Figure 2, is given by projecting each point of ${\mathbb{L}}^n$ onto the hyperplane $x^0=0$, using the rays emanating from (−1, 0, …, 0). The Poincaré model $\mathbb{B}$ is manifold-equipped with a Riemannian metric ${\mathfrak{g}}^{\mathbb{B}}$. This metric is conformal to the Euclidean metric ${\mathfrak{g}}^E={\mathit{I}}^n$ with the conformal factor ${\lambda }_x=\frac{2}{1-{\left|\right|x\left|\right|}^2}$, and ${\mathfrak{g}}^{\mathbb{B}}={\lambda }_x^2{\mathfrak{g}}^E$. Formally, an n-dimensional Poincaré unit ball (manifold) is defined as

$${\mathbb{B}}^n=\{x\in {\mathbb{R}}^n:\left|\right|x\left|\right|<1\},$$

(6)

where $\left|\right|·\left|\right|$ denotes the Euclidean norm. The distance between $x,y\in {\mathbb{B}}^n$ is defined as:

$$d(x,y)=\mathrm{arcosh}\left(1+2\frac{\left|\right|x-{y\left|\right|}^2}{{(1-|\left|x\right||}^2\left)\right(1-{\left|\right|y\left|\right|}^2)}\right).$$

(7)

3.2.4. Poincar é Half Plane Model

The closely related Poincaré half-plane model in hyperbolic space is a Riemannian manifold $({\mathbb{H}}^n,{\mathfrak{g}}^{\mathbb{H}})$, where

$${\mathbb{H}}^n=\{x\in {\mathbb{R}}^n:x_n>0\}$$

(8)

is the upper half space of an n-dimensional Euclidean space. And the metric ${\mathfrak{g}}^{\mathbb{H}}$ is given by scaling the Euclidean metric ${\mathfrak{g}}^{\mathbb{H}}=\frac{{\mathfrak{g}}^E}{x_n^2}$. The model ${\mathbb{H}}^n$ can be obtained by taking the inverse of the Poincaré model, ${\mathbb{B}}^n$, with respect to a circle that has a radius twice that of ${\mathbb{B}}^n$. The distance is

$$d(x,y)=\mathrm{arcosh}\left(1+\frac{\left|\right|x-{y\left|\right|}^2}{2x_n{y}_n}\right).$$

(9)

3.2.5. Hemisphere Model

The hemisphere model is also called the hemisphere model, which is not as common as the previous four models. Instead, this model is employed as a useful tool for visualizing transformations between other models. The hemisphere model is defined as

$${\mathbb{J}}^n=\{x=(x_0,...,x_n)\in {\mathbb{R}}^{n+1}:\left|\right|x\left|\right|=1,x_0>0\},$$

(10)

The five isometric models [47,48] are embedded sub-manifolds of ambient real vector spaces. In fact, these five models are equivalent models of hyperbolic space. There are closed-form expressions for mapping between these hyperbolic models. As illustrated in Figure 3, we display their model in a two-dimensional space and demonstrate their relationship.

4. Methodology

In this section, we will describe our method, the efficient hyperbolic residual MLP (HR-MLP). The entire neural architecture is illustrated in Figure 4. Initially, an image is taken as an input. It is divided into patches, just like ViT, ResMLP, g-MLP, and the MLP mixer [5,8,9,10], and then passed through a Euclidean linear layer to create patch embeddings. Then the Euclidean embeddings are converted to the hyperbolic domain via exponential mapping, particularly using the Lorentz model, considering the representation ability and the optimization trade-off. The hyperbolic domains are passed to a block, dubbed the HR block, which contains Lorentz linear layers for linear transformations in the hyperbolic domain. Finally, after passing through the block $\mathbf{N}\times$, the embeddings are then pooled using adaptive average pooling [49]. Like HyboNet [43], we use a hyperbolic MLP head to perform classification for the feature learned from the Lorentz model. In the following part, we will detail each neural component.

4.1. Lorentz Linear Embedding

Given an image $x\in {\mathbb{R}}^{H\times W\times D}$, we divide the image into overlapping patches [50] by using convolution with zero padding, which is our patch embedding layer. Specifically, an image $x\in {\mathbb{R}}^{H\times W\times D}$ is fed to a convolution with stride S, a kernel size of $2S-1$, a padding size of $S-1$, and output channels $C^{\prime }$. The output size is $\frac{H}{S}\times \frac{W}{S}\times C^{\prime }$. We rearrange the output shape to $P\times C^{\prime }$, where P is the number of patches and is calculated by $\frac{H}{S}\times \frac{W}{S}$. Let F be the feature learned from the patch embedding. As this feature, F, is still in Euclidean space, we need to lift the feature, such that it obtains the corresponding feature in the Lorentz manifold. To this end, the exponential map function is applied, which is

$${\mathrm{Exp}}_0\left(F\right)={\cos \mathrm{h}\left(\right|\left|F\right||}_{\mathbb{L}}{)+\sin \mathrm{h}(\left|\right|F\left|\right|}_{\mathbb{L}})\frac{F}{{\left|\right|F\left|\right|}_{\mathbb{L}}},$$

(11)

where we assume that the feature is at a tangent space of a feature point 0, in which we choose the origin of the Lorentz model. The feature representation F is then mapped to the Lorentz manifold. In this way, we obtain the feature representation that lies in the Lorentz manifold.

4.2. Lorentz Linear Layer

Basic operations, like addition, multiplication, and pooling, construct linear layers. They are easy and efficient to be applied in Euclidean space. The features F are now converted into hyperbolic space where Euclidean operations cannot be applied, as most are not manifold-preserving. Thus, currently, we need to provide Lorentz operations that are able to keep the learned feature on the manifold after the transformation.

An alternative route is to transfer each point in hyperbolic space to the tangent space, where the tangent space at that point is the Euclidean subspace. Therefore we can use Euclidean neural operations in this tangent subspace. Existing works [51,52], formalize most Euclidean operations for hyperbolic networks by transforming features between hyperbolic spaces and tangent spaces via logarithmic and exponential maps, such that neural network operations can directly be performed in the tangent spaces. However, tangent spaces are not the optimal choice for several reasons. First, the composition of these functions is complicated as features need to continuously move back and forth between two spaces for each layer. Second, such a transformation also leads to values ranging to infinity, which significantly reduces the stability of the model. Finally, the tangent space is a linear approximation of hyperbolic space, which would not make full use of the advantages of non-Euclidean space.

To avoid such a transformation between hyperbolic and tangent spaces, as inspired by Chen et al. [43], we proposed the Lorentz (hyperboloid) linear layer ($\mathrm{HL}(·)$) fully in hyperbolic space by formalizing Lorentz manifold-preserving operations for neural networks without using tangent spaces. Taking inspiration from the theory of special relativity, which uses the Minkowski space (Lorentz model), our framework selects the Lorentz model as the feature space. The operations are formalized via the relaxation of the Lorentz transformations to build hyperbolic neural networks, including the linear layer, attention layer, etc. Additionally, from Chen et al. [43], we know that performing a linear transformation in the tangent space at the origin of hyperbolic space [51,52] is equivalent to performing a Lorentz rotation with relaxed restrictions. Therefore, by using this Lorentz rotation, we can build a Lorentz linear layer in a fully hyperbolic space, which is faster, more stable, and can achieve comparable or even better results than previous methods.

Here, we show how to build the Lorentz linear layer. Simply, as in Figure 5, the linear layer can be a mapping matrix $f\in {\mathbb{R}}^{(m+1)\times (n+1)}$, which can project the feature $F\in {\mathbb{R}}^{n+1}$ from the $n+1$ dimension to $m+1$. As for the feature representation on the manifold, we can also decouple the linear operation into Lorentz boost and rotation parts, which are $\mathbf{v}\in {\mathbb{R}}^{n+1},\mathbf{W}\in {\mathbb{R}}^{m\times (n+1)}$. In this way, we can only let the operation perform on the rotation part, while letting the extra boost part work as a regularizer so that the entire features are always on the Lorentz manifold. The general formula is as follows:

$$\mathbf{y}=\mathrm{HL}\left(\mathbf{F}\right)=\left[\begin{array}{c}\sqrt{{\parallel \varphi (\mathbf{W}\mathbf{F},\mathbf{v})\parallel }^2-1/K}\\ \varphi (\mathbf{W}\mathbf{F},\mathbf{v})\end{array}\right]$$

(12)

where $\mathbf{F}\in {\mathbb{L}}_K^n$ is the feature input from the previous layer, $\mathbf{v}\in {\mathbb{R}}^{n+1},\mathbf{W}\in {\mathbb{R}}^{m\times (n+1)}$ are the linear operations mentioned above, and $\varphi$ denotes the other operation functions, e.g., dropout or activation function in the linear layer. This formula is derived from the Lorentz boost and Lorentz rotation, which are polar decompositions of Lorentz transformation [53]. One can easily verify that the norm of the feature is fixed and on the manifold. Compared to hyperbolic linear layers in Ganea et al. [51] and Nickel and Kiela [52], this linear layer is far more expressive, efficient, and stable as it does not use complicated logarithmic and exponential maps. The authors dubbed their transformation as the pseudo-Lorentz rotation as it was a relaxation of the Lorentz transformation.

4.3. Lorentz Cross-Channel and Cross-Patch Layers

Here we introduce how to construct the attention-free HR block, as illustrated in Figure 6. Attention-based methods often involve cost computations to calculate the correlation between long-range positions. However, MLPs try to avoid such a burden while introducing a much more efficient way to perform information aggregation. Of which, the cross-channel and cross-patch layers are introduced as the fundamental components of MLP and its variants [9].

The cross-channel and cross-patch layers are two function models, which are all-MLP architectures that use linear layers to perform feature information updating. They consist of multiple layers of identical size (which means they will not change the feature resolutions). The first one is also called the channel-mixing MLP, which acts on rows (channels) of the feature and tries to exchange the feature information from different patches. This model is shared across all rows. As a comparison, the second MLP cross-patch layer is also called the token-mixing MLP. This layer acts on the columns of each feature and exchanges information across different channels. By combining these two layers, our method offers a significantly more efficient way to update features compared to attention-based methods.

Our Lorentz cross-channel layer uses the Lorentz linear layer to perform transformations on all patches independently, where the architecture is very similar. The only difference is that the Lorentz linear layer ($HL(·)$) is followed to force the output features to be still on the manifold. Likewise, the Lorentz cross-patch layer is extended from its Euclidean counterpart, aiming to perform transformations on all channels independently. There is also a $HL(·)$ function to make sure the module is manifold-preserving.

4.4. Hyperbolic MLP Head

Once the feature is learned on the manifold, we will need to make the class prediction based on the feature representation of each sample, from the logits. However, as such features lie in non-Euclidean space, we cannot apply the Euclidean MLR to compute the logits of each category. Therefore, we build a hyperbolic MLP head to make the class predictions based on the Lorentz feature. In particular, we build a learnable classification hyperplane for each class. Then the Lorentz distances to each hyperplane are computed based on Equation (3). Once we have the distances, we can make the prediction based on the shortest distance.

Compared with the vision transformer and its variants, our method is much more efficient as the replacement of the self-attention sublayer by a linear layer with GELU nonlinearity. This also stabilizes the training as it eliminates batch-specific or cross-channel normalization, e.g., BatchNorm, GroupNorm, or LayerNorm, following the same training scheme as in DeiT [54].

5. Experiment

We train our HR-MLP model on CIFAR10, CIFAR100, and MiniImageNet for comparison. All the models were trained from scratch; in the following subsection, we describe the hyperparameters we used to train on the dataset.

5.1. Datasets and Metrics

The proposed method is evaluated on three publicly available datasets, i.e., CIFAR10 [44], CIFAR100 [44], and MiniImageNet [55]. CIFAR10 and CIFAR100 are commonly used in image classification tasks. As indicated by the names, there are 10 classes and 100 classes for them. For MiniImageNet, the dataset consists of 60,000 color images (RGB) of size $84\times 84$ with 100 classes, of which, each class has 600 examples. This dataset is more complex than CIFAR10 and CIFAR100, but fits in the memory of modern machines, making it very convenient for rapid prototyping and experimentation.

We report the accuracy of the testing part of each dataset. We compare it with state-of-the-art methods, including, CNN, MLP mixer [8], ResMLP [9], Vision Transformer [5], and, EfficientVIT [56], in a compact setting (the number of parameters is less than 1 million). Even though the neural architectures and embedding spaces are different, we make a fair comparison by constructing neural architectures with similar parameters.

5.2. Implementation

For Cifar10 and Cifar100, we used a batch size of 32 in all of our experiments. We used the cross-entropy loss function with a label smoothing value of 0.1. All the models were trained on the Tesla P100 16 GB GPU available on Kaggle. All the models used the GELU activation function [57] for nonlinearity. The HR-MLP model used Riemannian SGD [58] with a learning rate value of 0.005 and weight decay of 0.05. All the Euclidean models, such as the ResMLP-S12, ViT, EfficientViT, the MLP mixer, and the CNN model, used the Adamw [59] optimizer with a learning rate value of 0.005 and a weight decay value of 0.05. All models were trained for a total of 150 epochs with a 20-epoch warm-up and a cosine annealing scheduler starting from epoch 40. The models were trained without augmentation, except for converting the images to tensors and normalizing the tensors. The model was evaluated based on the accuracy of the dataset’s test set. For the MiniImageNet, we maintained the same hyperparameters but employed a batch size of 256 and utilized a Tesla V100 GPU. Our code is available at (https://github.com/Ahmad-Omar-Ahsan/HR-MLP).

6. Results

We compared different methods, including traditional CNN neural architecture and current advanced MLP neural architectures. As listed in

Table 1. Test accuracy on CIFAR10, CIFAR100, and MiniImageNet using HR-MLP, ResMLP, ViT, and EfficientViT.

Methods	Parameters (M)	Flops (M)	Dataset
			CIFAR10	CIFAR100	MiniImageNet
CNN	2.112 M	10 M	61.86%	29.59%	20.15%
MLP mixer	1.005 M	13.3 M	59.01%	27.01%	30.30%
ResMLP-S12	0.685 M	81 M	62.77%	35.29%	36.83%
ViT	5.912 M	89.2 M	59.2%	32.23%	21.67%
EfficientViT-B0	2.142 M	10 M	75.04%	46.05%	34.43%
HR-MLP (Ours)	1.038 M	219 M	76.44%	48.55%	37.47%

, we can see a clear advantage of the proposed method when compared with the other methods. The accuracy of our method is significantly higher than any other given method. For instance, when evaluating the CIFAR10 dataset, our method can be at least $13\%$ higher than previous state-of-the-art methods, in the compact setting with similar parameters. This superiority could be even higher when compared with traditional architectures, like CNN. Moreover, with around 1M parameters, our method achieves 48.55% on the CIFAR100 dataset, while CNN only achieves 29.59%. On the more challenging dataset, MiniImageNet, our method achieves 37.47%, which is 17.3% higher than the CNN method and better than the previous state-of-the-art method, ResMLP, which clearly demonstrates the effectiveness of the proposed method using hyperbolic space. Further, our method can also be better than vision transformer models, like ViT and EfficientViT, with only half of the parameters.

To summarize, in this compact setting (which is valuable for local applications with limited resources), our method shows a clear superiority on different public datasets when compared with the baseline methods in Euclidean space. This provides evidence for the potential of learning image embedding in hyperbolic space for downstream tasks.

Here, we also show the feature distribution using t-SNE [60] to further demonstrate the advantage of the proposed hyperbolic neural network. As illustrated in Figure 7, even though the models are very small, we can still observe clear clusters for different categories. However, its Euclidean counterpart appears to struggle to distinguish patterns with such limited computational resources. This further substantiates the effectiveness of the proposed method.

7. Discussion

The strength of hyperbolic geometry lies in its inherent nonlinearity, which endows neural networks with the potential to capture richer information when compared to their Euclidean counterparts of similar architecture. This nonlinearity results in a distinctly different distance metric within hyperbolic space compared to the flat Euclidean space. Consequently, neural networks operating in hyperbolic space are compelled to learn more intricate and fine-grained feature representations, as what may be considered as ’near’ features in Euclidean space can be considerably distant in hyperbolic space.

However, it is essential to note that this nonlinearity, while advantageous, also brings about certain limitations. For instance, it increases the computational demands of the model. Unlike in Euclidean space, where networks can often be decoupled into linear layers and non-linear activations, hyperbolic networks require a more intertwined and challenging learning process. Additionally, there is a lack of definitive evidence supporting the superiority of hyperbolic neural networks when there are no parameter constraints. In other words, it remains unproven whether larger hyperbolic models consistently outperform their Euclidean counterparts, raising questions about the scalability and practicality of hyperbolic models in certain scenarios. Thus, valuable future research will include the scalability of hyperbolic models and the efficiency of hyperbolic neural networks.

8. Conclusions

This paper proposes a new framework for image classification in hyperbolic space. Inspired by the success of graph learning using hyperbolic graph neural networks, we model the underlying hierarchical relationships using hyperbolic geometry. Our approach combines the success of the current recovery of MLP and the exponential growth of hyperbolic space, to address the research gap surrounding the utilization of hyperbolic neural networks for computer vision tasks. The proposed method constructs ResMLP (including cross-channel and cross-patch layers) in hyperbolic space, with neural operations fully performed in hyperbolic space, without the need for back-and-forth Euclidean/non-Euclidean space transformations. Our proposed methods offer several advantages for real-world image classifications with hierarchies. They provide feature embeddings with less distortion. Additionally, the models are generally much more compact than their Euclidean counterparts. Moreover, such hyperbolic neural architectures provide much better interpretability. The experimental results show that our methods achieve better performance on different datasets than previous methods in this compact setting. By embracing the hierarchical structures within images and leveraging the power of hyperbolic spaces, our work offers novel insights and presents a promising direction for advancing computer vision techniques.

However, one limitation of our hyperbolic method is that it focuses on compact models that provide better interpretability, neglecting that there are unlimited computational resources. It would be better if one explores bigger hyperbolic neural models and compares their Euclidean counterparts. Despite this limitation, our approach has proven to be effective in practice, especially when resources are limited. In the future, we plan to explore larger neural network architectures, potentially as deep/extensive as architectures like ViT. Additionally, we will aim to apply our method to various datasets and investigate strategies to improve the classification model as the diversity of training datasets increases.