3D Shape Generation via Variational Autoencoder with Signed Distance Function Relativistic Average Generative Adversarial Network

Abstract

3D shape generation is widely applied in various industries to create, visualize, and analyse complex data, designs, and simulations. Typically, 3D shape generation uses a large dataset of 3D shapes as the input. This paper proposes a variational autoencoder with a signed distance function relativistic average generative adversarial network, referred to as 3D-VAE-SDFRaGAN, for 3D shape generation from 2D input images. Both the generative adversarial network (GAN) and variational autoencoder (VAE) algorithms are typical algorithms used to generate realistic 3D shapes. However, it is very challenging to train a stable 3D shape generation model using VAE-GAN. This paper proposes an efficient approach to stabilize the training process of VAE-GAN to generate high-quality 3D shapes. A 3D mesh-based shape is first generated using a 3D signed distance function representation by feeding a single 2D image into a 3D-VAE-SDFRaGAN network. The signed distance function is used to maintain inside–outside information in the implicit surface representation. In addition, a relativistic average discriminator loss function is employed as the training loss function. The polygon mesh surfaces are then produced via the marching cubes algorithm. The proposed 3D-VAE-SDFRaGAN is evaluated with the ShapeNet dataset. The experimental results indicate a notable enhancement in the qualitative performance, as evidenced by the visual comparison of the generated samples, as well as the quantitative performance evaluation using the chamfer distance metric. The proposed approach achieves an average chamfer distance score of 0.578, demonstrating superior performance compared to existing state-of-the-art models.

1. Introduction

The advancements in Artificial Intelligence (AI) technology have enabled the use of 3D shapes in the computer vision (CV) and computer graphics (CG) domains. The demand for fine details and accurate 3D shapes in robotics, 3D games, medical imaging, and virtual and augmented reality applications has necessitated the development of robust deep learning algorithms such as generative adversarial networks (GAN), variational autoencoder (VAE) and so on, to generate 3D mesh-based shapes.

Recent advancements in unsupervised learning with deep neural networks, trained on robust 3D datasets [1,2,3,4], have led to more detailed representations of shape space in deep learning models. These advancements enable realistic 3D shape generation for applications such as 3D scene understanding, data augmentation, visualization, simulation, analysis, and interpretation in the field of AI technology [5,6,7]. However, convolutional neural network (CNN)-based generative models that use voxels [6,7,8] and point-cloud [9,10,11] data representations as inputs often yield unsatisfactory results due to low resolution and high memory usage with voxels and the unstructured nature of point-clouds. On the other hand, using mesh data representation as input for 3D-GANs and 3D-VAEs leads to the generation of high-resolution, compact, and computationally efficient 3D shapes. However, the irregular and complex organization of mesh data representation [12,13,14] makes it challenging for it to be used as input for learning in a CNN.

Recently, several variations of the 3D-VAE-GAN [6,8,15] have been proposed to address the problem of 3D shape generation. These models have the ability to produce accurate 3D shape samples, classify 3D shapes, and reconstruct 3D shapes from 2D images. They are based on the well-known original GAN architecture [16] and have a tendency towards unstable training. While these models excel in creating 3D shapes from single classes, their outputs tend to be coarse, and training on complex data distributions with multiple object classes in various poses can be challenging. This paper aims to improve upon existing VAE-GAN models by generating 3D mesh-based shapes using a 3D-CNN framework and jointly training over multiple classes to achieve fully data-driven modeling.

To generate high-quality 3D mesh-based shapes from 2D images and capture a more complex distribution, this paper presents an innovative deep learning approach by integrating a relativistic average loss function with a signed distance function-based VAE-GAN framework, referred to as 3D-VAE-SDFRaGAN.

The 3D-VAE-SDFRaGAN method is a novel approach for generating 3D shapes that combines the strengths of VAEs and GANs. By using a signed distance function (SDF), this method can overcome the limitations of traditional methods for generating 3D shapes and creating more realistic shapes with smoother surfaces. The relativistic average GAN (RaGAN) is also employed to enhance the stability and quality of the generated shapes.

This approach has broad potential applications in several fields, including computer vision, computer graphics, gaming, robotics, medical imaging, simulation, and virtual reality. The generated 3D shapes offer improved communication, better design, and an enhanced understanding of complex structures and concepts.

The proposed 3D-VAE-SDFRaGAN aims to produce a signed distance function field on a gridded domain by utilizing a deconvolution layer. In doing so, it helps to produce a polygon mesh surface reconstruction with a higher quality 3D shape generation and stabilizes the GAN training process through the proposed relativistic average GAN loss function. The main contributions of this work are as follows:

• VAE and GAN are integrated to enable the simultaneous learning of encoding, generating, and comparing data. The proposed method explicitly learns the latent spaces of 2D images, which are then used to generate corresponding signed distance functions of objects and reconstruct them into high-quality 3D mesh-based shapes.
• A relativistic average GAN loss function is proposed to enhance the stability of the GAN training process, leading to a better convergence with the VAE-GAN framework.
• The proposed 3D-VAE-SDFRaGAN model not only generates high-quality 3D shapes from their corresponding 2D images but also achieves superior results compared to other state-of-the-art 3D shape generative methods.

This paper is structured as follows: related works are presented in Section 2. The concept of the SDF, data pre-processing, and model architecture for the proposed method are discussed in Section 3. Section 4 describes the training procedure and the results of the experiment, while Section 5 discusses the limitations of the proposed model and future directions. Finally, the conclusion is presented in Section 6.

2. Related Work

2.1. Modeling and Generation of 3D Shapes

Early studies on 3D geometric shape models with recognizable 2D characteristics can be traced back to the pioneering works of Roberts [17] and Marr [18]. Despite the advent of more advanced methods, traditional approaches that use symmetric parts [19], skeletons [20], and CAD wireframes [21] continue to be popular. For instance, Miao et al. [22] proposed SymmSketch using a progressive method that involved self-symmetric shape components, mutual-symmetric shape components, and computational theory for recovering 3D construction curves to produce symmetric 3D free-form shapes. Huang et al. [23] explored the generation of 3D shapes using pre-trained templates that generate object structure and surface geometry. R. Kelly et al. [24] proposed ShapeAssembly that combined the benefits of neural and procedural shape modeling and generated complete objects by synthesizing and assembling multiple subparts. Another example was the Shape Part Slot Machine developed by Kai et al. [25], which retrieved and combined pre-existing high-quality part meshes to create new shapes. Unlike these models, the proposed 3D-VAE-SDFRaGAN generates 3D mesh-based shapes in an end-to-end unsupervised manner without the need to collect and assemble shapes from a database.

2.2. 3D Shape Generation via Deep Neural Networks

Recently, researchers have made significant advancements in creating distributions that can depict shapes without relying on prior knowledge. This has been achieved by learning complex functions from data and parameterizing them to generate distributions [7,26]. A number of algorithms have been proposed to achieve this, including deep convolutional autoencoders [27], autodecoders [28], variational autoencoders [9], convolutional neural networks [29], recurrent neural networks [30], deep belief networks [3], and generative adversarial networks [6,31]. These algorithms use various data representations, such as voxels, point clouds, and meshes.

Wu, Z. et al. [3] used volumetric data representation as an input to perform 3D shape completion and recognition tasks. Wu, J. et al. [6] proposed the use of a generative adversarial network (GAN) to generate 3D shapes from a probabilistic latent space. Gadelha et al. [32] introduced PrGANs, a 3D generative model that was trained using the GANs framework and was able to match the input distributions of 2D views. Zhu et al. [8] proposed an architecture that used a GANs framework and incorporated a 2D image enhancer network to efficiently train the 3D model generator network. The output of the architecture was a voxel-based 3D shape, which is computationally expensive and requires a large memory footprint.

Ruihui et al. [33] introduced the SP-GAN, a novel generative model that represented 3D shapes as point clouds. The input latent vector was disentangled into a global prior (sphere points) and a local prior (random latent vector), and the generator network used style embedding and adaptive instance normalization to develop the 3D shape. Jun et al. [34] proposed GET3D, which leveraged the success of differentiable surface modeling, differentiable rendering, and training GAN models from 2D images. GET3D produced high-quality and textured 3D meshes with complex topologies, rich geometric features, and high textures. The model was trained using adversarial losses and used a rasterization-based differentiable renderer to obtain 2D RGB and silhouettes. However, GET3D only relies on generated data for evaluation purposes and needs to be evaluated on real-world data for a more accurate evaluation. Shen et al. [35] proposed GINA-3D, which uses implicit neural representations to generate 3D models. It uses generative adversarial networks (GANs) to learn from a dataset of 3D scans of real-world objects and generates 3D models that are both realistic and diverse. GINA-3D introduces a novel neural network architecture that can efficiently represent 3D geometry in an implicit form and uses a combination of adversarial and perceptual losses to improve the quality of the generated 3D models. However, the 3D-VAE-SDFRaGAN can generate 3D shapes with fine details from 2D images.

In contrast to the existing works, our proposed model seeks to use a similar framework but with a signed distance function as a 3D data representation and a RaGAN loss function to train the model to produce mesh-based 3D shapes. The generated 3D shape is of better quality and is less computationally expensive with a smaller memory footprint. To the best of our knowledge, no work has directly mapped 2D images using the VAE-GAN framework and relativistic average GAN loss function to 3D mesh-based shapes.

2.3. 3D Shape Generation with Signed Distance Functions

The signed distance function (SDF) is a data representation that has recently gained popularity in the computer vision and graphics domain due to its ability to overcome certain limitations of voxels, point clouds, and meshes when used as input data for CNNs. SDF represents the structural relationship of distances on 3D surfaces and has been used to train various deep learning algorithms, such as GANs, VAEs, and AD algorithms, to generate detailed 3D mesh-based shapes. In the field of 3D mesh-based shape generation, Jiang and Marcus [36] proposed a 3D-GAN-based hierarchical detail enhancing mesh shape generation using SDF data representation. They generated SDF fields on a gridded domain to create higher-quality polygon mesh surfaces, but their network was limited by being driven by uninformative random vectors and trained only on 3D data. Kingkan et al. [37] proposed a 3D-VAE-GAN-based framework that mapped point clouds directly to other 3D shape representations. Zheng et al. [38] proposed SDF-StyleGAN, which extends the popular generative model for 2D images, StyleGAN, to the 3D domain using an implicit SDF-based approach. The network generates new shapes by sampling from the learned SDF representation and then extracting the surface mesh. SDF-StyleGAN is trained using a combination of adversarial and reconstruction losses to produce high-quality 3D shapes with controllable attributes. However, 3D-VAE-SDFRaGAN has an advantage in generating a greater variety of shapes by interpolating between different SDF representations. In contrast, our proposed 3D-VAE-SDFRaGAN framework is trained on both 2D images and 3D models concurrently to enhance the quality of 3D mesh-based shape generation and stabilize the training process. To date, none of the current research studies use a 3D-VAE-GAN framework with a relativistic average GAN loss function to map 2D directly to 3D mesh-based shapes. Here we offer a comparative analysis between the proposed model and related works in 3D shape generation with signed distance functions.

2.4. Improving GAN Training for Quality 3D Model Generation

Several recent studies have demonstrated the effectiveness of generative adversarial networks (GANs) in various applications [6,8,39]. However, GANs can be challenging to train due to their non-convex game with high-dimensional continuous parameters. The conventional approach to training GANs involves finding low values of cost functions using gradient descent techniques, rather than finding the Nash equilibrium of the game. This can result in non-convergence or instability during the training process. To address these issues, various techniques have been proposed to stabilize GAN training [40]. Smith and Meger [7] proposed a 3D-VAE-IWGAN algorithm that generates realistic 3D models by leveraging the Wasserstein distance normalized with a gradient penalty as a loss function. IWGAN stabilized GAN training and encouraged model convergence. Furthermore, Chen et al. [41] proposed an infoGAN by adding variational mutual information maximization to GAN’s loss function to train 3D face and chair images. Although these methods showed faster convergence and generated higher-quality results compared to the standard GAN loss function, they have not been applied to generate 3D mesh-based shapes. In this study, the relativistic average GAN loss function is proposed to stabilize the training process and generate high-quality 3D mesh-based shapes from 2D images.

In a nutshell, many of the existing works used data representations like voxels, point clouds, and meshes, while others have used the signed distance function (SDF) representation for creating detailed 3D mesh-based shapes. The proposed 3D-VAE-SDFRaGAN uses a similar framework as previous works but with a signed distance function as a 3D data representation and a RaGAN loss function to stabilize the training. The generated 3D shapes are of better quality and computationally less expensive than previous works.

Table 1. Comparative analysis between the proposed model and related works.

Criteria	Approach	Architecture	Training Data	Input Data Type	Output Data Type	Performance	Computational Complexity	Limitation	Future Work
SymmSketch [22]	Algorithmic	Symmetry-aware algorithm	2D sketches	2D sketches	Symmetric 3D shapes	Fast generation of symmetric 3D shapes with limited input	Low	Not suitable for non-symmetric shapes	Improving the ability to generate 3D shapes from multiple viewpoints
AS3DS-DLGM [23]	Deep Learning	Generative model of surfaces	3D shapes within a family	3D meshes	3D meshes	Accurate generation of diverse 3D shapes within a family	High	Requires a large amount of training data	Exploring ways to generate higher-quality shapes with more user control
ShapeAss [24]	Deep Learning	CNNs, LSTM	Database of 3D parts and their assembly rules	Parts and assembly rules	3D meshes	High structural integrity of complex 3D shapes	High	Requires a large library of parts and complex reasoning algorithms	Investigating methods for reducing the computational complexity
3D ShapeNet [3]	Deep learning based volumetric shape representation	CNN	Volumetric 3D shapes	Voxelized 3D shapes	Voxelized 3D shapes	Can generate high-quality 3D shapes with complex structures	High	Can be memory-intensive and computationally expensive	Improving the ability to generate 3D shapes from multiple viewpoints
SP-GAN [33]	Sphere-guided 3D shape generation	GAN with a sphere-guided generator	Volumetric 3D shapes	2D image + sphere representation	Voxelized 3D shapes	Can generate high-quality 3D shapes with better user control	High	Generated shapes may not be visually consistent with the input data	Investigating methods for reducing the computational complexity
PrGAN [32]	3D shape generation from 2D views and manipulation	Multi-view CNN	2D images of multiple objects from various viewpoints	2D images	Voxelized 3D shapes	Can generate 3D shapes from limited 2D views with moderate quality	Moderate	The generated shapes may be limited by the quality and diversity of the input images	Exploring ways to generate higher-quality shapes with more user control
GET3D [34]	Generative model of 3D textured shapes learned from images	GAN with a novel 3D convolutional architecture	3D textured shapes from various viewpoints	RGB images	3D textured meshes	Can generate high-quality 3D textured meshes	High	Computationally intensive and requires significant computational resources, including high-end GPUs and large amounts of memory	Investigating ways to improve the texture quality and resolution of generated meshes
3D-VAE-GAN [6]	Generative adversarial modeling of 3D shapes	GAN with a 3D generator and discriminator	2D images and corresponding 3D shapes	Voxelized 3D shapes	Voxelized 3D shapes	Can generate high-quality 3D shapes with a probabilistic latent space	High	The generated shapes may be limited by the quality and diversity of the input images	Exploring ways to better control the probabilistic latent space for generating 3D shapes
MSVAE-GAN [37]	Generative Model	VAE-GAN	Point cloud + SDF	Point cloud	Meshes	Can generate high-quality 3D shapes with complex structures	Moderate	Requires a large amount of training data to accurately capture the underlying shape distribution	Incorporation of semantic labels
HDEM-3DGAN [36]	Generative Model	3D-GAN	Latent vector + SDF	SDF	Meshes	Can generate 3D shapes from limited 2D views with moderate quality	High	Computationally expensive, it may be challenging to incorporate user-defined constraints or preferences into the generated shapes	Improved hierarchical representation
Ours	Generative Model	VAE-SDFRaGAN	2D images + SDF	2D image	Meshes	Can generate high-quality 3D shapes with better user	Moderate	It requires a large amount of training data and computational resources	Incorporation of texture and colour

presents a comparative analysis between the proposed model and related works.

Table 2. Comparative analysis between the proposed model and related works on improving GAN algorithms.

Criteria	3D-VAE-IWGAN [7]	InfoGAN [41]	Ours
Approach	Generative Model	Generative Model	Generative Model
Architecture	VAE-IWGAN-based	GAN-based	VAE-SDFRaGAN-based
Training Data	2D images and corresponding 3D voxels	Latent vector and 3D Face dataset	2D images and corresponding 3D shapes
Input Data Type	2D images	3D Face, 2D Images	2D Images
Output Data Type	3D voxels	Meshes	Meshes
Performance	High-quality 3D shape generation, smooth interpolation between shapes, controllable output via latent space manipulation	Disentangles latent factors for interpretable representations, can learn without explicit supervision	Fast training and generation, good visual quality, can handle large-scale 3D shapes
Adversarial Loss	Wasserstein GAN (WGAN)	GAN	Relativistic Average GAN (RaGAN)
Training Approach	Iterative Optimization	Adversarial and Mutual Information	Iterative Optimization
Benefits	High-quality 3D object generation, improved stability	Interpretable Latent Code, improved Stability	Improved 3D shape generation, improved stability
Latent Code Interpretability	Limited (WGAN)	High	High
Limitation	May suffer from mode collapse and instability during training	May require a larger number of training samples to achieve optimal performance	May require more complex pre-processing steps to convert 3D shapes into 2D images and signed distance functions
Potential Applications	Computer graphics, gaming, medical imaging, and virtual reality	Image synthesis, feature extraction	Computer graphics, gaming, medical imaging, and virtual reality

provides a comparative analysis of the proposed model with related works on improving GAN algorithms.

3. Proposed 3D-VAE-SDFRaGAN

The steps involved in the proposed 3D-VAE-SDFRaGAN for 3D shape generation are outlined in this section. A 3D signed distance function field and a 2D image are the inputs to the system. A geometry analysis pipeline is presented to convert the mesh-based shape data into a 3D signed distance function (SDF) field as a format that can be used for learning. Next, the process of creating the 2D image dataset is described. The variational autoencoder (VAE) [42] and the generative adversarial network (GAN) [16] are utilized and integrated to generate 3D shapes in this work. A brief background on these techniques is provided to give a better understanding of their implementation in our system. Additionally, a relativistic average GAN loss function is proposed to establish a mapping between the 2D images and the signed distance functions. As a result, realistic 3D shapes that correspond to the input 2D images can be generated by the network.

3.1. 2D Image and 3D SDF Dataset Generation

ShapeNet [1] was used as the dataset in the experiments. The dataset comprises 55 common objects, consisting of 51,300 3D models. Furthermore, the 2D dataset utilized images from Choy et al. [30], which contain rendered images of ShapeNet 3D models from 23 different views. Information about 2D images used in our experiment is described in

Table 3. Information about 2D images used in our experiment.

Categories	Size (Pixel)
Chair, Table, Car, Lamb, Sofa, Cabinet	64 × 64

.

To preprocess the 3D signed distance function (SDF) field, SDFs were first derived from the 3D meshes [43]. Motivated by [36,37], the network output took the form of SDF. An SDF is a subset of implicit functions that assigns a 3D point to a real value, expressing the structural relationship and distance to the 3D surface, rather than a likelihood. The SDF data representation, which is a 3D representation that uses signed values to represent a mesh object’s inside–outside characteristics, has become popular for mesh generation with deep convolutional neural networks, as its presentation is not limited by fixed topology, unlike meshes and point clouds. Additionally, the SDF representation has a higher resolution compared to the voxel resolution.

Given a spatial point $p\in R^3$, the sign distance function $k\left(p\right)\in R$ encodes the point’s distance to its closest surface point, where p lies inside (−) or (+) of the object. In mathematical terms, given a set of points in a 3D Euclidean space, $\Omega$, where $\Omega$ is a non-zero volume open set with a smooth enclosed boundary, $\delta \Omega$, the signed distance function, k, is defined as

$$k\left(x\right)=\left\{\begin{array}{cc}\hfill \mathrm{dis}(x,\delta \Omega ),\hfill & \hfill \mathrm{if}x\in \Omega \hfill \\ 0,\hfill & \hfill \mathrm{if}x\in \delta \Omega \hfill \\ \hfill -\mathrm{dis}(x,\delta \Omega ),\hfill & \hfill \mathrm{if}x\in {\Omega }^c\hfill \end{array}\right.$$

(1)

where $\delta \Omega$ denotes the boundary of $\Omega$. The distance from a point x, where x belongs to 3D Euclidean space to the boundary $\delta \Omega$ is defined as the infimum of the distances from the point x to all points on the boundary:

$$dis(x,\delta \Omega )=\underset{y\in \delta \Omega }{inf}dis(x,y)$$

(2)

The sign of a point x with respect to the boundary, $\delta \Omega$, is defined as follows:

$$sign(x,\delta \Omega )=\left\{\begin{array}{cc}\hfill 1,\hfill & \hfill \mathrm{if}x\in \Omega \hfill \\ \hfill 0,\hfill & \hfill \mathrm{if}x\in \delta \Omega \hfill \\ \hfill -1,\hfill & \hfill \mathrm{if}x\in {\Omega }^c\hfill \end{array}\right.$$

(3)

The signed distance functions in (1), which is the product of (2) and (3) can also be written as

$$k\left(x\right)=dis(x,\delta \Omega )·sign(x,\delta \Omega )$$

(4)

To convert a triangular mesh into an SDF, the triangular mesh is first centered and normalized. Next, a 3D unit grid of resolution ${64}^3$ is established around the geometry. The calculation of point-to-mesh distance is performed using an axis-aligned bounding box (AABB) tree, and the winding number of each point is computed to determine the sign of each point in the grid. Figure 1 illustrates examples of the transformation of 3D meshes into SDFs and 2D images. In the first row, 2D images of the car, table, sofa, and cabinet are shown. The chair and table meshes with their SDF views from different angles are shown in the second and third rows. The distance between each point and the surface is depicted in an SDF colormap. The colors in this colormap range from dark blue (inside) to yellow (outside).

3.2. VAE, GAN, and VAE-GAN

An extended version of an autoencoder network that imposes additional restrictions on latent variables is known as a variational autoencoder (VAE). The network is transformed into an algorithm that learns its input information from a latent variable model due to the restriction. The parameters that model the data from the probability distribution are learned by the VAE. The VAE network comprises encoder and decoder networks, with the encoder network compressing the input data over the latent distribution $p\left(z\right)$ regulated by the prior into a latent representation, while the decoder network generates a new instance of input data from the latent representation. The VAE’s weights are trained simultaneously by improving the reconstruction loss and Kullback–Leibler divergence between the learned latent distribution and the prior.

On the other hand, a generative modeling algorithm called the generative adversarial network (GAN) utilizes a generator network and a discriminator network to generate new instances of data. The generator network transforms a random vector drawn from a Gaussian distribution into a data sample that is similar to the training dataset, while the discriminator network evaluates if the generated data sample is similar to the training dataset. The generator and discriminator networks train each other by competing against each other until the generator produces high-quality data.

The integration of VAE and GAN to improve the quality of data generation is becoming increasingly popular. The combination is computationally more efficient and produces a better quality of data generation. For instance, Larsen et al. [44] proposed a combination of VAE and GAN for 2D image generation, which resulted in improved feature representation and similarity measures compared to using either VAE or GAN alone. Wu, J. et al. [6] combined 3D-VAE-GAN to generate a voxel-based 3D model from 2D images, while Kingkan and Hashimoto [37] combined 3D-VAE-GAN to generate SDF from 3D point clouds. Another work by Smith and Merger [7] combined 3D-VAE-IWGAN to perform voxel-based 3D model generation, 3D model reconstruction, and 3D shape completion from a 2D image. In this work, the authors propose a similar framework to learn the latent spaces of 2D images and map them to their respective signed distance functions.

3.3. 3D-VAE-SDFRaGAN Framework

A novel approach called 3D-VAE-SDFRaGAN is proposed to learn the latent representation of a 2D image and generate 3D mesh-based shapes that are similar to the 2D image. The use of the learned latent space of a 2D image from the 2D-VAE network as input to the SDF-generator network is the key innovation of this approach. A relativistic average GAN loss function is used to train the SDF-generator network, allowing it to generate high-quality 3D shapes from both the 2D image and SDF. The architecture of the 3D-VAE-SDFRaGAN consists of three main components: a 2D-image encoder network $\left(E\right)$, an SDF-generator network $\left(G_{sdf}\right)$, and an SDF-discriminator network $\left(D_{sdf}\right)$. The 2D-image encoder network compresses the 2D image into a latent representation that can be used as input to the SDF-generator network. 3D mesh-based shapes are generated by the SDF-generator network using this latent representation. The generated shapes are evaluated by the SDF-discriminator network, providing feedback to the SDF-generator network to continually improve its ability to generate high-quality 3D shapes. Figure 2 depicts the overall architecture of the 3D-VAE-SDFRaGAN, which demonstrates the three key components and their interaction. The overall architecture of the proposed 3D-VAE-SDFRaGAN is comprised of three components: (a) the encoder network, which converts a 2D image to latent spaces; (b) the generator network, which produces an SDF from a 2D image latent vector; and (c) the relativistic discriminator network, which evaluates the likelihood of an actual SDF being more authentic than a randomly selected generated SDF from the generator. The 2D image is encoded through a 2D-image encoder network in Figure 2a to obtain its latent vector $\left(z_{img}\right)$. The latent vector $\left(z_{img}\right)$ is concatenated with random noise $\left(z\right)$ before sending to $\left(G_{sdf}\right)$ network as shown in Figure 2b to generate a fake SDFs $(sd{f}_{gen},sd{f}_z)$. Next, the $(sd{f}_{gen},sd{f}_z)$ is concatenated with $\left(sd{f}_{real}\right)$ as input for $\left(D_{sdf}\right)$, as shown in Figure 2c, to return the prediction score.

3.3.1. 2D-Image Encoder Network $\left(E\right)$

The latent representation of a 2D image is learned by the encoder network in the proposed 3D-VAE-SDFRaGAN framework. This network is composed of five convolutional layers with channels of {$64,128,256,512,400$}, kernel sizes of {$11,5,5,5,8$}, and strides of {$4,2,2,2,1$}. To ensure the output of the convolutional layers is free of negative values and has consistent mean and variance, ReLU activation functions and batch normalization layers are included between the convolutional layers. A 400-dimensional latent vector is outputted by the last layer of the encoder network, which is split into a 200-dimensional mean latent vector and a 200-dimensional variance latent vector. Furthermore, a 200-dimensional latent vector is sampled from the Gaussian distribution by the encoder network’s sampling layer. The loss function of the encoder network ($L_E$) is composed of two parts: the KL divergence loss ($L_{KL}$) and the reconstruction loss ($L_R$) defined as

$$L_{KL}=D_{KL}(q\left(z_{img}\right|img)\parallel p\left(z\right))$$

(5)

$$L_R=\parallel G\left(\mathbb{E}\left(img\right)\right)-sd{f}_{real}{\parallel }_2$$

(6)

where $L_{KL}$ is divergence loss between the learned latent distribution ($z_{img}$) and the prior distribution $p\left(z\right)$ from a uniform distribution over $[-1,1]$, $sd{f}_{real}$ is an SDF of a 3D shape from the training set, $img$ is the corresponding 2D image, $q\left(z_{img}\right|img)$ denotes the variational distribution of the latent representation.

The KL divergence loss measures the difference between the learned latent distribution ($q\left(z_{img}\right|img)$) and the prior distribution $p\left(z\right)$. The reconstruction loss measures the difference between the output of the SDF-generator network when fed with the mean latent vector ($G\left(\mathbb{E}\right(img\left)\right)$) and the ground truth SDF of the 3D shape from the training set ($sd{f}_{real}$). The overall loss function encourages the encoder network to generate a latent representation that both follows the prior distribution and accurately reconstructs the SDF of the 3D shape. To allow the SDF-generator network to sample $z_{img}$ from the same distribution as $p\left(z\right)$, KL divergence is used to limit $q\left(z_{img}\right|img)$ to be as similar to $p\left(z\right)$ as possible.

3.3.2. SDF-Generator and SDF-Discriminator Networks

The SDF-generator network $\left(G_{sdf}\right)$ is composed of five transpose convolution layers with channel numbers $\{512,256,128,64,1\}$ and kernel sizes of $\{4,4,4,4,4\}$, with strides of $\{1,2,2,2,2\}$. Between the transpose convolution layers, ReLU and batch normalization layers are utilized, except for the last layer, which applies a Tanh function to map the output into $[-1,1]$. The SDF of a ${64}^3$ matrix is the output of the generator network $\left(SD{F}_{gen}\right)$, with values in the range of $[-1,1]$. The triangular mesh surfaces are generated from the SDF matrix using the MCA [45].

The SDF-discriminator network $\left(D_{sdf}\right)$ is similar to the SDF-generator network and uses leaky ReLU as an activation function. The sigmoid function is applied at the last layer to compress the output to $[0,1]$. It consists of five 3D-convolution layers with channel numbers $\{64,128,256,512,1\}$ and kernel sizes of $\{4,4,4,4,4\}$, with strides of $\{2,2,2,2,1\}$. The loss function for SDFRaGAN is

$$\begin{array}{cc}\hfill L_{SDFRaGAN}=\hfill & \hfill -{\mathbb{E}}_{sd{f}_{real}{\backsim }_P}\left[log\left(\overline{D}\left(sd{f}_{real}\right)\right)\right]-{\mathbb{E}}_z{\backsim }_Q\left[log(1-\overline{D}\left(z\right))\right]\hfill \\ \hfill \hfill & -{\mathbb{E}}_{z_{img}}{\backsim }_Q\left[log(1-\overline{D}\left(z_{img}\right))\right]\hfill \end{array}$$

(7)

where

$$\begin{array}{c}\hfill \tilde{D}\left(sd{f}_{real}\right)=sigmoid(C\left(sd{f}_{real}\right)-{\mathbb{E}}_{z\sim \mathbb{Q}}C(z,z_{img}))\hfill \\ \tilde{D}\left(z\right)=sigmoid\left(C\left(z\right)-{\mathbb{E}}_{sd{f}_{real}\sim \mathbb{P}}C\left(sd{f}_{real}\right)\right)\hfill \\ \hfill \tilde{D}\left(z_{img}\right)=sigmoid\left(C\left(z_{img}\right)-{\mathbb{E}}_{sd{f}_{real}\sim \mathbb{P}}C\left(sd{f}_{real}\right)\right)\hfill \end{array}$$

The total loss $L_{total}$ function used in the proposed 3D-VAE-SDFRaGAN framework consists of the sum of three components: reconstruction loss $\left(L_R\right)$, a cross-entropy $\left(L_{SDFRaGAN}\right)$, and KL divergence $\left(L_{KL}\right)$ to impose a limit on the distribution of the output of the 2D encoder network:

$$L_{total}=L_{SDFRaGAN}+{\gamma }_1{L}_{KL}+{\gamma }_2{L}_R$$

(8)

where ${\gamma }_1$ and ${\gamma }_2$ are weights of $L_{KL}$ and $L_R$, respectively.

4. Experiments

In this section, the training procedure is first discussed. Then, a comparison is made between our proposed 3D-VAE-SDFRaGAN and several state-of-the-art generative models. The qualitative and quantitative results are also presented. Qualitative evaluation involves the visual inspection of the generated 3D shapes, while quantitative evaluation involves the utilization of the chamfer distance (CD) metric. The performance of the proposed model is determined by comparing the results with those of the state-of-the-art models.

4.1. Training Procedure

The pair of $\left\{im{g}_i,sd{f}_{\mathrm{real}{}_i}\right\}$ that is drawn from the training dataset is used to train the proposed 3D-VAE-SDFRaGAN framework, where $im{g}_i$ is the 2D image, and ${sdf}_{rea{l}_i}$ is the corresponding signed distance function of the 3D object. During training, the latent representation $\left(z_{img}\right)$, which represents the image feature, is generated by the image encoder after encoding a 2D image $im{g}_i$. The SDF-generator network, which receives $z_{img}$ with a 200-dimension vector as input, generates $sd{f}_{gen}$ as output. A random vector z is sampled from a uniform distribution $p\left(z\right)$ and fed to the SDF-generator network to produce $sd{f}_z$. Both generated SDFs $(sd{f}_{gen},sd{f}_z)$, along with $sd{f}_{real}$, serve as the input to the SDF-discriminator network for classification purposes. The SDF-discriminator network distinguishes SDFs and determines whether the SDF is generated from the SDF-generator network or if it is the real SDF from the dataset. The learning rates with the values of ${10}^{-5}$, ${10}^{-3}$, and ${10}^{-3}$ are used during training for the discriminator network, generator network, and encoder network, respectively. For optimization purposes, we use the Adam optimizer with $\beta$ = 0.5. The proposed network is trained separately on each class of objects, with a batch size of 64. During the experiment, the values of 1 and 100 are set to ${\gamma }_1$ and ${\gamma }_2$, respectively. We used the ShapeNet dataset to train and test the proposed 3D-VAE-SDFRaGAN, with a dataset split of 80% and 20% for the training and test sets, respectively [30,46,47]. The model was trained for 1500 epochs on the table, lamp, sofa, and cabinet categories, while the car and chair categories were trained for 1700 epochs. The experiment was conducted using a single Nvidia GeForce GTX 1080 GPU, with a total training time of 192 h. The complex architecture of the proposed model, which combines a variational autoencoder (VAE) and a generative adversarial network (GAN), imposes a relatively high computational burden. During the training process, multiple iterations were carried out to cover the car, chair, cabinet, lamp, sofa, and table categories of the ShapeNet dataset. Model performance optimization was achieved by training some categories for 1500 epochs while others were trained for 1700 epochs. The computational burden was increased using a single Nvidia GeForce GTX 1080 GPU for training, resulting in longer training times due to limited computational resources. The following loss functions are used in the encoder, generator, and discriminator networks during training:

$$L_E={\gamma }_1{L}_{KL}+{\gamma }_2{L}_R$$

(9)

$$\begin{array}{cc}\hfill L_G=& \hfill -{\mathbb{E}}_z{\backsim }_Q\left[log(1-\overline{D}\left(z\right))\right]-{\mathbb{E}}_{z_{img}}{\backsim }_Q\left[log(1-\overline{D}\left(z_{img}\right))\right]-{\gamma }_2{L}_R\hfill \end{array}$$

(10)

$$\begin{array}{c}\hfill L_D=L_{\left(SDFRaGAN\right)}\end{array}$$

(11)

The parameters of the discriminator network are updated when the accuracy is less than 0.8 in each batch. Figure 3 illustrates the training process flow of our proposed framework. During training, we use a random noise sampled from a uniform distribution, $p\left(z\right)$, denoted as z. We also use the signed distance function, $sd{f}_{real}$, from the dataset. Additionally, we generate two new signed distance functions: $sd{f}_{gen}$ from random noise, z, and $sd{f}_z$ from an encoded image, $z_{img}$.

4.2. Performance Evaluation

A framework named 3D-VAE-SDFRaGAN, which can infer a 3D shape from its associated 2D image, is proposed in this paper. The results of the proposed model, which learns to build SDFs of chair, table, car, and sofa from their associated 2D images, are shown in Figure 4. The marching cubes algorithm is employed to extract triangular surfaces from the SDFs, which are then smoothed using Laplacian smoothing to generate the final 3D shapes, as presented in Figure 4c.

Several state-of-the-art models are compared against the proposed 3D-VAE-SDFRaGAN, following the evaluation process outlined in [46]. The volumetric outputs of the 3D-R2N2 model are converted into a mesh representation using the marching cubes algorithm, and the quality of the generated model is compared with the proposed model. The chamfer distance (CD) [9,46] is used for evaluating the accuracy of the 3D-VAE-SDFRaGAN model, which measures the difference between the generated 3D mesh and the ground truth mesh for the six categories presented in

Table 4. Performance comparison of the proposed method with state-of-the-art on the ShapeNet dataset.

Category	3D-R2N2 [30]	SIF [47]	N3MR [48]	MeshSDF [49]	DISN [50]	Ours
Chair	1.432	1.540	2.084	0.590	0.754	0.589
Table	1.116	1.570	2.383	1.070	1.329	0.672
Car	0.845	1.080	2.298	0.960	0.492	0.491
Lamb	4.009	3.420	3.013	1.490	2.273	0.662
Sofa	1.135	0.800	3.512	0.780	0.871	0.566
Cabinet	0.750	1.100	2.555	0.780	1.130	0.314
Average	1.545	1.585	2.641	0.945	1.142	0.578

. Also, the proposed model is compared with state-of-the-art models in

Table 5. Comparative analysis between the proposed model and state-of-the-art models.

Criteria	3D-R2N2 [30]	SIF [47]	N3MR [48]	MeshSDF [49]	DISN [50]	Ours
Approach	Multi-view reconstruction using 3D convolutional neural networks	Reconstruction using implicit functions and shape templates	Rendering 3D shapes from learned feature maps	Reconstruction using implicit functions and iso-surface extraction	Reconstruction using implicit functions and single-view input	Learning a generative model for 3D shapes via 2D images
Architecture	2D-CNN + 3D-LSTM + 3D-DCNN	Multi-layer perceptron	Neural network with geometric encoding	Multi-layer perceptron	Convolutional neural network	VAE-SDFRaGAN
Dataset	ShapeNet and PASCAL 3D	ShapeNet	ShapeNet	ShapeNet	ShapeNet	ShapeNet
Training Data	Set of 2D views with corresponding 3D shapes	3D shapes	3D mesh models	3D mesh represented as SDFs	3D shapes	2D images and corresponding 3D shapes
Input Data Type	2D images of an object from different viewpoints	Stack of depth images around the mesh	3D mesh	SDF	2D images	2D images
Output Data Type	Volumetric 3D shape	Learned shape templates	3D mesh	3D meshes	3D meshes	3D meshes
Performance	Utilizes a 3D recurrent neural network to generate a 3D voxel grid from a 2D image or a sequence of images	Learns a 3D shape template from a set of 3D shapes using implicit functions	Uses a neural network to estimate the visibility of each vertex and generates the 2D image using a differentiable renderer	Uses a differentiable iso-surface extraction algorithm to generate the SDF and trains a neural network to approximate the SDF	Uses a deep neural network to estimate the implicit function and generates the 3D mesh using marching cubes	Can generate high-quality 3D meshes with complex structure
Computational Complexity	High	Low	High	Low	Low	High
Limitation	Limited to objects that can be approximated by a set of primitive shapes	Limited to learning shapes that can be described by implicit functions	Limited to rendering objects that have a corresponding mesh representation	Limited to shapes that can be represented by a signed distance function	Limited to learning shapes that can be represented by implicit functions	It requires a large amount of training data and computational resources
Potential Future Work	Improving the accuracy and resolution of the reconstructed 3D models	Incorporating texture information into the model	Improving the efficiency and speed of the rendering process	Improving the model’s robustness to handle noise and outliers in the input data	Investigating the use of more complex neural network architectures to improve the model’s accuracy and efficiency	Incorporating texture, Incorporating semantic labels

.

Table 4 also presents the results of this evaluation. The performance of the proposed 3D-VAE-SDFRaGAN was evaluated by computing the CD scores for the six categories. The chamfer distance (CD) is a standard evaluation metric in 3D deep learning, which is used to assess the similarity between two meshes by calculating the distance between the points of each mesh and the nearest surface point of the other mesh. The computation is performed by determining the closest point in the opposite mesh for each point in each mesh and then summing up the square of the distances between the two sets of points. The accuracy of the object is better measured when the CD score is smaller. The equation of the CD is defined as

$$d_{CD}\left(S_1,S_2\right)=\sum _{x\in S_1}\underset{y\in S_2}{min}{\parallel x-y\parallel }_2^2+\sum _{y\in S_2}\underset{x\in S_1}{min}{\parallel x-y\parallel }_2^2$$

(12)

where $S_1$ and $S_2$ are the original mesh and generated mesh, respectively, and x and y are vertices of the original mesh and generated mesh, respectively.

The results of the experiment demonstrated that other state-of-the-art methods were surpassed in all categories by the proposed model. This was attributed to the significant improvement in the stability of the 3D-VAE-SDFRaGAN network, which was achieved by incorporating a relativistic discriminator and a non-standard GAN loss function, without any additional computational cost. Superior quality in generating 3D shapes was exhibited by the results of the proposed method, and a comparative analysis with other models was performed.

Objects with finer details that cannot be achieved with 3D-R2N2 using different model views [30] were generated by our proposed model, which demonstrated its efficacy. Additionally, the 3D-R2N2 algorithm was based on a 3D-autoencoder that inherited the limitations of autoencoder algorithms and was primarily used for reconstruction purposes, unlike our proposed method, which was designed for the generation task. The proposed model had the capability to generate objects with diverse topologies, unlike the N3MR approach, which lacked this feature [48]. The proposed model also addressed a key issue with the SIF method by utilizing SDF data representation to accurately capture the intricate details of an object’s structure [47]. Furthermore, a well-detailed 3D shape is produced by the proposed model, which is sacrificed by the MeshSDF framework [51]. Finally, the proposed 3D-VAE-SDFRaGAN not only effectively scaled, but also produced highly detailed shape representations that were previously unattainable with the DISN model. This was because the shape representation in DISN’s model was limited to a fixed-length feature vector, which only captured the global shape and was unable to represent fine details [52]. Figure 5, Figure 6 and Figure 7 present a graph to show the performance of CD metrics on a chair, cabinet categories, and average chamfer distance of the proposed model with other state-of-the-art models.

In Figure 8, the qualitative comparison of the generated samples and the learned representations of the proposed model with other 3D GAN-based models for visual evaluation is shown. (a) Sample of car, chair, and table generated by J. Wu et al. [6]. (b) Sample of car, chair, and table generated by J. Zhu et al. [8]. (c) Sample of chair and table generated by Smith et al. [7]. (d) Sample of car, chair, and table generated by Jiang et al. [36]. (e) Sample of car, chair, and table generated by Kingkan et al. [37]. (f) Sample of car, chair, and table generated by our model. It can be observed that noticeably better and higher qualitative performance is obtained by the smooth meshed surfaces recovered from signed distance function fields powered by the proposed 3D-VAE-SDFRaGAN framework and 2D image features in our work. The results of Kingkan et al. [37] is compared with the generated samples in Figure 8f, despite the complete 3D data used in their work. This shows that the low-cost 2D images readily available with corresponding SDFs from the proposed method are better dataset options for training 3D mesh-based generator models leveraging relativistic average GAN loss function. Moreover, better mesh-based 3D shape generation with the proposed 3D VAE-SDFRaGAN model is aided by the combination of 2D images and the SDFs dataset. In addition, the proposed model, which is simple yet efficient, was evaluated by comparing its trainable parameters to those of Jiang et al.’s [36] two-stage model implementation. The trainable parameters of [36] amounted to 176,260,554, while our proposed model has only 41,645,290 trainable parameters, making it computationally efficient. The proposed model produced appealing results, as it was trained end-to-end using only a low-cost 2D image input, while Jiang et al.’s model involved a computationally expensive process of generating SDF in two stages.

5. Limitations and Future Directions

One limitation of 3D-VAE-SDFRaGAN is that it requires a large amount of training data and computational resources, which can be a challenge for many researchers and practitioners. Another limitation is that the performance of the proposed model is sensitive to the choice of hyperparameters, such as the size of the latent space, the learning rate, and the regularization parameters. The model is also limited in its ability to handle complex shapes and geometries, particularly those that involve non-uniform scaling or non-rigid deformation.

One potential direction for future research is to explore ways to improve the accuracy and detail of the generated 3D models, perhaps by incorporating additional data sources or refining the network architecture. Another promising direction is to investigate ways to extend the model to handle more complex shapes and geometries, such as those found in medical imaging or robotics applications. It may also be useful to explore ways to incorporate semantic information or other forms of prior knowledge into the model to improve its ability to generate models that are relevant to specific domains or applications.

6. Conclusions

In this paper, a novel approach for generating 3D shapes using 2D images as input is proposed. The proposed method, 3D-VAE-SDFRaGAN, leverages the power of variational autoencoders (VAEs) to learn the latent space of the 2D images and maps it to the corresponding 3D shapes represented by signed distance functions (SDFs). The experimental results demonstrate that the latent space of 2D images has a significant impact on the performance of the model and determines the amount of information that can be transferred from the 2D image encoder network to the SDF-generator network. To further improve the performance of the model, a relativistic average GAN loss function is incorporated into the proposed 3D-VAE-SDFRaGAN framework. This improves the stability of the model during training and results in higher-quality shape generation. The proposed method successfully generates smooth surface attributes of the corresponding SDF using features extracted from the 2D image. The proposed 3D-VAE-SDFRaGAN model exhibits superior quantitative performance over state-of-the-art methods, achieving an average chamfer distance score of 0.578 across the selected categories of the ShapeNet dataset. Additionally, the proposed model exhibits superior performance compared to existing methods, particularly in terms of qualitative output, which is supported by a visual comparison of the generated samples.

As the underlying task involves generating 3D shapes using generative modeling algorithms, the use of accuracy metrics such as F1, which are typically employed in classification tasks, may not be suitable for evaluation purposes. Instead, we utilized chamfer distance (CD), a standard evaluation metric for 3D deep learning, especially for mesh generation tasks [46,49,50]. In response to feedback, we have included this metric in the abstract to highlight the accuracy measurement used to evaluate our work, and have revised the abstract accordingly for the reader’s consideration.