Dimensionality Reduction for the Real-Time Light-Field View Synthesis of Kernel-Based Models

Abstract

Several frameworks have been proposed for delivering interactive, panoramic, camera-captured, six-degrees-of-freedom video content. However, it remains unclear which framework will meet all requirements the best. In this work, we focus on a Steered Mixture of Experts (SMoE) for 4D planar light fields, which is a kernel-based representation. For SMoE to be viable in interactive light-field experiences, real-time view synthesis is crucial yet unsolved. This paper presents two key contributions: a mathematical derivation of a view-specific, intrinsically 2D model from the original 4D light field model and a GPU graphics pipeline that synthesizes these viewpoints in real time. Configuring the proposed GPU implementation for high accuracy, a frequency of 180 to 290 Hz at a resolution of $2048\times 2048$ pixels on an NVIDIA RTX 2080Ti is achieved. Compared to NVIDIA’s instant-ngp Neural Radiance Fields (NeRFs) with the default configuration, our light field rendering technique is 42 to 597 times faster. Additionally, allowing near-imperceptible artifacts in the reconstruction process can further increase speed by 40%. A first-order Taylor approximation causes imperfect views with peak signal-to-noise ratio (PSNR) scores between 45 dB and 63 dB compared to the reference implementation. In conclusion, we present an efficient algorithm for synthesizing 2D views at arbitrary viewpoints from 4D planar light-field SMoE models, enabling real-time, interactive, and high-quality light-field rendering within the SMoE framework.

1. Introduction

Light fields enable users to move around with six degrees of freedom (6DoF) within camera-captured media. For example, a common way users can experience such freedom is through virtual reality (VR) systems with a head-mounted display. While 6DoF functionality is well established in 3D graphics applications (e.g., 3D video games), it remains challenging when applied to camera-captured content.

A successful system for representing, transmitting, and displaying light fields should have all of the following properties simultaneously:

• Plenoptic abilities: A unique advantage of light field recordings is that they can capture direction-dependent and complex light phenomena, such as refractions, smoke, fire, and reflections of curved surfaces.
• Camera-captured material: The system should be able to create representations from real-world camera-captured images or videos. This comes with a few challenges, such as the lack of accurate depth information and dealing with calibration information (e.g., regular sampling grids cannot be assumed).
• Six degrees of freedom: The representation should be fit for allowing a user to move in three translational and three rotational degrees of freedom, synthesizing views accordingly.
• Large immersive volumes: The representation should allow the user to explore a large volume containing significant occluders (e.g., entering different rooms).
• Panoramic: As the user has complete freedom to look around, content needs to be available in an immersive way in all viewing directions.
• Low latency and low decoding complexity: Due to the interactive 6DoF nature, it is not a priori known how the user will consume the content. E.g., the user can behave unpredictably, wander off, walk far away, make abrupt movements, etc. The representation and system have to be able to keep up with these non-predetermined ways of consuming content.
• Streamable representation: Ideally, one would like to deliver these interactive experiences remotely. This requires the representation at hand to be streamable while respecting the interactive nature of content consumption.
• Real-time playback: The system and representation should be fast enough to deliver all of the above aspects in real time and on commodity hardware. For VR applications, the system should ideally achieve rendering at 90 frames per second with dual output.

Although there are a wide variety of light-field-related technologies [1], there is no common consensus on which of them (or combinations thereof) will serve as the basis for such a successful system. Several technological approaches, such as 3D-graphics-based, depth-image-based rendering (DIBR), neural radiance fields (NeRFs), and kernel-based regression methods, are all actively researched. Each technology or system makes a trade-off between the described challenging properties. For example, traditional and stereoscopic 360° images/videos lack the 6DoF aspect. Approaches that rely on reconstructing the physical geometry using conventional 3D triangle meshes usually have all light information baked into the textures, taking away the unique advantages of light fields, while often struggling to represent fine details. DIBR systems tend to achieve high quality and high frame rate but typically have small immersive volumes, have trouble with disocclusions, and additionally need accurate depth information, where small inaccuracies in depth often cause unpleasant artifacts. Recently, NeRF approaches have proven to be promising, although not yet suitable for interactive frame rates on commodity hardware. Kernel-based representations such as SMoE have an appealing combination of properties, such as random access, small, and high streamability, but can be slow to render (see Section 2.2 for details).

This paper proposes a novel method that solves the real-time playback problem of higher-dimensional kernel-based models for arbitrary 6DoF viewpoints. We demonstrate this methodology more specifically on planar 4D light-field SMoE models, which use Gaussian kernels. Our method uses a modern GPU graphics pipeline to first transform the 4D model to a view-specific 2D model in the geometry shader, which emits ellipses that are rasterized with a custom fragment shader. We use OpenGL and geometry shaders for simplicity, but our method can easily replace geometry shaders with instanced rendering and be adapted to a wide variety of hardware and graphics APIs, such as Vulkan, DirectX, and mobile GPUs. For a video overviewing the proposed technique, as well as the public source code, see the data availability statement.

This paper is structured as follows. First, Section 2 gives an overview of related works in light-field technologies and a summary of existing work on SMoE. Then, in Section 3, we present our proposed GPU-based rendering technique, which relies on our novel scheme to obtain a 2D view-specific representation of the original 4D light-field model. Next, in Section 4, we validate our method thoroughly and compare it against the state-of-the-art NeRF representation from ‘instant-ngp’ [2]. Our conclusions are formulated in Section 5. Finally, formulas and derivations of our scheme are detailed in Appendix A.

2. Related Works

Due to the plenoptic requirements, as discussed in Section 1, we consider the following signature of the plenoptic function:

$$\mathcal{L}:{\mathbb{R}}^5\to {\mathbb{R}}^3:(x,y,z,\theta ,\varphi )\mapsto (R,G,B).$$

(1)

This variant of the plenoptic function describes the light at every point $(x,y,z)$ in space and incoming from every direction $(\theta ,\varphi )$ by a color in some RGB color space. For simplicity, we have left out the time dimension. Taking pictures with a camera in a scene essentially records samples from this plenoptic function, where $x,y,z$ match the position of the camera sensor, and $\theta$ and $\varphi$ vary according to the field of view of that camera and lens. When one takes enough pictures of a scene (i.e., gathers a lot of samples from the plenoptic function), reconstructing novel (i.e., unseen) viewpoints based on the captured data becomes a reasonable feat. Creating these unseen viewpoints is called light-field synthesis.

The remainder of this section is divided into two parts: a general overview of light-field-related technologies and a more detailed summary of existing work regarding Steered Mixtures of Experts (SMoEs) for light-field representations.

2.1. Light-Field-Related Works

This section provides an overview of approaches to represent plenoptic data. Not all representations are part of one coherent system that supports interactive rendering, bt they are discussed anyway to provide deeper insight into the specific properties we desire. We consider a few classes of representations, which are briefly discussed next.

2.1.1. Three-Dimensional Graphics

The 3D graphics technique aims to reconstruct the physical environment of a scene using textured 3D meshes, but this introduces noticeable artifacts due to the difficulty of the reconstruction process [3,4,5]. Textures fail to capture non-Lambertian lighting effects, leading to unnatural results [6]. Notable examples include Microsoft’s mixed reality solutions and 8i Studio [7,8].

2.1.2. Depth + Image-Based Techniques

Depth maps or other geometric information can be used to improve the ray sampling strategy. Unstructured Lumigraph Rendering was one of the first to use a real-time approach, using a blending field that determines the contribution of each input image to the synthesized view [9]. Neural Lumigraph Rendering improved upon this concept by optimizing a neural signed distance field and an emissivity function [10]. Another approach is the one by Overbeck et al., which combines image-based techniques and depth maps with video coding technology [11]. Broxton et al. used dynamic layered meshes textured by alpha-enabled video streams to handle reflections [12]. The MPEG-I standards suite and the MPEG Immersive Video (MIV) standard use atlases of texture and depth information, encoded with H.264 AVC, H.265 HEVC, or H.266 VVC [13]. In summary, these techniques are capable of producing real-time results, but they face challenges with disocclusions and ghosting artifacts.

2.1.3. Fourier Disparity Layers

Fourier Disparity Layers (FDLs) represent the light-field data as a stack of Fourier transformed images [14]. Every layer corresponds to a certain depth of visual information. FDL data are then compressed using HEVC to arrive at a compact file format [15]. As a translation in the pixel domain corresponds to a multiplication by a complex number in the Fourier domain, the authors present view interpolation within the plane of original viewpoints (i.e., two degrees of freedom). Additionally, thanks to the fact that every layer is already in the Fourier domain and separated by depth, simulating different apertures is an elegant process. Nonetheless, FDL is limited to two degrees of freedom in view interpolation, which restricts its flexibility for more complex 3D scene navigation.

2.1.4. Neural Radiance Fields (NeRFs)

NeRFs have garnered significant attention due to their impressive modeling and view-synthesis capabilities. The original NeRF implementation utilizes a dense neural network (5D input, position and direction; and a 4D output, density and RGB color) to perform volumetric rendering along marched rays [16]. This neural network is optimized using a differentiable ray-marching procedure by training on the rays corresponding to the source images’ pixel data. Since the release of Mildenhall et al.’s seminal paper, numerous improvements and/or variations have been proposed to the original NeRF implementation [17,18]. Most notably, Mip-NeRF [19] and Mip-NeRF 360 [20] were able to substantially improve upon the original NeRF’s quality. Additionally, instant-ngp improved the original NeRF’s performance by two orders of magnitude, achieving real-time inference speeds (above 1 Hz for 1080p resolution) [2]. More recently, Tri-MipRF was shown to simultaneously achieve Mip-NeRF 360’s quality while retaining instant-ngp’s performance [21]. While these frameworks are promising, their inference times are currently still far too high for real-time 6DoF camera-captured VR experiences (on commodity hardware).

2.1.5. Kernel-Based Methods

The work on Steered Mixture-of-Experts (SMoE) by Verhack et al. establishes the fundamentals for the kernel-based modeling of light fields and other multi-dimensional image modalities [22,23]. The core idea is that the plenoptic function underlying to a set of images captured from a scene is sufficiently simple to be able to approximate it in a continuous and geometrically sensible way. SMoE carries this out by composing the 4D light-field function using simple 4D building blocks called kernels. In essence, these are mixture models approximating the underlying plenoptic function. Once an SMoE model is built, the continuous plenoptic function itself is available, and synthesizing new views is conceptually as trivial as reconstructing captured views. The challenge is to make sure that the continuous plenoptic extension of the discretely captured pixel data makes sense such that the synthesis of unseen viewpoints produces reasonable results. Section 2.2 dives deeper into SMoE for light fields. Bochinski et al. and Liu et al. worked on similar concepts but applied them to images and video instead of to light fields [24,25]. Although Verhack et al. described synthesizing in-plane views for planar light fields, this synthesis process takes 10 s to 10 min per frame, which makes it inadequate for real-time free-viewpoint playback [23,26]. This synthesis algorithm was in essence a block-accelerated version of the per-pixel SMoE reconstruction process because it only considered nearby kernels for each block. More recently, kernel-based methods have gained more interest since the publication of 3D Gaussian Splatting [27] and 2D Gaussian Splatting [28]. The main difference between SMoE and Gaussian Splatting lies in the representation of direction-dependent light information. For SMoE, kernels live in four dimensions and change their position and appearance based on the requested virtual camera direction. In Gaussian Splatting, each kernel has a fixed position in 3D or 2D space combined with a set of spherical harmonics to code/represent the angle-dependent color emitted from the kernel position.

In summary, kernel-based methods are a promising approach for modeling light fields and synthesizing new views. This geometric and continuous extension allows for a smooth and intuitive reconstruction of unseen viewpoints. Currently, 3D Gaussian splatting kernels present a real-time solution, and SMoE lags behind 3D Gaussian splatting due to its slow view synthesis process, which can take between 10 s to 10 min per frame. This limitation is a key challenge that the current paper seeks to address. The next subsection will delve deeper into SMoE’s disadvantages.

2.2. Summary of SMoE for Light Fields

2.2.1. Preliminaries

SMoE is a mixture of experts inspired by Gaussian Mixture Models (GMMs) used to model camera-captured visual data. In the case of planar light fields, these data have four intrinsic dimensions: $c_{\mathrm{x}}$ (camera-x), $c_{\mathrm{y}}$ (camera-y), $p_{\mathrm{x}}$ (pixel-x), $p_{\mathrm{y}}$ (pixel-y). More specifically, when considering a two-dimensional array of cameras on a plane, every camera has a $(c_{\mathrm{x}},c_{\mathrm{y}})$ coordinate in that plane. Each camera takes a picture such that every pixel has a $(p_{\mathrm{x}},p_{\mathrm{y}})$ coordinate within the picture. Thus, every recorded sample (i.e., pixel) has a four-dimensional identifying coordinate $(c_{\mathrm{x}},c_{\mathrm{y}},p_{\mathrm{x}},p_{\mathrm{y}})$ with a corresponding color value $(c_{\mathrm{r}},c_{\mathrm{g}},c_{\mathrm{b}})$. Note that the order of the dimensions implies our coordinate space convention: first camera dimensions, then pixel dimensions.

An SMoE model is like a 4D canvas on which every component (i.e., expert) locally paints in a 4D region of the canvas. The paint color is determined by a component-specific color approximation function (i.e., regressor). These components paint on top of the canvas in a predetermined order with a certain opacity (also known as alpha), by means of alpha compositing (also known as alpha blending). This way, components work together to paint the objects and, more generally, the entire scene. An object being occluded by another object from a range of viewpoints is modeled by components painting over the region of the canvas that was already painted in by the other components representing the occluded object. This is very similar to how a video game can draw the (translucent) objects in the game world back to front, thereby occluding distant objects with nearby objects.

For a given point in the four-dimensional coordinate space, the alpha (i.e., opacity) and color approximation functions of all components are calculated. The colors are then alpha-composited on top of each other in the predetermined order of the model. When combining enough components, covering the entire region of captured samples, a full, high-quality model can be constructed.

In this work, the alpha functions are clipped density functions of multivariate normal distributions, while the color regressors are linear. The i-th component is characterized by its center ${\overrightarrow{\mu }}_i\in {\mathbb{R}}^4$, its covariance matrix $R_i\in {\mathbb{R}}^{4\times 4}$, its sharpness parameter $s_i\in \mathbb{R}$, its center color ${\overrightarrow{\xi }}_i\in {\mathbb{R}}^3$ with the corresponding color gradient matrix $W_i\in {\mathbb{R}}^{3\times 4}$, and an additional alpha-scaling factor ${\alpha }_i\in \mathbb{R}$. The alpha-function ${\gamma }_i\left(\overrightarrow{x}\right)\in \mathbb{R}$ for component i at position $\overrightarrow{x}\in {\mathbb{R}}^4$ is given by

$$\begin{array}{cc}\hfill {\gamma }_i\left(\overrightarrow{x}\right)≔& {\alpha }_{i}min(1,e^{s_i}\sqrt{{\left(2\pi \right)}^4\left|R_i\right|}\mathcal{N}(\overrightarrow{x};{\overrightarrow{\mu }}_i,R_i))\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill & ={\alpha }_{i}exp\left(-{\displaystyle \frac{1}{2}}max\left(0,{(\overrightarrow{x}-{\overrightarrow{\mu }}_i)}^{\mathsf{T}}{R}_i^{-1}(\overrightarrow{x}-{\overrightarrow{\mu }}_i)-2s_i\right)\right),\hfill \end{array}$$

(3)

where the exponent 4 corresponds to the dimensionality of the model and $\mathcal{N}$ represents the density function of a multivariate normal distribution. The multivariate normal distribution density function $\mathcal{N}$ is purposely rescaled with the factor $\sqrt{{\left(2\pi \right)}^4\left|R_i\right|}$. This factor is the inverse of the typical normalization factor found in 4-variate normal distributions, such that the choice of $R_i$ does not influence the peak amplitude of the alpha.

Given these alpha functions, all components are combined into the final color $\overrightarrow{f}\left(\overrightarrow{x}\right)\in {\mathbb{R}}^3$ by alpha compositing (on a black background)

$$\overrightarrow{f}\left(\overrightarrow{x}\right)≔\sum _{k=0}^{K-1}{\overrightarrow{f}}_k\left(\overrightarrow{x}\right){\gamma }_k\left(\overrightarrow{x}\right)\prod _{i=k+1}^{K-1}(1-{\gamma }_i\left(\overrightarrow{x}\right)),$$

(4)

where ${\overrightarrow{f}}_i:{\mathbb{R}}^4\to {\mathbb{R}}^3$ denotes the linear approximation function of the i-th component:

$${\overrightarrow{f}}_i\left(\overrightarrow{x}\right)≔{\overrightarrow{\xi }}_i+W_i(\overrightarrow{x}-{\overrightarrow{\mu }}_i).$$

(5)

2.2.2. Related Works in SMoE for Light Fields

Equation (4) allows us to evaluate an SMoE model at arbitrary positions if the model parameters are known. The question remains how one uses this formula in practice. The most straightforward approach is to implement the alpha-composited interpretation of this formula as CPU or GPU code and evaluate it for every pixel of the desired view. However, in order to evaluate this formula for one pixel, all components’ alphas and color approximations need to be computed using Equations (2) and (5). This approach would result in very poor performance, getting worse with increasing output resolution and an increasing number of model components. The complexity can be denoted as $O\left(KN\right)$, where K is the number of components and N is the number of output pixels.

3. Proposed Rendering Method

This section presents a novel method for efficiently synthesizing arbitrary viewpoints (including out-of-plane), by preprocessing the model parameters on a per-component basis (see Section 3.1), which are then efficiently rendered on a modern GPU pipeline (see Section 3.2). Loosely speaking, the complexity of this rendering method resembles $O(N+K)$, with K as sthe number of components and N as the number of output pixels. This is in contrast to the naive approach described in Section 2.2.2, for which the complexity is denoted as $O\left(KN\right)$. As such, our approach yields very fast render times, as evaluated in Section 4.

3.1. Constructing a View-Specific 2D Representation

Suppose that given a viewpoint $\overrightarrow{v}$, camera rotation, and projection matrix, we wish to synthesize a view from a 4D SMoE light-field model. This implies that the dimensionality of $\overrightarrow{x}$ in Equations (3) and (5) is also four. However, the rendered result for a particular view is naturally 2D (such as the 2D pixel grid of a digital monitor). To obtain a 4D model coordinate, we can cast a ray from the virtual viewpoint to the camera plane. As defined in Section 2.2.1, the four coordinates are camera-x, camera-y, pixel-x, and pixel-y. The first two are the camera-x and camera-y coordinates of the intersection point. The last two are the pixel coordinates obtained after projecting the ray direction using the projection matrix of the original cameras. This procedure is explained in detail in Appendix A.3. We can query the original 4D model by applying the described mapping procedure to every pixel of the virtual viewpoint. The mapping function is denoted as ${\overrightarrow{q}}_{\mathsf{s}}\left(\overrightarrow{s}\right)\in {\mathbb{R}}^4,\forall \overrightarrow{s}\in {\mathbb{R}}^2$. Naturally, using this function ${\overrightarrow{q}}_{\mathsf{s}}\left(\overrightarrow{s}\right)$ in Equations (3) and (5) gives 2D screen-space equivalents:

$$\begin{array}{cc}\hfill {\gamma }_{\mathsf{s},i}\left(\overrightarrow{s}\right)& ={\gamma }_i\left({\overrightarrow{q}}_{\mathsf{s}}\left(\overrightarrow{s}\right)\right),\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill {\overrightarrow{f}}_{\mathsf{s},i}\left(\overrightarrow{s}\right)& ={\overrightarrow{f}}_i\left({\overrightarrow{q}}_{\mathsf{s}}\left(\overrightarrow{s}\right)\right).\hfill \end{array}$$

(7)

These are, respectively, the alpha function and the color regression function of the i-th component in screen-space coordinates $\overrightarrow{s}$.

To devise a quick and specialized rendering algorithm on a GPU-based modern graphics pipeline, we need to extract the screen-space center and covariance matrix for every component. However, as Appendix A.2 elaborates on in detail, this mapping function is non-linear, which prevents us from extracting this information easily. To overcome this, we begin by replacing the non-linear mapping function ${\overrightarrow{q}}_{\mathsf{s}}$ by its first-order Taylor approximation. For every model component, we calculate the optimal point around which this Taylor approximation ${\overrightarrow{q}}_{\mathsf{s}}^{\prime }$ is developed. As such, we introduce the approximated screen-space equivalent functions based on ${\overrightarrow{q}}_{\mathsf{s}}^{\prime }$ instead:

$$\begin{array}{cc}\hfill {\gamma }_{\mathsf{s},i}^{\prime }\left(\overrightarrow{s}\right)& ={\gamma }_i\left({\overrightarrow{q}}_{\mathsf{s}}^{\prime }\left(\overrightarrow{s}\right)\right),\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill {\overrightarrow{f}}_{\mathsf{s},i}^{\prime }\left(\overrightarrow{s}\right)& ={\overrightarrow{f}}_i\left({\overrightarrow{q}}_{\mathsf{s}}^{\prime }\left(\overrightarrow{s}\right)\right).\hfill \end{array}$$

(9)

Simplifying these formulas allows us to extract the per-component screen-space information.

To conclude, in Appendix A, the reader can follow the (non-trivial) derivations to obtain the parameters and formulas of a two-dimensional component. All components’ two-dimensional approximations put together, in the same way as in Equation (4), form the visual output image of a virtual camera. This means that the camera position, orientation, and projection matrix can be configured freely.

3.2. GPU Implementation

In Section 3.1, we explained how to obtain a two-dimensional model that approximates the view as seen from a virtual camera. In this section, we present a technique to efficiently render such a 2D model, using the final forms of Equation (8) obtained in Appendix A, at Equations (A42) and (A43).

Our technique relies on the fact that every component’s impact becomes negligible at a certain distance from the component center ${\overrightarrow{\mu }}_i$ or equivalently from a certain distance from ${\overrightarrow{s}}_{\mathsf{opt}}$ on the screen. When considering the two-dimensional alpha function in Equation (A42), one can indeed see that as the Mahalanobis distance $M\left(\overrightarrow{s}\right)$ becomes larger, the alpha quickly becomes very small. Equation (A42) can be used to solve the Mahalanobis distance $\overline{M}$ at which the function yields a desired threshold alpha $\overline{\alpha }$. This Mahalanobis distance $\overline{M}$ can be used as a cut-off distance, beyond which we consider the impact of a component to be negligible. In the case of a monitor with eight-bit color depth, an $\overline{\alpha }$ between $0.125/256$ and $2/256$ is a reasonable range. As a good trade-off between speed and accuracy, we suggest $\overline{\alpha }=1/256$. Larger thresholds will cause the cut-off distance to be smaller but might start causing noticeable edges (this effect is evaluated later). Solving Equation (A42) for the screen-space cut-off distance $\overline{M}$ yields

$$\begin{array}{cccc}& & \hfill \overline{\alpha }& ={\alpha }_{i}exp\left(-{\displaystyle \frac{1}{2}}max\left(0,{\overline{M}}^2+{(\Delta \overrightarrow{q})}^{\mathsf{T}}{R}_i^{-1}\Delta \overrightarrow{q}-2s_i\right)\right)\hfill \\ \hfill \iff & & \hfill \overline{M}& =\sqrt{2log\left({\displaystyle \frac{{\alpha }_i}{\overline{\alpha }}}\right)-{(\Delta \overrightarrow{q})}^{\mathsf{T}}{R}_i^{-1}\Delta \overrightarrow{q}+2s_i}.\hfill \end{array}$$

(10)

The rendering of an SMoE model is a matter of a straightforward alpha compositing of every component, one by one, after transforming them from 4D to 2D using the formulas from Section 3.1. As our implementation was written in OpenGL, the remainder of this section uses OpenGL terminology, but can be it can be carried out in any graphics API supporting geometry shaders. Specifically, we propose the following implementation strategy.

First, store the parameters of the 4D components in a vertex buffer, such that one vertex contains the attributes of one component, and there are as many vertices as components. A single component should fit in nine 4-vectors.

The shader program consists of a vertex, geometry, and fragment shader. The vertex shader passes all vertex attributes as-is to the geometry shader.

The geometry shader is responsible for executing the required calculations described in the previous section to obtain the two-dimensional parameters for the current component. The required information to carry this out, such as the viewpoint $\overrightarrow{v}$, the matrices M and P of the virtual camera, and the $P_{\mathrm{o}}$ matrix, are all passed in as uniforms. Given this parameterization, the geometry shader should then emit a triangle strip with vertices at positions ${\overrightarrow{v}}_k$ in the shape of an ellipse around ${\overrightarrow{s}}_{\mathsf{opt}}$. The shape of the ellipse corresponds to the $2\times 2$ screen-space covariance matrix ${\widehat{R}}_i$ and is bound by the computed Mahalanobis-cutoff distance $\overline{M}$. We do this by multiplying 10 coordinates along the unit circle with the Cholesky decomposition of ${\widehat{R}}_i$, followed by a corrective factor.

$$\begin{array}{cc}\hfill {\overrightarrow{v}}_{k,\mathrm{circle}}≔& \overline{M}·\left[\begin{array}{c}\hfill cos({(-1)}^{k}2\pi k/10)\\ \hfill sin({(-1)}^{k}2\pi k/10)\end{array}\right],\forall k\in (0\dots 9),\hfill \end{array}$$

(11)

$${\overrightarrow{v}}_{k,\mathrm{ell}}≔\mathrm{chol}\left({\widehat{R}}_i\right)·{\overrightarrow{v}}_{k,\mathrm{circle}},$$

(12)

$${\overrightarrow{v}}_k≔{\overrightarrow{s}}_{\mathsf{opt}}+{\overrightarrow{v}}_{k,\mathrm{ell}}\frac{\overline{M}}{\Vert L_i^{-1}\left({\overrightarrow{q}}_{\mathsf{s}}\right({\overrightarrow{s}}_{\mathsf{opt}}\mathrm{+}{\overrightarrow{v}}_{k,\mathrm{ell}})-{\overrightarrow{q}}_{\mathsf{opt}})\Vert },$$

(13)

where,

$$\begin{array}{cccc}\hfill L_i≔& \mathrm{chol}\left(R_i\right)\hfill & & \in {\mathbb{R}}^{2\times 2},\hfill \end{array}$$

(14)

$$\begin{array}{cccc}\hfill {\overrightarrow{q}}_{\mathsf{s}}\left(\overrightarrow{s}\right)≔& D·{\overrightarrow{d}}_{\mathsf{n}}\left(\overrightarrow{s}\right)+\overrightarrow{e}\hfill & & \in {\mathbb{R}}^4,\hfill \end{array}$$

(15)

$$\begin{array}{cccc}\hfill {\overrightarrow{d}}_{\mathsf{n}}\left(\overrightarrow{s}\right)≔& -{\left(\overrightarrow{d}\left(\overrightarrow{s}\right)\right)}_{\mathrm{xy}}/{\left(\overrightarrow{d}\left(\overrightarrow{s}\right)\right)}_{\mathrm{z}}\hfill & & \in {\mathbb{R}}^2,\hfill \end{array}$$

(16)

$$\begin{array}{cccc}\hfill \overrightarrow{d}\left(\overrightarrow{s}\right)≔& M{P}^{-1}\langle \overrightarrow{s}|1\rangle \hfill & & \in {\mathbb{R}}^3.\hfill \end{array}$$

(17)

The Cholesky decomposition of ${\widehat{R}}_i$ is easy to compute in the shader itself:

$$\begin{array}{c}\mathrm{chol}\left({\widehat{R}}_i\right)≔\left[\begin{array}{cc}a& 0\\ b& c\end{array}\right],\end{array}$$

(18)

$$\begin{array}{c}a≔\sqrt{{\widehat{R}}_{i,11}},b≔{\widehat{R}}_{i,12}/a,c≔\sqrt{{\widehat{R}}_{i,22}-b^2}.\end{array}$$

(19)

The emitted vertices are additionally attributed with their corresponding ${\overrightarrow{v}}_{k,\mathrm{circle}}$ position, their color ${\overrightarrow{f}}_{\mathsf{s},i}\left(\overrightarrow{s}\right)$ (as determined by Equation (A43)), the alpha ${\alpha }_i$ and sharpness $s_i$, and the constant term ${(\Delta \overrightarrow{q})}^{\mathsf{T}}{R}_i^{-1}(\Delta \overrightarrow{q})$ in the squared Mahalanobis distance from Equation (A42).

The fragment shader then receives all interpolated vertex attributes. The squared Euclidean norm of the interpolated ${\overrightarrow{v}}_{k,\mathrm{circle}}$ serves as $M^2$ in Equation (A42), which is then evaluated to obtain ${\gamma }_{\mathsf{s},i}^{\prime }\left(\overrightarrow{s}\right)$ for the current pixel. The output of the fragment shader is the four-vector $\langle {\overrightarrow{f}}_{\mathsf{s},i}^{\prime }\left(\overrightarrow{s}\right)|{\gamma }_{\mathsf{s},i}^{\prime }\left(\overrightarrow{s}\right)\rangle$ containing the $(r,g,b,\mathrm{alpha})$ values.

We recommend a 16-bit (also known as, half-precision) floating-point frame buffer (GL_RGB16F) as a trade-off between speed and precision. As a side note, note that our models happen to use color values outside of the range $(0,1)$, and as such, the floating point formats are desired for correct reconstructions. Otherwise, an eight-bit color depth represented as integers would suffice (and in fact comes with a noticeable increase in speed). On the other hand, we have observed that 32-bit (also known as single-precision) floating point frame buffers (GL_RGB32F) cause a significant slowdown (up to 12 times slower) on several tested machines.

Finally, using the standard alpha-compositing blend mode (glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA)), the full pipeline computes the equivalent of Equation (4).

In conclusion, an overview has been presented of how a GPU-based rendering algorithm should be implemented, using the transformations described in Section 3.1. The necessary adjustments to improve the numerical stability of these computations are also discussed.

4. Evaluation

This section evaluates the speed and quality of the proposed method. More specifically, as the proposed dimensionality reduction method introduces some approximations, we quantified the resulting quality degradations in terms of peak signal-to-noise ratio (PSNR) and Structural Similarity Index Measure (SSIM) by comparing the quality results from the GPU implementation to those of the reference implementation on CPU, as explained in Section 4.1.3. The SSIM is calculated and averaged over R, G, and B color channels of the sRGB image data, using skimage.metrics.structural_similarity (A, B, data_range=1.0, gaussian_weights=True, use_sample_covariance=False, sigma=1.5, channel_axis=2, multichannel=True).

Additionally, as the primary goal of this proposed method of rendering is high performance, we measured the rendering speeds of our GPU implementation. As NeRFs are conceptually comparable to SMoE (i.e., volumetric rendering and the model-based representation of light-field data), we compare our proposed technique with the NeRF implementation of instant-ngp [2], which also focuses on delivering speed, and is the most commonly used implementation of NeRF (in terms of GitHub stars and forks). For extra reference, we report the speed of our block-optimized CPU reference renderer as explained in Section 4.1.3.

For our experiments, we perform the same view-rendering tasks using all three approaches. The dataset, reconstruction tasks, experiment setup, and results are discussed in the following sections.

4.1. Experiment Setup

4.1.1. Dataset

For our research and evaluation, we are seeking a dataset that has wide-area coverage for an immersive experience that allows users to walk around a few meters. We also prefer a dataset that is realistic, making it easier to assess the subjective quality. Additionally, we place importance on having good specular lighting and volumetric effects in the dataset as light fields can appear unimpressive without them. Datasets offering all of the above properties simultaneously were rare, so, in previous work, we created SILVR and made it publicly available [29].

SILVR is short for “Synthetic Immersive Large-Volume Ray dataset”, and it consists of 180° fisheye-projection images. We reprojected these images to square rectilinear images of size 512 px with a horizontal and vertical field of view of 102.68° (i.e., the equivalent of a 14 mm lens on a 35 mm by 35 mm sensor).

In addition to the SILVR scenes, we selected one frame from the real-world-captured “painter” [30] and synthetic “kitchen” [31] light-field video sequences, commonly used in the MPEG community. Camera-captured content contains inaccuracies caused by imperfect camera localization, color differences, and so on. These inaccuracies lead to reduced modeling performance, resulting in more components or lower visual quality. The impact on the proposed rendering technique is limited to an increase in the number of components used in the model. Example views can be seen in Figure 1.

In summary, we have the following scenes:

• barbershop (SILVR, cuboid/right face): The “barbershop” scene from the Blender Institute short movie Agent 327: Operation Barbershop [32] (licensed CC-BY). Camera grid: $30\times 10$, spaced 11 cm apart.
• lone monk (SILVR, cuboid/front face): This scene was created by Carlo Bergonzini from Monorender (licensed CC-BY). Camera grid: $21\times 21$, spaced 20 cm apart.
• zen garden (SILVR, cuboid/back face): This scene was created by Julie Artois and Martijn Courteaux from IDLab MEDIA (licensed CC-BY 4.0). Camera grid: $21\times 21$, spaced 10 cm apart.
• kitchen (MPEG, frame 20): This dataset by Orange Labs was software-generated and features a kitchen with a table ready for breakfast and an owl and a spider [31]. This dataset has a rather small area. Camera grid: $5\times 5$. Resolution $1920\mathrm{px}\times 1080\mathrm{px}$.
• painter (MPEG, frame 246): This dataset was captured with real cameras and features a man in front of a canvas [30]. This dataset has a rather small area and few specularities. Camera grid: $4\times 4$. Resolution $2048\mathrm{px}\times 1088\mathrm{px}$.

We used our still-in-development modeling software to create SMoE models from these datasets with 25,000 components. However, the number of components retained in the final models varied due to pruning strategies (barbershop: 16,938, lone monk: 15,910, zen garden: 13,668, kitchen: 21,634, painter: 21,647).

4.1.2. Reconstruction Tasks

We define several tasks to evaluate quality and performance. A task consists of a series of viewpoints (i.e., the combination of a position, camera orientation, and field of view). Within the MPEG community, these are called “pose traces”. The indicated tasks are the following (to visually examine the tasks, see the data availability statement):

• ‘spin’ A trajectory where the camera moves from left to right while panning from right to left.
• ‘push–pull’ A trajectory where the camera moves several meters backward, while the field of view narrows, similar to the cinematic push–pull or dolly zoom effect.
• ‘zoom-to-xyz’ A trajectory where the camera moves both forward and narrows the field of view slightly towards a certain object of interest in the scene.

All experiments were executed on a Linux machine (Super Micro Computer Inc., San Jose, CA, USA) with an Intel i7-5960X (Intel Corporation, Santa Clara, CA, USA) and an NVIDIA RTX 2080Ti (Nvidia Corporation, Santa Clara, CA, USA).

4.1.3. SMoE CPU Implementation Details

The CPU versions of the SMoE reconstructor are multi-threaded C++ programs using Eigen for the linear algebra operations. This program is compiled with GCC 9.4.0 with -O3 -mavx2. There are two variations, CPU-Full and CPU-Block, both implemented in the same codebase but with different execution strategies. The CPU-Full version evaluates all components of an SMoE model for all pixels. Because of that, the number of computations performed by this implementation is independent of the view: both the number of components of the model and the output resolution (i.e., the number of pixels) are the same for every rendered view.

The CPU-block version is a block-optimized interpretation of Equation (4). The view-synthesis task is split into square blocks, where for every block, only relevant components are considered. The block size is manually optimized per scene to find a trade-off between the number of components per the block (more unnecessary computations), and the number of blocks (more time spent selecting components for the blocks). If a component would cause a color contribution of more than a quarter of a quantization level (i.e., ${\displaystyle \frac{1}{4}}/256$), it is used for reconstructing the block. Blocks are constructed in parallel, on the eight-core test machine, using a queue. Rendering 10% of all views with CPU-Full, the CPU-Block version was validated to be correct, with PSNR scores ranging from 60 to 80 dB and maximal pixel error values of 1 (i.e., rounding error). As such, to reduce experiment time, we used CPU-block results for most of our comparative evaluations.

4.1.4. SMoE GPU Implementation Details

Every view in a task is rendered 20 times, to allow the GPU driver to properly pipeline consecutive frames, allow clock speeds to adjust, and reduce variability, in order to get reliable results. We measure the duration of the entire loop of 20 repetitions followed by a final call to glFinish(). This is repeated eight times, such that eight time measurements are performed, of which the fastest result is selected. The reported time is the average time per frame.

4.1.5. NeRF Experiment Details

For every scene, two NeRF models are built with “instant-ngp” [2]: base and small. The ‘base’ model is made with the authors’ default configuration. The “small” model starts from the base configuration but takes the authors’ suggestions on making the model faster for rendering (The “small” configuration uses $2^{11}$ instead of $2^{19}$ hash table slots. The slowest element of the instant-ngp NeRF rendering process is the hash table lookups. The neural network computations are in comparison fast and resolve hash collisions. Therefore, the “small” configuration speeds up rendering substantially by having faster hash table lookups but introduces more hash collisions by doing so, which are then taken care of by the neural network.) Creating the NeRF models, we positioned the camera plane parallel to the XY plane and adjusted it to scale and translation to fit the entire contents in the model bounding box. All reconstructions were generated using five samples per pixel (spp), again for pipelining and reducing overhead. The results are reported (in Section 4.2.3) by rescaling to 1 spp for a fair comparison with SMoE.

4.2. Results and Discussion

4.2.1. Speed Versus Quality Trade-off

The quality and speed aspects of our proposed method are under the great influence of the $\overline{\alpha }$ threshold in Equation (10). This $\overline{\alpha }$ threshold directly determines the component radius $\overline{M}$ and can thus control the number of pixels being evaluated. To this end, we quantified the impact of $\overline{\alpha }$ when varying this threshold between $0.125/256$ and $15/256$ and comparing the results to the exact CPU reference renders. As can be seen in Figure 2, higher threshold values will cause higher errors but will reduce rasterization workload and hence cause a speedup (approximately 40% faster at $\overline{\alpha }=2/256$). In practice, using values of $\overline{\alpha }$ up to 3/256 produces results with imperceptible artifacts (PSNR is still around 50 dB). As a demonstration of the type of artifacts to be expected with extreme values for $\overline{\alpha }$, Figure 3 shows the result of the tilted view from Figure 2 for $\overline{\alpha }=15/256$, which ends our test at 39 dB. As such, we have a way to trade off quality for speed, depending on what is desired. Note that the top and bottom edges of the view in Figure 3 correspond with the edges of the dataset. The modeling process relies on a diverse set of cameras that are not available in these edge regions. As a result, the quality in these areas is lower due to factors beyond our rendering approach.

When targeting a VR headset, 90 Hz is desirable. However, the workload is higher as two separate high-resolution views now need to be synthesized. In order to maintain the required 90 Hz dual-output playback speed on our HTC Vive Pro VR headset, we had to select $\overline{\alpha }=2/256$ on a NVIDIA RTX 2080Ti. As mentioned earlier, artifacts cannot be seen at this value of $\overline{\alpha }$, and in fact, we found the quality of the image to be satisfying until $\overline{\alpha }=5/256$ in our VR headset. A video demonstrating this can be found in the data availability statement. To conclude, the proposed method achieves real-time rendering speeds for SMoE.

4.2.2. Correctness

In Figure 4 and

Table 1. Validation of the proposed GPU implementation for the tested datasets. Note that rendering was performed on a resolution of $2048\times 2048$ with a threshold $\overline{\alpha }$ of $0.125/256$ to demonstrate exactness. Quality metrics are measured between the renders from the GPU implementation and the CPU-Block implementation as ground truth. Render times are per frame. The indicated numbers represent the mean and standard deviation over the resulting renders for each task.

Scene	Task	PSNR	Time GPU
barbershop	push–pull	$59.7$ dB	$4.5$ ms
	spin	$57.0$ dB	$4.6$ ms
lone monk	push–pull	$58.4$ dB	$5.3$ ms
	spin	$54.6$ dB	$5.2$ ms
zen garden	push–pull	$62.3$ dB	$4.1$ ms
	spin	$59.4$ dB	$4.5$ ms
kitchen	push–pull	$59.3$ dB	$4.3$ ms
	spin	$59.2$ dB	$4.2$ ms
	zoom-to-sink	$58.9$ dB	$3.5$ ms
	zoom-to-table	$58.8$ dB	$4.7$ ms
painter	push–pull	$61.7$ dB	$3.6$ ms
	spin	$61.5$ dB	$3.5$ ms
	zoom-to-painting-1	$61.7$ dB	$3.5$ ms
	zoom-to-painting-2	$61.0$ dB	$3.4$ ms

, we show an analysis of the correctness of our proposed method using the highest-quality setting ($\overline{\alpha }=0.125/256$). We compare the proposed method with the CPU ground truths based on the tasks described earlier. We observe that as long as the virtual camera is aligned with the original cameras of the dataset (as in push–pull), the results are nearly perfect. That is, the PSNR reaches beyond 60 dB, and the maximal pixel color error does not exceed two quantization levels. In fact, the source of the errors is the reduced precision of the 16-bit floating point frame buffer. Repeating the experiment for 32-bit floating point frame buffers yields a $\pm 2\mathrm{dB}$ increase along with a maximal pixel color error of only one quantization level.

As soon as the camera tilts out of the original optical axis (as in spin), the first-order Taylor approximation becomes apparent, and we observe a quality decrease to 45 dB in the worst measured case. This observation was consistent across the different scenes. It should be noted that the drop in PSNR due to the Taylor approximation is imperceptible in practical applications. Furthermore, the error introduced by this approximation is minimal compared to the visual artifacts caused by the modeling process, which suffers from a lack of captured data when rendering views from extreme angles, leading to reduced quality.

4.2.3. Comparison with NeRFs

Table 2. Comparison of the proposed SMoE rendering technique with instant-ngp NeRFs in terms of speed and model quality based on the original views. Note that SMoE GPU rendering was performed with a threshold $\overline{\alpha }$ of $1/256$. SMoE rendering speed drops when the resolution is small. This happens because resolution-independent work (such as performing the geometry shader computations, and other overhead) takes a more significant portion of the total time. The comparison is therefore more meaningful at higher resolutions (such as with “kitchen” and “painter”).

Model	Resolution	SMoE ($\overline{\mathit{\alpha }}=1/256$)				NeRF Base				NeRF Small
		PSNR	SSIM	Time	Speed	PSNR	SSIM	Time	Speed	PSNR	SSIM	Time	Speed
barbershop	$512\times 512$	26.6 dB	0.85	1.2 ms	222 MP/s	31.1 dB	0.95	65 ms	4.2 MP/s	26.3 dB	0.81	21 ms	12.6 MP/s
lone monk	$512\times 512$	24.9 dB	0.78	1.1 ms	234 MP/s	27.9 dB	0.88	52 ms	5.5 MP/s	23.6 dB	0.70	16 ms	16.4 MP/s
zen garden	$512\times 512$	29.2 dB	0.78	1.0 ms	259 MP/s	32.0 dB	0.88	112 ms	2.4 MP/s	28.4 dB	0.73	43 ms	06.2 MP/s
kitchen	$1920\times 1080$	28.6 dB	0.82	2.4 ms	865 MP/s	36.5 dB	0.95	799 ms	2.6 MP/s	27.0 dB	0.75	229 ms	09.1 MP/s
painter	$2048\times 1088$	29.6 dB	0.82	2.4 ms	935 MP/s	36.2 dB	0.93	1425 ms	1.6 MP/s	29.2 dB	0.78	650 ms	03.4 MP/s

compares our proposed SMoE rendering approach (with $\overline{\alpha }=1/256$) to the performance of NeRFs of “instant-ngp” [2]. These results are obtained by reconstructing the views corresponding to the original images from the dataset and measuring the quality. Observe that instant-ngp’s NeRFs with the small configuration require on average 229 ms to reconstruct the 1080p resolution views of kitchen. As such, these NeRFs are not suitable for real-time interactive frame rates at a 1080p resolution. On the other hand, SMoE reconstructs the same 1080p views in 2.4 ms, reaching 416 Hz (i.e., far above the typical 60 Hz for interactive desktop applications). Additionally, for similar resolutions, instant-ngp has rendering times that vary greatly across scenes (e.g., for the tests at $512\times 512$, it varies from 52 ms for lone monk to 112 ms per frame for zen garden for the base configuration and from 16 ms to 43 ms per frame for the small configuration). SMoE has a more predictable performance (i.e., between 1.0 ms and 1.2 ms correspondingly). We hypothesize that the greater variation in NeRF’s cross-scene rendering times stems from its use of geometry-dependent optimizations. For example, NeRF uses occupancy grids, whose performance highly depends on the scene’s geometry, while our proposed method uses no such geometry-based optimizations. Additionally, our rendering time complexity is practically dominated by the scene-independent number of pixels N and not by the scene-dependent number of components K. More research can be carried out to validate our hypothesis and quantify how scene-dependent elements influence the rendering times.

Table 3. Relative results from Table 2 when comparing SMoE ($\overline{\alpha }=1/256$) with both NeRF configurations of instant-ngp.

Model	SMoE vs. NeRF Base		SMoE vs. NeRF Small
	$\Delta$PSNR	Speedup	$\Delta$PSNR	Speedup
barbershop	−4.5 dB	$\phantom{0}\times 53$	+0.3 dB	$\phantom{0}\times 18$
lone monk	−3.0 dB	$\phantom{0}\times 42$	+1.3 dB	$\phantom{0}\times 14$
zen garden	−2.9 dB	$\times 109$	+0.8 dB	$\phantom{0}\times 42$
kitchen	−7.9 dB	$\times 329$	+1.6 dB	$\phantom{0}\times 96$
painter	−6.6 dB	$\times 597$	+0.4 dB	$\times 273$

compares the results from Table 2 in terms of speed and quality. We see that SMoE is up to 273 times faster than the small NeRF configuration and achieves comparable quality. When considering the default base NeRF configuration, SMoE is up to 597 times faster. While instant-ngp NeRFs do achieve around 3 dB to 8 dB higher PSNR scores, the rendering time for a single frame at 1 sample per pixel is above half a second.

Figure 5 shows a visual comparison between the ground truth, SMoE, NeRF base, and NeRF small for the scene painter. As confirmed by the results from Table 2 and Table 3, NeRF base has the highest visual quality, whereas NeRF small has the lowest, and SMoE has a slightly better quality than NeRF small. The lower SMoE model qualities are because our modeling software has difficulties accurately representing content near the edges of the data domain, among a few other things. Improving the SMoE modeling techniques is a subject for future work.

To expand in greater detail on the results of Table 2, Figure 6 shows a scatter plot of the quality and reconstruction time for each individual reconstructed view. We can see that all combinations of scenes and approaches yield compact clusters, which is desirable for a predictable and consistent quality of experience.

We do not anticipate that the proposed technique will be memory-constrained as storing the largest scene (Painter) requires less than 5 MiB, which can be further reduced by, for example, leveraging the symmetry of the covariance matrix. For completeness, an indication of the file sizes of each SMoE and NeRF model after ZIP compression is given in

Table 4. Comparison of ZIP-compressed file sizes of the model files from each technique. File sizes are expected to be significantly lower using custom-engineered entropy coding.

Scene	SMoE	NeRF Base	NeRF Small
barbershop	3.6 MB	24 MB	2.8 MB
lone monk	3.0 MB	24 MB	1.8 MB
zen garden	3.1 MB	24 MB	2.5 MB
kitchen	3.6 MB	24 MB	1.6 MB
painter	3.6 MB	23 MB	1.1 MB

. For both, it is important to stress that this general-purpose ZIP compression is inefficient compared to the compression ratio we expect from custom-made entropy coding. This table is therefore given as a mere order-of-magnitude indication of the size of these model files.

To conclude, when model quality is on par with NeRF, the proposed light-field-rendering technique is 273 times faster, although the maximum quality we are able to reach with the current state of the SMoE modeler is still 3 dB to 8 dB lower. Improving the SMoE models themselves is a subject for future work, but we foresee the impact of having better models (with more components) to minimally impact the rendering performance, and potentially even improve if those models use fewer components per pixel.

5. Conclusions

In this paper, we have proposed two important techniques that together enable efficient real-time free-viewpoint 6DoF Steered Mixture of Experts rendering on GPUs. First, the mathematical operations required to transform the 4D planar light-field model into a view-specific 2D model were derived. Next, a description of how to render these 2D models on a modern GPU, using OpenGL employing a geometry shader was presented. The mathematical transformation is lightweight compared to the fragment shading workload. Configuring the proposed GPU implementation for high accuracy, 180 to 290 Hz at a resolution of $2048\times 2048$ pixels on an NVIDIA RTX 2080Ti is achieved. The mathematical formulas derived make use of an approximation, which is partially compensated. This approximation degrades rendering accuracy, which is quantified with PSNR between 47 dB and 63 dB or SSIM values between 0.9985 to 0.9995.

Comparing our light-field-rendering solution to instant-ngp neural radiance fields (NeRFs), we achieved 42 to 597 times faster rendering while scoring around 6 dB lower PSNR when reconstructing the original views. Reducing the complexity of the NeRF model and bringing the quality on par with SMoE, we still achieve 14 to 273 times faster rendering. Additionally, when allowing for (near-)imperceptible artifacts in reconstruction quality (not changing the model), our proposed SMoE rendering technique gains an additional 40% speedup. Based on the presented work, it can be stated that SMoE becomes a viable representation and framework for representing interactive light fields.

Future work can study and improve upon the approximations, as well as investigate new techniques for further improving the rendering performance. More broadly, future work should include improving the quality of the models themselves.