Improving Monocular Camera Localization for Video-Based Three-Dimensional Outer Ear Reconstruction Tasks

Abstract

This work addresses challenges related to camera 3D localization while reconstructing a 3D model of an ear. This work explores the potential solution of using a cap, specifically designed not to obstruct the ear, and its efficiency in enhancing the camera localization for structure-from-motion (SfM)-based object reconstruction. The proposed solution is described, and an elaboration of the experimental scenarios used to investigate the background textures is provided; data collection and software tools used in the research are reported. The results show that the proposed method is effective, and using the cap with texture leads to a reduction in the camera localization error. Errors in the 3D location reconstruction of the camera were calculated by comparing cameras localized within typical ear reconstruction situations to those of higher-accuracy reconstructions. The findings also show that caps with sparse dot patterns and a regular knitted patterned winter hat are the preferred patterns. The study provides a contribution to the field of 3D modeling, particularly in the context of creating 3D models of the human ear, and offers a step towards more accurate, reliable, and feasible 3D ear modeling and reconstruction.

1. Introduction

After the sound wave reaches the listener, the size and shape of the ear changes the spectrum of sound reaching the eardrum. These spectral distortions act as a unique feature of the human body to help us understand the location of the sound source. This phenomenon is called the head-related transfer function (HRTF)—a response describing how we receive sound waves from different points in space. Other parts of the human body such as the head, shoulders, and torso also change the sound spectrum, but only frequencies up to 3 kHz are affected in this way. Frequencies above 3 kHz are shaped by the individual anatomy of the external ear [1]. Hair can also affect the HRTF spectrum, especially at high frequencies, but studies have shown that this should not affect the user’s ability to accurately localize a sound source [2].

The pinna is one of the most individual parts of the human body, even variations of a few millimeters in its geometry have a strong effect on the HRTF filter [3]. When analyzing the entire human population, different pinna shapes result in changes of up to 20 dB in the HRTF spectrum above 4 kHz [4]. However, it has been shown that not all parts of the pinna are equally important for the HRTF spectrum. The cavum conchae, fossa triangularis, and scapha have the greatest influence, while the posterior half of the helix changes the spectrum the least [5].

There are two main factors that allow a human listener to locate a sound source. These are the interaural time difference (ITD) and the interaural level difference (ILD). An ITD occurs when the sound source is at a different distance from the two ears. Studies show that humans can detect ITDs as small as 10 µs, which corresponds to differences in distance of about 0.3 mm [6]. The ILD is the result of the acoustic shadow created by the head, which acts as a barrier to high frequencies. For example, a wavelength of 2000 Hz is shorter than the distance between human ears. Below 2000 Hz, ILD becomes ineffective because the lower frequencies are able to bypass the detected obstacle [7].

The spatial resolution of an HRTF is determined by the number of positions of the sound source distributed over the surface of the virtual sphere in a certain interval of degrees. It has been shown that this resolution may not be the same for sounds coming from the front and from the side. The minimum audible angle (MAA) is the smallest angular deviation perceptible to a person; on the horizontal axis, it is 1 degree for frontal sounds and proportionally increases to 10 degrees for lateral sounds. On the vertical axis, a person feels a change in frontal sounds from 4 degrees, and for sounds coming from the side, the MAA increases up to 20 degrees [8].

Individual HRTFs are important for improving the user experience in virtual reality. Only by using it can a person hear the virtual sound through headphones exactly as they would hear the sound if the source existed in the natural environment. Use cases range from entertainment games to assistive systems for the blind. The user can be navigated by voice, giving them the right direction and eliminating the need to look at a map [9]. Online work meetings that have become popular during the COVID-19 pandemic could be more realistic, with participants hearing immersive virtual acoustics. Everyone who joined the conference could sit in a different part of the virtual room and hear the information being said by the other participants from the side corresponding to their position.

Most of the virtual reality products available today still use a common HRTF, which is the average of several measured subjects, but averaging over individuals does not produce good perceptual results [10]. Virtual source localization experiments have shown that the use of a common HRTF has a very high possibility of up–down and front–back confusion, while the use of an individual HRTF significantly reduces the occurrence of this effect [11].

One way to obtain a personalized HRTF is through binaural acoustic measurements, where microphones are inserted into the subject’s ear canal and a large number of sound sources are placed in a semicircle around the human head at different elevation angles, but maintaining the same radius. The subject is seated in an automated rotating chair and measurements are taken each time the chair is repositioned at a fixed azimuth [12]. Such measurements are difficult and available only for scientific purposes, requiring expensive equipment, an anechoic room, and expertise. Although there have been attempts to simplify this process to one that could be performed at home using only a single speaker and a head tracking system [13], or using a smartphone by moving it around the head, storing the phone’s position in space, and recording the test signal played by the phone with microphones built into headphones [14].

Researchers are constantly looking for ways to easily obtain as close to an individual HRTF as possible without complicated measurement procedures. There are suggestions to select the most appropriate HRTF from the ones available by analyzing the anthropometric data of the subject whose personalized HRTF we are looking for and identifying the most similar object in the selected database [15,16]. Another way to select the HRTF matcher is to classify the anthropometric data of the database objects into clusters, highlight the centers of these clusters, identify which cluster best fits the subject’s ear, and select the central HRTF of that cluster as the subject’s individual HRTF [17].

In addition to acoustic measurements and the most similar HRTF fitting methods, numerical calculations are also possible when trying to calculate the expected HRTF spectrum by analyzing how sound waves of different frequencies will reach the eardrum point geometrically from all possible directions. For such algorithms, it is necessary to provide an accurate 3D model of the head and ears. The mesh can be created in many ways—the head can be laser scanned [18] or it can be reconstructed photogrammetrically [19,20]. With an accurate 3D model, we can calculate a human-customized HRTF filter; researchers have developed tools for this task [21]. An HRTF filter can also be computed using deep neural networks (DNNs) with anthropometric measurements and ear images [22] or sound source directions [23] as input. DNN can also be used to detect an ear in an image and automatically label the points that define its shape [24,25,26,27,28,29].

Image-based reconstruction and modeling of objects [30,31,32], scenes [33,34,35,36,37,38], or processes [39,40], is a widely available and relatively inexpensive technique for information acquisition [41,42,43]. One of the photogrammetric methods is the structure-from-motion (SfM) technique [44,45,46]. This provides a noninvasive, cost-effective means of creating three-dimensional models of objects and environments from sets of two-dimensional images [47,48,49,50,51]. The SfM approach works by first estimating camera positions and then progressively reconstructing the 3D coordinates of the points in the scene [52,53,54]. SfM is highly valuable in the sense that it allows for the reconstruction of 3D structures using images taken from hand-held cameras [55,56,57,58,59]. However, the accuracy of SfM is intrinsically linked to the quality of the input images and the precision of camera localization [60,61,62].

Deep neural networks have been used to implement SfM tasks that include feature detection, matching, and outlier rejection [63,64,65,66,67]. Deep learning can automatically learn feature representations and mapping functions from large volumes of data, which has been shown to improve the robustness and accuracy of SfM [68,69,70]. Deep learning has also been combined with traditional SfM techniques to create hybrid models, which can take advantage of the strengths of both approaches [71,72]. Deep neural networks can be used for monocular depth and camera motion estimation to regress the camera pose [73,74,75].

An essential factor in the success and precision of object 3D reconstruction is the accurate localization of the camera in a three-dimensional space [76]. As the SfM technique depends on determining the three-dimensional structure of a scene from a series of two-dimensional images taken from different viewpoints, accurately determining the camera’s spatial position and orientation at each viewpoint is paramount [41,77]. Inaccuracies in camera 3D localization can lead to errors in the derived depth information, the resulting three-dimensional reconstruction, and any subsequent analyses [78]. Such errors can propagate and magnify, ultimately undermining the fidelity of the reconstructed model. This can, in turn, adversely impact the usability of the ear’s 3D model to derive precise HRTF.

This work addresses challenges related to camera 3D localization while reconstructing an ear’s 3D model. Efforts are put towards developing and refining methodology that enhances the precision of camera 3D localization. This will pave the way for more accurate, reliable, and robust 3D reconstruction of the ear.

The novelty and contributions of this work can be summarized as follows:

• Proposed a simple solution for the improvement of camera 3D localization for the ear 3D reconstruction tasks. The proposed solution is based on enriching the background texture.
• Presented a dataset construction approach for the evaluation of camera 3D localization.
• Performed comparative evaluation of different cap textures’ influence on the precision of camera 3D localization.
• Presented comparative results of camera 3D localization improvement solutions; the results show that the proposed method is effective and using a cap with texture allows a reduction in camera localization error. The results also show that caps with sparse dot patterns and a regular knitted patterned winter hat are the preferred patterns.

The problem of and proposed solution for camera 3D localization improvement is summarized in Figure 1.

The outline of the paper is as follows. In Materials and Methods (Section 2), the proposed solution for improving camera 3D localization via enhancing the background texture around the ear is described; an elaboration of the experimental scenarios used to investigate the background textures is provided; the data collection and software tools used in the research are reported. Results (Section 3) gives experimental comparison results of the background enhancement approaches using various caps and the overall usefulness of the solution and provides an interpretation of the findings and practical implications. Finally, Section 4 gives the conclusions of this work.

2. Materials and Methods

Three-dimensional reconstruction of an object, an ear in our case, by leveraging a dataset of photographs and applying the structure-from-motion (SfM) technique, requires a precise computation of camera positions.

SfM generates 3D reconstructions of an object or a scene from a series of 2D images, taken from different viewpoints. The process also estimates the camera’s position and orientation for each of the images. The overall SfM pipeline consists of several steps: feature extraction; feature matching; camera motion estimation; dense point cloud generation; mesh generation; and texturing.

The key mathematical concepts underlying SfM are projective geometry and optimization. The relationship between the 3D point and its 2D projection on an image can be expressed using homogeneous coordinates and the camera projection matrix. The purpose of the optimization process is to find the 3D structure and camera parameters that best explain the observed 2D image points. This is usually formulated as a non-linear least-squares problem.

Performing photogrammetry using the SfM technique for the reconstruction of the ear requires a sequence of actions that must be carried out. First, capturing the ear from various angles is a necessary step. This step provides a comprehensive set of images necessary for the reconstruction process. Second, the removal of the background in the images. The background is unsuitable for the reconstruction process due to the head movements relative to the background during the capture process. An unfixed background can lead to inaccurate results in the reconstruction process. Third, calculating camera positions in 3D space is an essential process. These positions can be determined by associating feature points within the head region across different frames. However, a notable problem arises in the side area of the head—the texture is not rich, therefore, it is difficult to extract a sufficient number of distinctive feature points that can be accurately matched across the frames.

In this research, we propose augmenting the side region of the head with a suitable texture for the reconstruction process. This research explores the potential solution of using a cap and its efficiency in enhancing the camera 3D localization for SfM-based object reconstruction.

2.1. Improvement of Camera 3D Localization

The primary source of camera 3D localization inaccuracies is the scarcity of texture on the side of the head, which leads to the lack of unique feature points that can be tracked in the image set.

The solution we are investigating is aimed at improving texture in the background of images of the ear in a such way that the background would be fixed relative to the ear. There are several requirements for the approach: (1) the simplicity of implementation, (2) actual improvement in the 3D localization of cameras, and (3) an additional advantage would be if it would help to reconstruct the metric scale of the scene.

The most convenient solution to add texture to the face, without obstructing the portion under reconstruction (the ear), is to place a cap with an appropriate texture on the head, ensuring that it does not cover the ear. This brings us to the research questions we seek to answer in this study: what type of texture is best suited for this purpose, and does the placement of such a cap indeed improve camera localization. To answer these questions, 13 textures cases were designed and investigated as separate experimental scenarios. These experimental scenarios are depicted in Figure 2 and summarized in Section 2.2.

2.2. Experimental Scenarios

The experimental scenarios were carefully designed to investigate the usefulness of the proposed solution to enhance the background texture around the ear using a cap. To evaluate the potential benefits of wearing a cap, scenarios both with and without a cap were set up. To compare the usefulness of different textures on the caps, 12 additional scenarios were tested, each with a distinct cap texture. These different setups allowed for a comprehensive analysis of the influence of the cap texture on the enhancement of the background texture around the ear, which is an important factor for accurate 3D localization of cameras.

The experimental scenarios (cases) for the investigation of the effect of cap usage on camera 3D localization accuracy (refer to Figure 2) were as follows:

• No cap—regular case of image collection for ear reconstruction;
• Dots 1—regularly arranged dot pattern of the 1st size;
• Dots 1s—sparse dot pattern of the 1st size;
• Dots 2—regularly arranged dot pattern of the 2nd size;
• Dots 2s—sparse dot pattern of the 2nd size;
• Dots 3—regularly arranged dot pattern of the 3rd size;
• Dots 3s—sparse dot pattern of the 3rd size;
• Dots 4—regularly arranged dot pattern of the 4th size;
• Dots 4s—sparse dot pattern of the 4th size;
• Painted blobs—pattern of randomly painted blobs of different sizes;
• Checkerboard—a well-known pattern used for camera calibration;
• Printed irreg.—printed pattern of irregularly scattered small dots;
• Winter hat—a regular knitted patterned winter hat.

2.3. Camera Localization Quality Evaluation

The effects of cap usage on the accuracy of camera 3D localization were evaluated and compared across various experimental scenarios. Errors in the 3D location reconstruction of the camera were calculated by comparing camera localizations within typical ear reconstruction situations to those of higher accuracy. Figure 3 presents the experimental setup used to assess the precision of the camera reconstruction. The upper branch is dedicated to the calculation of high-accuracy camera localizations, while the lower branch illustrates a typical ear reconstruction situation. In a standard ear reconstruction scenario, the background around the head in the image would be removed because it is typically non-static and unsuitable for the photogrammetric reconstruction of the ear.

High-accuracy camera localization is achieved by taking advantage of a specially created background with a rich texture placed behind the mannequin’s head. This background is not only texturally rich, but also non-flat. To simulate images that would be acquired for reconstruction of the ears, the surrounding background of the head in the images was masked. The position of the head in the image sequence was tracked and the same bounding box (BBox) of the head was transferred between all images in the set.

Camera localization was performed using the Meshroom software and the structure-from-motion algorithm for sparse reconstruction. Sparse reconstruction was performed for both initial and masked images. The 3D locations of the reconstructed cameras were aligned by a 3D similarity transformation, comprising translation, rotation, and scaling. This research did not involve reconstruction of the absolute scale of the model. To estimate the absolute scale, additional information is required [79]. Scale differences were eliminated during the alignment of the 3D locations of the reconstructed cameras in the test cases to the 3D locations of the reference camera. Therefore, the comparative evaluation of the cap usage scenarios does not require scale information.

After aligning the locations of the camera sets, the Euclidean distances between the locations were computed. The median of these distances was used as an error metric for camera 3D localization within a specific cap type and experimental setup scenario (camera distance and camera motion trajectory).

2.4. Software Used

The software tools and programming languages used in this research are as follows:

• MATLAB programming and numeric computing platform (version R2022a, The Mathworks Inc., Natick, MA, USA) for the implementation of the introduced improvements to the baseline reconstruction algorithm by integrating with AliceVision/Meshroom; MATLAB was also used for data analysis and visualization;
• Meshroom (version 2021.1.0) (https://alicevision.org accessed on 15 September 2022) [80], 3D reconstruction software based on the AliceVision photogrammetric computer vision framework. Used for the execution of the SfM reconstruction algorithms;

2.5. Setup and Data Collection

The study involved the collection of an image dataset designed specifically for the experimental evaluation of the proposed solutions aimed at enhancing camera 3D localization. This dataset was acquired through the recording of videos and extracting frames. A total of four different experimental setups were implemented, each consisting of 13 experimental cases (scenarios).

The four experimental setups differed in the distances between the camera and a mannequin head, along with variations in the radius of the camera’s circular motion trajectory. After setting the distance from the camera to the mannequin’s head, the radius of the camera’s trajectory was tuned so that the head of the mannequin would never go outside of the frame while the camera was moving. The 13 experimental cases (scenarios) consisted of cases involving the presence of different caps or the absence of a cap on the mannequin’s head (summary in Section 2.2).

The precision of the reconstructed 3D locations of the reference camera was increased by specially creating a background. A background image has to have a rich pattern, which is favored by the photogrammetry algorithms. The background was added in such a way that later it could be completely masked using image processing steps after detecting the head’s bounding box (BBox). An image of the mannequin’s head without a background was used to perform 3D localization of the cases under investigation.

The videos were acquired using the smartphone Samsung Galaxy S10+ standard camera app. For the comparative evaluation of the proposed solutions (head caps), 195 videos were taken. The acquisition conditions can be grouped into four experimental setups: videos were recorded at four different distances and slightly different camera motion trajectories. The capture conditions of the same experimental setup were kept as homogeneous as possible: fixed orientation of the smartphone, lighting conditions, artificial background, frame rate of 24 frames/s, frame size of $3840\times 2160$ pixels, ISO 800, shutter speed 1/350, F1.5, average length of the videos $17.2\pm 2.6$ s. The movement pattern of the phone while capturing was the same for all videos—a circle, the radius of which changed between the four experimental setups. The motion of the camera in a circular or near circular trajectory was established with the help of a device which allowed repeating the same camera motion trajectory while changing experimental cases.

3. Results and Discussion

In this research, we addressed the problem of camera 3D localization for tasks involving the reconstruction of a 3D model of the ear. We proposed a simple solution to improve the accuracy of 3D camera localization based on enriching the background texture—specifically, using a hat with a particular pattern.

To ascertain the effectiveness of this ’hat’ solution and compare how different patterns might enhance camera localization accuracy, we generated a specific dataset to test these hypotheses. We performed a comparative evaluation of the influence of different cap textures on the precision of 3D camera localization and presented the comparative results of solutions to improve camera 3D localization in Figure 4.

The results of the experimental comparison presented in Figure 4 comprise a comparison of 13 experimental scenarios (cases). The first column (block of data points) in the chart presents the results of the baseline case when no cap was used. The subsequent columns display the results of scenarios when different caps were used. The experimental scenarios are depicted in Figure 2 and summarized in Section 2.2. The camera 3D localization accuracy was evaluated in different experimental cases (the evaluation method is described in Section 2.3), and the errors are graphed in Figure 4.

The data for evaluation were collected in four different experimental setups (as described in Section 2.5); therefore, the results are grouped in Figure 4, and the color along with the marker type denotes the particular experimental setup. Each data point corresponds to a separate experiment. The data points are arranged in four virtual columns, and the fifth column shows the summary statistics using a box plot that incorporates all the data points from the first four columns.

The main conclusion drawn from the results presented in Figure 4 is that the use of a hat reduces camera localization errors—errors in the first data block (“no cap” experimental case) are the highest.

We observed that some reconstruction experiments were unsuccessful (the photogrammetry pipeline failed to reconstruct the sparse scene). These failures are summarized and presented as bar charts beneath the horizontal axis. The height of the bar indicates the percentage of failed reconstructions. The “dots 4” (regularly arranged dot pattern of the fourth size dots), “dots 3”, “checkerboard”, “no cap”, and “dots 1” cases included failed reconstructions.

The lowest reconstruction errors and all successful camera 3D location reconstructions were achieved using a regular knitted patterned winter hat (“winter hat” case), caps with sparse dot patterns (“dots 1s”, “dots 4s”, “dots 3s”, “dots 2s”), and a hat with a pattern of randomly painted blobs of different sizes (“painted blobs”). All the hats with sparse-dotted patterns performed well regardless of the pattern’s dot size. Caps with a printed dot pattern offer the greatest potential to aid in the reconstruction of the scale of the 3D ear model. However, the weakness of the “painted blobs” hat is that it was painted manually.

The results demonstrate that the proposed method is effective and that the use of a cap with texture reduces camera localization error. The findings also suggest that caps with sparse dot patterns and a regular knitted patterned winter hat are the preferred patterns.

4. Conclusions

This study provides a contribution to the field of 3D reconstruction, particularly in the context of creating a 3D model of the human ear. Our research shows that using a cap, specially designed not to cover the ear, is a straightforward and efficient solution to improve camera localization in this task.

The effectiveness of this approach comes from the enrichment of the background in the image of the ear. The fixed relationship between the cap and the ear prevents independent movement of the background and usage of an appropriate cap ensures that the ear remains visible.

The 3D camera localization accuracy was evaluated in 13 experimental scenarios—12 different caps and no cap. The experimental results show that the use of a cap with texture reduces camera localization errors. In some cases, performing sparse reconstruction without the cap is impossible, i.e., the structure-from-motion (SfM) algorithm from Meshroom fails to determine the camera positions and to reconstruct the scene. Our experiments demonstrate that the use of caps can alleviate this issue by aiding in the determination of camera positions. The results demonstrate that caps with sparse dot patterns and a regular knitted patterned winter hat are the preferred patterns.

Furthermore, a cap with the appropriate texture could also be useful for reconstruction of the metric scale of the object. This is crucial when metric measurements are required. Further experiments can be carried out to compare cap patterns for reconstruction of the metric scale of the ear.