6-DoF Pose Estimation from Single RGB Image and CAD Model Retrieval Using Feature Similarity Measurement

Abstract

This study presents six degrees of freedom (6-DoF) pose estimation of an object from a single RGB image and retrieval of the matching CAD model by measuring the similarity between RGB and CAD rendering images. The 6-DoF pose estimation of an RGB object is one of the important techniques in 3D computer vision. However, in addition to 6-DoF pose estimation, retrieval and alignment of the matching CAD model with the RGB object should be performed for various industrial applications such as eXtended Reality (XR), Augmented Reality (AR), robot’s pick and place, and so on. This paper addresses 6-DoF pose estimation and CAD model retrieval problems simultaneously and quantitatively analyzes how much the 6-DoF pose estimation affects the CAD model retrieval performance. This study consists of two main steps. The first step is 6-DoF pose estimation based on the PoseContrast network. We enhance the structure of PoseConstrast by adding variance uncertainty weight and feature attention modules. The second step is the retrieval of the matching CAD model by an image similarity measurement between the CAD rendering and the RGB object. In our experiments, we used 2000 RGB images collected from Google and Bing search engines and 100 CAD models from ShapeNetCore. The Pascal3D+ dataset is used to train the pose estimation network and DELF features are used for the similarity measurement. Comprehensive ablation studies about the proposed network show the quantitative performance analysis with respect to the baseline model. Experimental results show that the pose estimation performance has a positive correlation with the CAD retrieval performance.

1. Introduction

In 3D computer vision, 6-DoF (degrees of freedom) pose estimation is a study of finding the 3D position and 3D orientation of a real object in images or videos. Precise knowledge of the object’s 6-DoF pose with respect to the camera coordinate system enhances the performance in various applications such as eXtended Reality (XR), Augmented Reality (AR), robotic pick and place, autonomous vehicle, etc. In the studies of robotic pick and place, precise pose estimation is essential for picking up objects using camera images. The complete pose information of an object can be effectively used even when the object is partially occluded. In the studies of autonomous vehicles, estimating the pose of surrounding objects is necessary for recognizing the environment and obstacles, thereby enabling stable and safe driving [1].

For XR and AR applications, 6-DoF pose estimation of real objects is necessary to precisely align graphic models with the objects in the image space. A graphic model, such as a CAD model in the world coordinate system, should be transformed and projected to the image plane of the camera so that the projection of the CAD model is precisely aligned with the object area. Especially for XR and AR applications, the shape-matching model with the real object should be retrieved from a CAD model database before aligning the CAD model with the object in the image space. Therefore, not only the pose estimation of an RGB object but also the CAD model retrieval problem should be addressed simultaneously to align the RGB real object and CAD model exactly.

Conventional computer vision methods for estimating 6-DoF pose involve extracting and matching image features. Well-known image features include Scale-Invariant Feature Transform (SIFT) [2], Speeded Up Robust Features (SURF) [3], Oriented FAST and Rotated BRIEF (ORB) [4], and so on. Conventional feature-based 6-DoF pose estimation methods face limitations when the number of extracted features is small or when an object is placed with a complex background due to high sensitivity to data quality and outliers. Other 6-DoF methods include Iterative Closest Point (ICP) [5] and Perspective-n-Point (PnP) algorithm [6]. ICP estimates correspondences between different point clouds to find rotation and translation transformations, and PnP estimates object pose using the perspective projections between 3D points and corresponding 2D image points.

Recent advances in deep learning techniques and the availability of big data have overcome some limitations of traditional feature-based methods. In addition to RGB images, 3D data from depth cameras, such as RGB-D and point clouds, have been utilized. Several approaches have been developed to estimate the 6-DoF pose of objects using deep learning methods. PoseCNN [7] and Deep-6DPose [8] employ convolutional neural networks (CNNs) for regression-based pose estimation. PoseNet [9] and PPR-Net [10] use point clouds as input data for pose estimation, while DenseFusion [11] combines various data types to extract comprehensive information for accurate pose estimation. Furthermore, methods such as Dpod [12] and Rnnpose [13] utilize deep learning networks to predict object pose even when parts of the object are occluded. These advancements highlight the capability of deep learning networks to estimate object pose more accurately, even in the presence of various obstacles.

The 3D CAD model retrieval problem is to search for a CAD model from CAD datasets, which is the best matching with an input RGB object. Conventional CAD retrieval methods mostly address the search for matching CAD models from RGB or RGB-D images [14]. However, there are few studies of simultaneous searching for the best CAD model and estimating the 6-DoF pose of an input RGB image [15,16]. Park et al. [15] and Gümeli C. et al. [16] introduced frameworks similar to the proposed method. These methods detect an RGB object from an input image, estimate the 4-DoF or 6-DoF pose, and search the matching CAD model. Because each problem of 6-DoF pose estimation and CAD model retrieval from an input RGB is inherently not easy, simultaneous pose estimation and CAD model retrieval has not been commonly addressed.

In this study, we propose a combined method of simultaneous 6-DoF pose estimation and CAD model retrieval from a single RGB image. From the single RGB image, we detect a real object and estimate the rotational and translational transformation of the object from the reference CAD to the camera coordinate systems. Then, the best-matching CAD model is retrieved from a dataset by the proposed similarity measurement between RGB and rendered CAD images.

Our contributions are twofold, and the summary of the proposed methods is as follows. First, for 6-DoF pose estimation, we employ “PoseContrast: Class-Agnostic Object Viewpoint Estimation in the Wild with the Pose-Aware Contrastive Learning” (PoseContrast) [17] as the baseline model. PoseConstrast estimates the 3-DoF rotational pose of an object regardless of the object classes. In the original PoseContrast, the number of public datasets for pose estimation is relatively small compared to those used for other tasks [18]. Therefore, using PoseContrast as the baseline model is advantageous since it is based on contrastive learning. In the proposed pose estimation network, we add new modules to the baseline to generate attention weights, which are used to update the feature output from the network rather than using features extracted by the multilayer perceptron (MLP) of the baseline network.

Subsequently, several network modules are added to increase the attention weights of the important parts of the features. To do this, the probability variance of each 3-DoF rotational angle is estimated, and the attention weights to meaningful aspects of the features are refined. Then, the refined features are used as inputs for the MLP. Additionally, we enhance the network’s performance by utilizing a loss function based on quaternion.

Second, we propose a CAD model retrieval method to determine the best matching CAD model with the RGB object. The CAD model retrieval is based on deep features by “Large-Scale Image Retrieval with Attentive Deep Local Features” (DELF) [19], which has been introduced to find the 2D affine transformation between images. For the retrieval experiment, we select ten CAD candidates in each object class. Using the pose estimation result, all candidates are transformed to the camera coordinate system, and 2D projections are rendered. Then, image similarity between the candidates and the RGB object is measured to determine the best matching model.

Therefore, the image similarity between the rendered CAD model and the RGB object is directly related to the pose estimation result. The better the pose of the RGB object is estimated, the more the image similarity between the matching CAD model and the RGB object is increased. To evaluate this correlation, we propose a correlation measurement between the two performances of pose estimation and CAD retrieval. The experimental results show that there is a positive correlation between the two performances.

The architecture of the proposed method is shown in Figure 1 and it can be summarized as follows. First, an RGB object is detected and segmented from a single RGB image using DetectoRS [20]. Second, 3-DoF rotation transformation is estimated by the proposed pose estimation network. Third, by using Blender [21], several 2D rendered images of 3D CAD candidates of the detected object category are generated. Fourth, the best CAD model is retrieved by similarity matching between the rendered and the object images. Finally, the translation parameters of the RGB object are determined by aligning the CAD rendering with the RGB segmentation area.

In the experiment, we show that the enhanced pose estimation performance also enhances the performance of CAD model retrieval. The performance relationship between the pose estimation and CAD model retrieval is quantitatively analyzed by a correlation parameter. The structure of this paper is as follows. In Section 2, we briefly explain camera parameters related to this study. Section 3 describes all the steps of the proposed method. In Section 4, we show several quantitative and qualitative evaluation results. Section 5 presents the ablation study for the proposed pose estimation model. Finally, in Section 6, we conclude this paper.

2. Internal and External Pose Parameters

The 6-DoF pose estimation of a 3D object in an RGB camera image is to determine the external orientation and translation of the object from a reference coordinate system to the camera coordinate system. In addition, the projection of the transformed 3D object into the image plane is expressed by the internal camera parameters. If we know the 6-DoF pose of the object and the camera parameters, we can transform and project a shape-matching CAD model into the image space so that the projection exactly aligns with the object area.

To project a 3D CAD model into an RGB image space, it is essential to understand the projection of a 3D point into the image plane. Let $\mathit{P}(X_w,Y_w,Z_w)$ be a point in the 3D space (reference coordinate system), $\mathit{P}(X_C,Y_C,Z_C)$ the corresponding point in the camera coordinate system, and $\mathit{p}(x,y)$ the projected point in the image plane (pixel coordinate system). Point $\mathit{p}(x,y)$ is determined through the following Equations (1)–(3). We assume $f_x=f_y=f$ where $f$ represents the focal length of the camera.

$$\left[\begin{array}{c}{X}_C\\ Y_C\\ Z_C\end{array}\right]=\left[\begin{array}{cccc}{R}_1& R_2& R_3& t_x\\ R_4& R_5& R_6& t_y\\ R_7& R_8& R_9& t_z\end{array}\right]\left[\begin{array}{c}\begin{array}{c}{X}_W\\ Y_W\end{array}\\ \begin{array}{c}{Z}_W\\ 1\end{array}\end{array}\right]$$

(1)

$$\left[\begin{array}{c}u\\ v\\ w\end{array}\right]=\left[\begin{array}{c}\begin{array}{ccc}{f}_x& 0& c_x\end{array}\\ \begin{array}{ccc} 0& { f}_y& c_y\end{array}\\ \begin{array}{ccc}0& 0& 1\end{array}\end{array}\right]\left[\begin{array}{c}{X}_C\\ Y_C\\ Z_C\end{array}\right]$$

(2)

$$\mathit{p}\left(x,y\right)=\left[\begin{array}{c}x\\ y\end{array}\right]=\left[\begin{array}{c}{\displaystyle \raisebox{1ex}{$u$}\!\left/ \!\raisebox{-1ex}{$w$}\right.}\\ {\displaystyle \raisebox{1ex}{$v$}\!\left/ \!\raisebox{-1ex}{$w$}\right.}\end{array}\right]$$

(3)

In this paper, the 3-DoF rotation parameters are represented by the Euler angles, as shown in Figure 2. The other 3-DoF is related to the translation between the reference to the camera coordinate systems. Among the 3-DoF translation parameters, the translation along the z-axis affects the size of the object in the image plane, as the size is affected by both the translation along the z-axis and the camera’s focal length ($f$). The camera translations along the x and y axes, $t_x$ and $t_y$, do not affect the object’s size. However, estimation of z-axis translation $t_z$ has inherent ambiguity because the object size is also affected by the focal length. Therefore, to estimate the 3-DoF parameters associated with the translation, both $t_x, t_y$ and the two scale factors ($t_z, f$) should be estimated.

3. Proposed Methods

3.1. Dataset for Reference CAD Models

Pascal3D+ [22] has been commonly used in many 6-DoF pose estimation studies. This study also uses Pascal3D+ to train the proposed pose estimation network. In addition, we use Pascal3D+, Pix3D [23], and ObjectNet3D [24] as the test datasets. The Pascal3D+ dataset consists of 12 categories, including airplanes, bicycles, chairs, and automobiles, and consists of approximately 36,000 images. Among these, 28,100 images are used for training, and 2113 images are used for testing. The Pix3D dataset contains 9 indoor object categories, generally categorized for IKEA household items, with about 10,000 images. Among them, the chair category contains the largest portion, with around 4000 images. ObjectNet3D includes 100 categories and consists of approximately 90,000 images. Unlike the previous two datasets, ObjectNet3D encompasses categories for relatively small objects such as keys, pencils, and pens.

For CAD model retrieval, the 3D CAD dataset from ShapeNetCore is used. It is a subset of the ShapeNet [12] database. ShapeNetCore provides 3D CAD models across 55 upper categories for indoor and outdoor objects, with various subcategories. In this study, 10 upper categories (bag, bed, bottle, bench, car, chair, clock, motorcycle, stove, and table) and 10 subcategories within each category are selected, resulting in a total of 100 CAD models. The selection criteria for these categories are based on the following principles, and the 100 CAD models are shown in Figure 3.

• The upper category should have a matching category in the Microsoft COCO dataset [25] because the object detection network is trained on this dataset.
• The appearance of the CAD models in a subcategory should be different enough to facilitate similarity measurement.
• The appearance and shape of the CAD models across all categories should be general so that they appear in a sufficient amount of RGB image data.

3.2. Dataset for RGB Images

The RGB images were collected by the authors from Google Images and Bing Search Images, along with the 100 reference CAD models. The criteria for the collection of RGB images are summarized below. A total of 20 RGB images were collected for each model, resulting in a total of 2000 RGB images.

• To ensure usability in object detection and pose estimation networks, the image size is constrained between 224 × 224 and 1600 × 1600 pixels.
• Accepted image formats include jpg, jpeg, and png.
• For similarity measurement between the RGB object and the CAD rendering, it is necessary that the RGB object is minimally occluded.

Given a single RGB image, object detection and segmentation are performed using DetectoRS [20]. If there is no detected object in the image, it is excluded from the dataset. Furthermore, when multiple types of objects are detected within a single RGB image, it is used when any object in the image is in the same category as one in the CAD dataset. When the objects in an image are different types from the CAD dataset, it is excluded. For images with multiple instances of the same object, all instances are retained. Following this inclusion and exclusion process, a total of 1562 RGB images are used for CAD model retrieval. Figure 4 shows examples of RGB images collected for a clock CAD model.

3.3. Proposed Pose Estimation Network

As mentioned in an earlier section, the proposed 6-DOF pose estimation is performed by a deep learning network whose baseline is PoseContrast [17]. PoseContrast estimates the rotational pose of an object from a single RGB image. This network uses ResNet50 [26] as the encoder backbone and a multilayer perceptron (MLP) as the predictor. Here, the 3-DoF rotational pose is represented by three Euler angles from the reference CAD coordinate system to the camera coordinate system. The three Euler angles are defined as Azimuth, Elevation, and Inplane rotation. In PoseContrast, each rotation angle is predicted into 15° sized bins with an offset value ranging from −1 to +1. The final rotation angle is then calculated as in Equation (4).

$$angle=15\times (bin index+offset)$$

(4)

The loss functions employed in the bin prediction and offset prediction are Cross Entropy Loss [27] (Equation (5)) and Smooth L1 Loss [28] (Equation (6)), respectively. In Equation (5), $\mathit{p}$ and $\mathit{q}$ can be considered as bin vectors of MLP prediction and GT, respectively. Each bin vector consists of 24 elements of probability to each 15° bin. In Equation (6), $x$ is the MLP prediction to the angle offset. In PoseContrast, the same loss functions are applied to predict all three rotation angles.

$$Cross Entropy Loss\left(\mathit{p},\mathit{q}\right)=-\sum _{i=1}^N{p}_i\log \left(q_i\right)$$

(5)

$${Smooth}_{L1}\left(x\right)=\left\{\begin{array}{c} 0.5x^2, if |x|<1\hfill \\ \left|x\right|-0.5, otherwise\hfill \end{array}\right.$$

(6)

In this paper, we modify the original PoseContrast network by adding one sub-network (VUF) and two attention modules (GCE and CGF) to the original network. The structure of the proposed network is illustrated in Figure 5. First, we introduce a sub-network that generates variational uncertainty features from the output of ResNet50. The sub-network generates uncertainty features and they are used to generate the uncertainty weight to the original ResNet50 features. Thus, we call this sub-network the VUF (Variational Uncertainty Feature) network.

Similar to the original PoseContrast, VUF also predicts Azimuth, Elevation, and Inplane rotation using two MLP networks, as shown in Figure 5. The first MLP provides the probability to each angle bin, and the highest bin index is determined based on the probability values. The output of the first MLP is used to compute the uncertainty weight of each angle bin, as shown in the figure. To compute the uncertainty weight, $V_{diff,i}$ is first calculated as in Equation (7) to represent the degree of difference between the highest probability bin index $i_{max}$, and another bin index $i$. To provide smooth weight values to the next process, the bin angle difference is represented in the logarithm scale as in Equation (7). Once $V_{diff,i}$ in each angle bin is obtained, $V_{uncer}$, variance uncertainty, is calculated using Equation (8). Here, $\mathrm{P}\mathrm{r}\left(i\right)$ is the probability of bin index i. The final weight, $w_{bin}$, for each angle bin is calculated as in Equation (9). This process is applied respectively to Azimuth, Elevation, and Inplane rotation.

$$V_{diff,i}=\log \{{\left(i-i_{max}\right)}^2+1\}$$

(7)

$$V_{uncer}=\sum _{i=0}^j\left| V_{diff,i}\ast \mathrm{P}\mathrm{r}(i) \right|$$

(8)

where, $j=\left\{\begin{array}{c}23, \mathrm{for} Azimuth, Inplane rotation\\ 11, \mathrm{for} Elevation\hfill \end{array}\right.$.

$$w_{bin}=\ln \left(V_{uncer}+e\right)-1,$$

(9)

where, $bin\in \left\{Azimuth,Elevation,Inplanerotation\right\}$.

Weight obtained in Equation (9) is multiplied to the original features trained for angle prediction. In each angle prediction, the features consist of 200 channels. The weighted 200 features are then concatenated to form an integrated feature module. Through the second MLP, the integrated feature module is then refined and expanded to a feature map of size 2048 × 7 × 7, which is the same size as the original feature map from the ResNet50.

Our second modification consists of two modules, the Guided Cost Volume Excitation (GCE [29]) module and the Context and Geometry Fusion (CGF [30]) module, as shown in Figure 5. The main purpose of adding two modules is to fuse the feature map from the original ResNet50 and the features from the VUF sub-network. Through the two modules, the features from ResNet50 are excited to predict the angular pose of an object. Following the GCE module, a Convolutional Block Attention Module (CBAM [31]) is applied to extract key features from the excited features and the original features. CBAM provides more attention to the key features; consequently, it enhances the pose estimation performance. These features are then processed through the CGF module, which generates a spatial attention weight. Finally, CBAM is applied again to the feature, and then it is used as input for the predictor.

In addition to the new sub-network and module employment, we enhance the proposed network by introducing a new loss function. Conventionally, the cross entropy loss function in Equation (5) computes the uniform loss between the prediction and the ground truth. For instance, when predicting an Azimuth angle with a GT angle of 0°, the prediction of 15° differs significantly from 225° in terms of angle difference. However, the cross entropy calculates both losses as the same magnitude. To address this issue, we propose a quaternion-based loss function. Quaternion is a generalized form of a complex number comprising three real components and one imaginary component, as shown in Equation (10). To employ the quaternion-based loss function, we convert the rotation angles from PoseContrast and the ground truth (GT) into quaternions using Equations (11)–(14). Here, Azimuth is denoted as $\alpha$, Elevation as $e,$ and Inplane rotation as $i$. Equation (15) shows the quaternion loss function, with the parameter α empirically set to 3. Given the importance of accurately predicting the bin index, we formulate the final loss function by integrating the quaternion loss into the existing loss function.

$$Q(a,e,i)=w+xi+yj+zk$$

(10)

$$x=\cos \left({\displaystyle \frac{a}{2}}\right)·\cos \left({\displaystyle \frac{e}{2}}\right)·\sin \left({\displaystyle \frac{i}{2}}\right)-\sin \left({\displaystyle \frac{a}{2}}\right)·\sin \left({\displaystyle \frac{e}{2}}\right)·\cos \left({\displaystyle \frac{i}{2}}\right)$$

(11)

$$y=\cos \left({\displaystyle \frac{a}{2}}\right)·\sin \left({\displaystyle \frac{e}{2}}\right)·\cos \left({\displaystyle \frac{i}{2}}\right)+\sin \left({\displaystyle \frac{a}{2}}\right)·\cos \left({\displaystyle \frac{e}{2}}\right)·\sin \left({\displaystyle \frac{i}{2}}\right)$$

(12)

$$z=\sin \left({\displaystyle \frac{a}{2}}\right)·\cos \left({\displaystyle \frac{e}{2}}\right)·\cos \left({\displaystyle \frac{i}{2}}\right)-\cos \left({\displaystyle \frac{a}{2}}\right)·\sin \left({\displaystyle \frac{e}{2}}\right)·\sin \left({\displaystyle \frac{i}{2}}\right)$$

(13)

$$w=\cos \left({\displaystyle \frac{a}{2}}\right)·\cos \left({\displaystyle \frac{e}{2}}\right)·\cos \left({\displaystyle \frac{i}{2}}\right)+\sin \left({\displaystyle \frac{a}{2}}\right)·\sin \left({\displaystyle \frac{e}{2}}\right)·\sin \left({\displaystyle \frac{i}{2}}\right)$$

(14)

$$Quaternion Loss\left(Predected,GT\right)=\alpha \times \left\{1-{\left(Q\left(Predicted\right(a,e,i\left)\right)·Q\left(GT\right(a,e,i\left)\right)\right)}^2\right\}$$

(15)

The quaternion-based loss function solves the issue of uniform loss between predicted and GT bin indices in the cross entropy loss function. For example, when the GT bin index is 0 and the predicted bin index is 1 (15°) or 15 (225°), the network is trained with different loss values. On the contrary, when the predicted bin is 23 (335°) or 1 (15°), the same loss value is obtained because they are equidistant from the GT. Especially, when the predicted bin index is 12, exactly opposite angle to the GT bin, the loss value becomes maximum, as shown in Figure 6. By employing these non-uniform loss values, the proposed quaternion-based loss function enhances the performance of angular pose estimation with the proposed network.

3.4. Similarity Measurement Between RGB Image and CAD Rendering

Using the modified PoseContrast network, the rotational pose of a real object in a single RGB image has been predicted. Using the predicted rotational pose, we can transform a CAD model from the reference to the camera coordinate system. However, to apply the proposed 6-DoF pose estimation technique to AR or XR applications, it is necessary to retrieve and transform the CAD model whose category and shape match exactly with the RGB object. However, because the PoseContrast network is a class-agnostic pose estimator, we need to find the exact subcategory of the RGB object before projecting the CAD model to the image space.

Until the previous section, we have known that the upper category of an RGB object from the DetectRS network. However, there are various shapes of CAD models in the same upper category from the dataset. Examples of the bench category in the ShapeNetCore are shown in Figure 7. On the left side of the figure is a bench RGB image and the right side shows bench CAD models. On the right side, 10 different shapes of the bench category are shown, and in this study, we call each different shape a subcategory.

This section describes how to determine the best matching subcategory from 10 candidate models. To find a matching subcategory, we employ a deep feature matching network, DELF (Large-Scale Image Retrieval with Attentive Deep Local Features) [19]. As shown in Figure 7, the rotational pose estimation result is used to transform candidate CAD models to the camera coordinate system and to render them into the 2D image plane. Therefore, using DELF, we propose to measure 2D image similarity between the RGB object and the rendered CAD images and to determine the best similar rendering as the final matching model.

Details of similarity measurement are as follows. First, the background of the RGB object is removed using the DetectRS segmentation result. Second, we obtain 10 rendered images of subcategories. Third, between each pair of the RGB object and rendered image, we derive a 2D affine transformation matrix using DELF. From the DELF feature detection and RANSAC outlier removal, we obtain the affine transformation matrix. The motivation for obtaining a 2D affine matrix is that the ideal image space transformation between the RGB object and the rendered CAD model is only unknown scaling. The ideal rotational pose of the matching CAD model results in only scale difference with unknown z-axis translation and focal length.

The 2D affine transformation matrix $A$ in Equation (16) consists of six elements representing 2D rotation, scaling, and skewing transformations, as well as translation. In this equation, two translation elements are not needed because only rotational pose is applied to the CAD rendering. Thus, only the front 2 × 2 matrix in $A$ is considered, and it is defined as matrix $A_{sim}$, as in Equation (17). The four elements of $A_{sim}$ consist of multiplications of scale (S), skew ($h_x$,$h_x$) and 2D rotation ($\theta$).

$$A=\left[\begin{array}{c}\begin{array}{ccc}{a}_1& a_2& t_x\end{array}\\ \begin{array}{ccc}{a}_3& a_4& t_y\end{array}\end{array}\right]$$

(16)

$$A_{sim}=\left[\begin{array}{c}{a}_1 a_2\\ a_3 a_4 \end{array}\right]=\left[\begin{array}{cc}{S}_x\cos \theta & S{h}_x\sin \theta \\ -S{h}_y\sin \theta & S_y\cos \theta \end{array}\right]$$

(17)

The first image similarity measurement is based on the 2D affine transformation, called SM-AT. The size of the 2D rendering image is fixed at 600 × 600 pixels, whereas the size of RGB object images varies significantly in the collected dataset. Therefore, even if the rotational poses of the RGB object and CAD rendering are identical, there is still an image scale difference. Thus, practically, the 2D rotation transformation between RGB and CAD rendering should not be an identity matrix but rather multiplied by two scale factors, as shown in Equation (18). As the rotational pose between the RGB object and the CAD rendering becomes similar, matrix $A_{sim}$ should have only scale elements in Equation (18). In the ideal case, if the rotational poses in both RGB and CAD rendering images are the same, $\cos \theta =1$ and $S_x=S_y=S$. Here, scale S can be obtained from Equation (19). To normalize $A_{sim}$, we derive the scale-normalized affine matrix $A_{normal}$ as in Equation (20).

$$R=\left[\begin{array}{cc}{S}_x& 0\\ 0& S_y\end{array}\right]$$

(18)

$$S={\displaystyle \frac{\sqrt{{a_1}^2+{a_4}^2}}{2}}={\displaystyle \frac{\sqrt{{{(S}_x\cos \theta )}^2+({S_y\cos \theta )}^2}}{2}}$$

(19)

$$A_{normal}=\left[\begin{array}{cc}{a}_1/S& a_2/S\\ a_3/S& a_4/S\end{array}\right]$$

(20)

From the scale-normalized matrix $A_{normal}$, we define $e_{rot}$ and $s_{AT}$ using Equations (22) and (23). Finally, the rank for CAD model retrieval is determined by the $Ran{k}_{AT}$ value as in Equation (24), where a higher value means a higher rank. Additionally, by defining ${Rank}_{AT}$ as in Equation (24), we consider not only the error for the rotation angle but also the affine consistency for retrieving the matching CAD model. In Equation (24), $N_{inlier}$ is the number inlier in deriving the affine transformation matrix $A$.

$$E=A_{normal}-\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$$

(21)

$$e_{rot}=\sum \left|E_i\right|$$

(22)

$$s_{AT}=\left(1-2\ast \left({\displaystyle \frac{1}{1+e^{-e_{rot}}}}-{\displaystyle \frac{1}{2}}\right)\right)$$

(23)

$$Ran{k}_{AT}=s_{AT}\ast \left(N_{inlier}\right)$$

(24)

As the second measurement, we only use the number of feature-matching inliers (SM-NI). With this measurement, the CAD retrieval ranking is determined from the highest number of inliers to the lowest number.

Using these two measurements, the similarity between every pair of RGB image and CAD rendering image within a detected upper category is evaluated. For the evaluation of the two measurements, we have manually generated the annotations of CAD subcategories for all 2000 RGB images. In the experiments, we show quantitative analysis of CAD model retrieval based on SM-AT and SM-NI with the ground truth.

3.5. Translation and Focal Length Estimation

After retrieving the matching CAD model, it is now possible to estimate the scale factor ($f, t_z)$ by projecting the CAD model to the RGB image space. In the pin-hole camera model, the projection size of the CAD model is affected by both $f$ and $t_z$. However, solving both parameters together is an ambiguous problem. Thus, one parameter must be set as an arbitrary value while the remaining parameter is estimated. In this study, we define the limited range of focal length and z-axis translation because there is no GT focal length parameter in the RGB images from the Internet. Using the limited ranges and an iterative algorithm, we estimate both parameters.

In this iterative algorithm, $t_x$ and $t_y$ are set to 0 because we assume the center of the bounding box corresponds to the center of the object. By the pin-hole camera model, a 3D vertex point $\mathit{P}(X_v, Y_v,Z_v)$ on the 3D CAD model is projected to a 2D image point $\mathit{p}(x_v, y_v)$, where $v\in \{1,2,\dots ,n\}$ and $n$ is the number of vertices in the 3D CAD model. The scale parameters are ranged as follows: $f\in \{50,70,90,\dots ,1460,1480\}$ and $t_z\in \{0.05,0.1,0.15,\dots ,1.9,1.95\}$. Principal points $c_x$ and $c_y$ are decided to be at the center of the cropped image. By applying these values from Equations (1)–(3), the 3D vertices are projected into the image plane. A new bounding box is created using the 2D projection points such as $x\in \{\underset{x}{\min }\mathit{p}\left(x_v,y_v\right),\underset{x}{\max }\mathit{p}\left(x_v,y_v\right)\}$ and $y\in \{\underset{y}{\min }\mathit{p}\left(x_v,y_v\right),\underset{y}{\max }\mathit{p}\left(x_v,y_v\right)\}$. Then, the Intersection Over Union (IoU) between this newly created bounding box and the existing RGB object bounding box is calculated. The values of $f$ and $t_z$ that maximize the IoU are selected as the final estimation. An example of CAD vertex projection and bounding boxes is shown in Figure 8.

After estimating the scale factor, $t_x$ and $t_y$ must be recalculated from the original RGB image using Equations (1)–(3). Principal points $c_x$ and $c_y$ represent the center points of the original image. Scale factors are estimated from the cropped image without changing the pixel size; thus, these values remain valid in the original image. Since the center of the 3D CAD model is at the origin of the reference coordinate system, $t_x$ and $t_y$ can be determined by Equations (1)–(3). By setting $\mathit{P}\left(X_v,Y_v,Z_v\right)=(0,0,0)$ and projecting it to the image space, we have the center of the 2D bounding box $\mathit{p}\left(x_v,y_v\right)$ in the cropped RGB images. In the original image space, the center of the projected bounding box becomes $\mathit{p}\left(x_v+x_{off},y_v+y_{off}\right)$, where $\left(x_{off},y_{off}\right)$ is the upper-left corner of the bounding box in the original image. Then external translation $t_x$ and $t_y$ can be calculated as:

$$t_x=t_z{\displaystyle \frac{x_v+x_{off}}{f}} \mathrm{and} t_y=t_z{\displaystyle \frac{y_v+y_{off}}{f}}.$$

(25)

4. Experiment and Evaluation

4.1. Experimental Setup

We trained the proposed pose estimation network using Adam optimizer with a batch size of 16 and an initial learning rate of $8\times {10}^{-5}$. We trained for 15 epochs on an Intel Xeon CPU with a single NVIDIA A5000 GPU equipped with 24 GB memory (Santa Clara, CA, USA).

4.2. Evaluation of Pose Estimation

We evaluated the pose estimation performance through two quantitative metrics, Acc30 and Median Error, as described in [32,33]. Acc30 evaluates the proportion of instances where the angular difference between the predicted and GT angles is within 30°. It is measured in percentage (%). Equations (26)–(31) show the conversion process of Euler angles into rotation transformation matrices, where $a,e,$ and $i$ denote Azimuth, Elevation, and Inplane rotation, respectively. Median Error quantifies the median of the differences in rotational angles between the predicted and ground truth values, measured in degrees (°).

$$R_{x,y,z}=\left[\begin{array}{ccc}{r}_{11}& r_{12}& \sin \left(e^{\prime }\right)\sin \left(a^{\prime }\right)\\ r_{21}& r_{22}& \sin \left(e^{\prime }\right)\cos \left(a^{\prime }\right)\\ \sin \left(i^{\prime }\right)\sin \left(e^{\prime }\right)& -\cos \left(i^{\prime }\right)\sin \left(e^{\prime }\right)& \cos \left(e^{\prime }\right)\end{array}\right]$$

(26)

$$r_{11}=\cos \left(i^{\prime }\right)\cos \left(a^{\prime }\right)-\sin \left(i^{\prime }\right)\cos \left(e^{\prime }\right)\sin \left(a^{\prime }\right)$$

(27)

$$r_{12}=\sin \left(i^{\prime }\right)\cos \left(a^{\prime }\right)+\cos \left(i^{\prime }\right)\cos \left(e^{\prime }\right)\sin \left(a^{\prime }\right)$$

(28)

$$r_{21}=-\cos \left(i^{\prime }\right)\sin \left(a^{\prime }\right)-\sin \left(i^{\prime }\right)\cos \left(e^{\prime }\right)\cos \left(a^{\prime }\right)$$

(29)

$$r_{22}=-\sin \left(i^{\prime }\right)\sin \left(a^{\prime }\right)+\cos \left(i^{\prime }\right)\cos \left(e^{\prime }\right)\mathrm{c}\mathrm{o}\mathrm{s}\left(a^{\prime }\right)$$

(30)

$${(a}^{\prime },e^{\prime },i^{\prime })=(a,e,i)\times {\displaystyle \frac{\pi }{180}}$$

(31)

The performance was compared with PoseContrast using three test datasets, Pix3D, Pascal3D+, and ObjectNet3D, as shown in

Table 1. Pose estimation performance using Pascal3D+ (number of images is in parenthesis and bold font indicates the best performance in each column).

	Aero-plane (275)	Bicycle (118)	Boat (232)	Bottle (251)	Bus (154)	Car (308)	Chair (244)	Dining Table (21)	Motorbike (136)	Sofa (39)	Train (113)	Tv Monitor (222)	Average (2113)
Acc30
Grabner et al. [17]	80	82	57	90	97	94	72	67	90	80	82	85	81
StarMap [17]	82	86	50	92	97	92	79	62	88	92	77	83	82
3DPoseLight [17]	80	82	58	93	96	92	77	57	88	82	80	79	80
PoseContrast	81	82	59	92	96	93	84	52	88	85	81	85	82
Proposed	87	81	65	92	97	94	83	62	90	82	81	86	83
Median Error
Grabner et al. [17]	10.9	12.2	23.4	9.3	3.4	5.2	15.9	16.2	12.2	11.6	6.3	11.2	11.5
StarMap [17]	10.1	14.5	30.3	9.1	3.1	6.5	11.0	23.7	14.1	11.1	7.4	13.0	12.8
PoseContrast	11.25	15.63	22.39	7.68	2.68	4.67	10.7	20.88	11.48	10.76	5.64	11.19	11.25
Proposed	11.7	14.71	18.47	7.84	2.84	4.83	9.98	11.71	11.98	10.75	6.18	10.88	10.16

,

Table 2. Pose estimation performance using Pix3D (number of images is in parenthesis and bold font indicates the best performance in each column).

	Tool (46)	Misc (61)	Bookcase (130)	Wardrobe (166)	Desk (297)	Bed (394)	Table (738)	Sofa (1092)	Chair (2894)	Average (5818)
Acc30
3DPoseLight [17]	9	10	62	57	66	58	40	94	50	58
PoseContrast	11	25	76	58	78	71	53	97	86	62
Proposed	11	21	85	80	76	68	53	96	85	64
Median Error
PoseContrast	128.75	58.36	17.29	22.76	12.49	13.27	23.79	14.49	11.7	33.66
Proposed	107.18	75.99	12.44	16.11	12.76	13.06	25.4	13.13	11.66	31.97

and

Table 3. Pose estimation performance using ObjectNet3D.

	Base Classes	Novel Classes	All Samples
Acc30
PoseContrast	46	45	50
Proposed	47	48	52
Median Error
PoseContrast	51.32	56.92	29.43
Proposed	50.14	50.16	27.57

. We trained our proposed model using the Pascal3D+ dataset. We tested the model on the Pix3D and ObjectNet3D datasets to evaluate generalization performance. Table 1 presents the quantitative results on Pascal3D+, showing that the proposed method outperforms the baseline, PoseContrast. Moreover, as shown in Table 2 and Table 3, our model performs better than the PoseContrast in generalization evaluations.

Employing two evaluation metrics enables complementary analysis. Evaluation only using Acc30 might lead to misleading conclusions; for instance, an error of 31° would be classified as incorrect despite being very close to the reference angle. Consequently, incorporating Median Error provides the distribution of errors, which is a more comprehensive evaluation for performance comparison.

4.3. Evaluation of CAD Model Retrieval

Table 4. Number of RGB images that retrieve the matching CAD model as the specific rank.

	Rank 1	Rank 2	Rank 3	Rank 4	Rank 5
SM-AT	720	250	156	117	87
SM-NI	741	258	179	101	91

presents the evaluation results of CAD model retrieval using 1562 RGB images. Originally, 2000 RGB images were used for CAD model retrieval; however, 438 images were rejected from the test by the rules described in Section 3.2. The matching CAD model with the RGB object is retrieved based on the image similarity between the RGB object and the CAD rendering images. Given an RGB object, renderings of the 10 subcategories of CAD models are ranked from 1st to 10th. In Table 4, the number of RGB images that find the matching CAD models among the top five rankings is shown.

For the SM-AT method, the rate of retrieving the matching CAD model within the top three rankings is 72.087%, and within the top five rankings, it is 85.147%. For the SM-NI method, these rates are 75.416% and 87.708%, respectively. These results show that both similarity measurements are useful for evaluating the similarity between the RGB image and the CAD image domains.

4.4. Correlation Analysis Between Pose Estimation and CAD Retrieval

This study addresses not only the 6-DoF pose estimation of RGB objects but also CAD model retrieval for the purpose of AR and XR applications. Thus, it is expected that the performance enhancement of the proposed pose estimation positively affects the performance of the CAD retrieval. In this section, we investigate qualitative and quantitative analysis to verify the positive relationship between the performance of the two tasks.

First, we evaluate the performance relationship qualitatively. In this evaluation, we visually compare 2D rendered images of the matching CAD model with the RGB object. If the predicted pose perfectly matches with the RGB object, then the pose appearance should be very similar. The visual comparison is performed with two conventional methods: one is PoseContrast, and the other is PosefromShape. In each comparison, we generate a side-by-side rendering image pair from the conventional and the proposed methods, as shown in Figure 9. For each RGB test image, the side-by-side rendering image pair is concatenated, and their poses are visually compared. Based on the comparison, we classify the results into four cases:

• Better: Rendering images from the proposed network yields a better pose appearance than the conventional method. An example is shown in Figure 9a and Figure 10a.
• SS (Same or Similar): Both CAD rendering images show the same or similar pose appearance. An example is shown in Figure 9b and Figure 10b.
• Worse: Rendering images from the conventional method is better than the proposed network. An example is shown in Figure 9c and Figure 10c.
• Bad: Both pose estimation results are significantly different from the pose of the RGB object. An example is shown in Figure 9d and Figure 10d.

We consider that the predicted rotational pose is significantly different from the RGB object if the pose difference is more than about 45°. For the performance correlation analysis, the ‘Bad’ case is excluded. In Figure 9, the left image is the estimation result from PoseContrast, the center image is from the proposed method, and the right image is RGB objects.

Table 5. Results of the qualitative evaluation compared to PoseContrast (number of detected images is in parenthesis).

	Better	SS	Worse
Bag (144)	47	33	37
Bed (110)	22	13	18
Bench (146)	47	57	23
Bottle (184)	32	79	24
Car (188)	18	153	14
Chair (199)	50	106	26
Clock (173)	56	43	27
Motorbike (170)	45	87	30
Stove (176)	66	52	34
Table (72)	10	56	6

and

Table 6. Results of the qualitative evaluation compared to PosefromShape (number of detected images is in parenthesis).

	Better	SS	Worse
Bag (144)	68	25	37
Bed (110)	15	36	46
Bench (146)	87	20	31
Bottle (184)	109	3	8
Car (188)	75	89	6
Chair (199)	95	73	16
Clock (173)	51	19	26
Motorbike (170)	70	81	7
Stove (176)	116	31	14
Table (72)	47	21	4

present the results of the qualitative evaluation. Each evaluation is denoted as ‘Better’, ‘SS’, and ‘Worse’, respectively. Table 5 shows the evaluation with respect to PoseContast and Table 6 is to PosefromShape. The greater number of ‘Better’ evaluations is obtained in Table 6 because the performance enhancement compared to PosefromShape is much higher than that of PoseContrast. For comparative analysis with PosefromShape, we evaluate this method using the same dataset as our proposed method. Unlike PoseContrast, which predicts the pose from a single RGB image and renders the CAD model using the same predicted angle for all instances, PosefromShape predicts individual values for each pair of an RGB image and a CAD model used as input. Therefore, for our evaluation, we use the rendering of the ground truth CAD model to estimate the similarity measurement.

Using the visual evaluation of the pose estimation, we performed an additional quantitative analysis by introducing a correlation score between the pose estimation and the CAD retrieval. In Table 4, we have shown the rank numbers of the matching CAD model using the two image similarity measurements.

Let $r_i$ be the rank number of the matching CAD model with the i-th test RGB images, $r_{i,Proposed}$ the rank number results from the proposed method, and $r_{i,Conventional}$ the rank number results from the conventional methods, PoseContrast or PosefromShape. We use Equation (32) to get the correlation score $s_{i,corr}$ of the i-th test image. This simple equation gives a positive reward score if the image similarity rank from the proposed method is higher (rank number becomes small) than that from PoseContrast or PosefromShape. On the contrary, this equation gives a negative score if the image similarity rank from the proposed method is lower (rank number becomes large).

$$s_{i,corr}={r_{i,Conventional}-r}_{i,Proposed}$$

(32)

For example, if the matching CAD model is ranked 1st using the proposed network and 3rd using a conventional method, the score is +2. On the contrary, if the conventional method yields a better rank, we give a negative score. If both results yield the same rank, the score is 0. The proposed correlation score is based on the rank difference between the two networks rather than the absolute rank obtained from each network. After adding all scores from all RGB images, we get the average by dividing by the number of all RGB images.

Table 7. Correlation score per sample between pose performance and similarity measurement. Pose performance comparison is with respect to PoseContrast.

Similarity Measure	Better	SS	Worse
SM-AT	0.519	0.0	−0.364
SM-NI	0.338	0.110	−0.226

and

Table 8. Correlation score per sample between pose performance and similarity measurement. Pose performance comparison is with respect to PosefromShape.

Similarity Measure	Better	SS	Worse
SM-AT	1.138	0.271	−0.349
SM-NI	1.075	0.261	−0.097

show the correlation score of the proposed method with two image similarity measurements. In both measurements, a positive score is obtained in the ’Better’ case, while a negative score is obtained in the ’Worse’ case. Additionally, the ’SS’ case yields a score close to zero. This quantitative analysis shows that the performance improvement from the proposed pose estimation method also improves the performance of CAD model retrieval. In addition, the pose estimation failures also negatively affect the CAD model retrieval. As shown in Table 8, correlation scores with respect to PosefromShape, the positive correlation scores in the ‘Better’ case are higher than those in Table 7. This means that the correlation score evaluation with respect to PosefromShape is much higher than PoseContrast.

5. Ablation Study of the Proposed Pose Estimation Model

In this section, we perform ablation experiments on the Pascal3D dataset to validate the effectiveness of the proposed model. Unlike the baseline model, the proposed model includes three key modules and is trained using an additional quaternion loss function.

Table 9. Configurations of the ablation models. The baseline model is PoseContrast.

	VUF	GCE	CGF	Q loss
PoseContrast (baseline)
VUF-module	$\surd$			$\surd$
VUF+GCE	$\surd$	$\surd$		$\surd$
VUF-full	$\surd$	$\surd$	$\surd$
Proposed	$\surd$	$\surd$	$\surd$	$\surd$

shows how we have conducted ablation studies for each of these modules. All model variants were trained under the same conditions outlined in the implementation details.

Table 10. Comparison of the quantitative results of the ablation models. The baseline model is PoseContrast. The best performance is highlighted in bold.

	Aero- plane (275)	Bicycle (118)	Boat (232)	Bottle (251)	Bus (154)	Car (308)	Chair (244)	Dining Table (21)	Motorbike (136)	Sofa (39)	Train (113)	Tv Monitor (222)	Average (2113)
Acc30
PoseContrast	81	82	59	92	96	93	84	52	88	85	81	85	82
VUF-module	82	81	62	92	95	93	82	62	89	87	79	80	82
VUF + GCE	85	81	60	93	97	93	86	57	89	90	81	85	83
VUF-full	82	84	64	92	99	92	87	57	89	92	82	85	84
Proposed	87	81	65	92	97	94	83	62	90	82	81	86	83
Median Error
PoseContrast	11.25	15.63	22.39	7.68	2.68	4.67	10.7	20.88	11.48	10.76	5.64	11.19	11.25
VUF-module	12.66	14.31	21.79	9.05	3.56	5.61	10.31	14.02	12.92	10.99	7.13	12.5	11.24
VUF + GCE	10.62	14	20.18	8.17	3.01	4.85	10.41	14.27	12.13	11.64	5.58	12.29	10.6
VUF-full	10.47	15.67	18.96	7.5	3.08	4.86	9.85	13.51	11.95	13.05	5.56	11.85	10.53
Proposed	11.7	14.71	18.47	7.84	2.84	4.83	9.98	11.71	11.98	10.75	6.18	10.88	10.16

presents the results for each modified model. In Table 9, VUF refers to the variance uncertainty model in our proposed approach, GCE denotes the Guided Cost Volume Excitation module, CGF refers to Context and Geometry Fusion, and Q loss specifies whether a quaternion loss was employed during training.

As shown in Table 10, adding the proposed modules to the baseline progressively improves performance. Specifically, the VUF module alone shows little improvement over the baseline. The VUF module leverages variance uncertainty for each predicted angle to enhance performance; however, it incurs feature loss during the module’s internal attention process, so an additional auxiliary module is required to boost performance further. By supplementing VUF with the GCE module (VUF+GCE), the network compensates for this loss and improves performance, as observed in Table 10. Next, to assess the effect of the quaternion loss function, we compare the proposed model to a VUF-full ablation variant that excludes the quaternion loss. The results show that the addition of the quaternion loss during training enhances the performance of the proposed method.

6. Conclusions

In this study, we propose 6-DoF pose estimation and CAD model retrieval methods to align the projection of the CAD model with an object in an RGB image. The pose estimation is performed by using a deep learning network which is based on the PoseContrast network. To enhance the performance of the class-agnostic PoseContrast network, we add two sub-networks to emphasize the feature weight from the original ResNet50. Variance uncertainty weight and attention sub-network modules enhance the performance of pose estimation. After the pose estimation of the RGB object, we retrieve the matching CAD model from a CAD dataset. The quantitative analysis of the correlation between the 6-DoF pose estimation and the CAD model retrieval has been conducted. We evaluate the performance of pose estimation on Pascal3D+, Pix3D, and ObjectNet3D datasets. Through the experiments, we show that the proposed network reduces median errors in calculating the rotation angle of a 3D object in the RGB image. Furthermore, it is shown that the matching CAD model can be effectively retrieved using two similarity measurements (SM-AT and SM-NI). In addition, we find that there is a positive correlation between the pose estimation and CAD model retrieval performances.

In our experiments, there is a limitation in the training stage of the proposed network. The Pascal3D+ dataset, used for training, exhibits a bias in the distribution of Euler angles of the RGB objects. This bias may affect the network’s ability to estimate the pose of the objects. In future studies, we will focus on developing a new training dataset with evenly distributed training angles.