Depth Estimation of a Deformable Object via a Monocular Camera

Abstract

The depth estimation of the 3D deformable object has become increasingly crucial to various intelligent applications. In this paper, we propose a feature-based approach for accurate depth estimation of a deformable 3D object with a single camera, which reduces the problem of depth estimation to a pose estimation problem. The proposed method needs to reconstruct the target object at the very beginning. With the 3D reconstruction as an a priori model, only one monocular image is required afterwards to estimate the target object’s depth accurately, regardless of pose changes or deformability of the object. Experiments are taken on an NAO robot and a human to evaluate the depth estimation accuracy by the proposed method.

1. Introduction

In the case of human–robot cooperation, deep reinforcement learning (DRL) is used to train the robot to undertake the task. For an instance of the bolt screwing task, the human partner’s arms might be obstacles for the robot during the working process (Figure 1). In order to train the capability of avoiding obstacles for the robot through DRL method, one has to prepare a huge number of samples, which are usually hard to generate. In this case, it can be done by reconstructing the 3D imaging [1,2,3] of a human worker executing the task. The reconstruction sequence of the human’s arms can be used as moving obstacles to train the obstacle avoidance capability of the robot in a virtual environment (Figure 2).

In the scenarios above, a common prerequisite is the accurate pose information of a human or a robot. However, when an object is projected onto the camera plane, its depth information along the optical axis is lost, which possibly makes two actually far separated objects look close to each other [4]. This results in incorrect estimation of the pose without correct depth information. Although lasers are used to provide depth information, they are usually expensive. Low-cost equipment such as a Kinect are also prevailing alternatives, but their accuracy is not high enough (Kinect has an error of at least 4 mm and has a dead zone within 0.5 m. Additionally, the farther the distance is, the more inaccurate the measuring result is).

Many previous studies propose different approaches to produce depth estimation from images. Among them, a few researchers used optimization methods to handle the problem. Ranftl et al. propose a novel motion segmentation method to produce dense depth map from two consecutive frames through a single monocular camera. They segment the optical flow field into a set of motion models, with which the scene is reconstructed by minimizing a convex program [5]. Smith et al. present a method which estimates depth from a single polarisation image by solving a large, sparse system of linear equations [6]. Karsch proposes a technique that automatically generates plausible depth maps from videos using non-parametric depth sampling [7]. Optimization methods need the manually defined constraints to guarantee the accuracy of the resulted depth estimation. Thus, these methods require human experiences to give effective constraints.

Additionally, approaches using learning-based methods to model manually extracted features are other promising alternatives as well. Saxena et al. propose a Markov Random Field (MRF) learning algorithm to handle monocular cues like texture gradients and variations, defocus, etc. They incorporate these cues into a stereo system to obtain depth estimation results [8]. Ma et al. improve the ResNet-50 network by transfer learning to tackle the depth estimation problem through a single image [9]. Haim et al. propose a phase-coded aperture camera for depth estimation. They equip the camera with an optical phase mask to produce unambiguous depth-related color characteristics for the captured image [10]. Gan et al. present a convolutional neural network architecture that pays more attention to the relationships of different image locations and incorporates the absolute and relative features [11]. These methods depend on the extracted features. In some scenarios, these features might not incorporate sufficient cues of the depth information.

Recently, approaches using deep learning to generate depth maps from images become prevailing. Fu et al. propose a space-increasing discretization (SID) strategy to discretize depth and recast depth network learning as an ordinal regression problem [12]. Jiao et al. propose an approach to handle the depth estimation and semantic labeling task simultaneously. They present a concept called the attention-driven loss for the network supervision, and a synergy network to learn the relevance between the two tasks [13]. Godard et al. trained an unsupervised deep neural network with binocular stereo data to address the ground truth depth data deficient problem in traditional depth estimation methods. A novel training loss was proposed for the deep convolutional neural network to perform single image depth estimation with a high quality [14]. Inspired from the concept of autoencoders, Garg et al. trained the first convolutional neural network end-to-end from scratch for single view depth estimation in an unsupervised manner [15]. Deep learning methods are powerful tools for regressions. However, they usually need expensive platforms to run the algorithm and they usually need many training samples related to the target object for accurate depth estimation.

Different from the aforementioned previous approaches, the proposed method in this paper estimates depth information by means of 3D reconstruction. The first thing we need to do is to circle a camera around the target object and get its reconstruction at the very beginning. Afterwards, only a single monocular image is required to accurately estimate the depth information of the object no matter how it moves or becomes deformable in front of the camera. The proposed method is therefore more applicable in the scenarios where the target object is not rigid and where accurate depth information is necessary. With the point cloud of a target object (in a certain static pose) reconstructed beforehand, the proposed method can estimate the pose and reconstruct the point cloud of a target object (in its other poses) by a single input RGB image. This work can be used on not only humans and humanoid objects, but on other deformable objects as well.

The remainder of this paper is structured as follows. Section 2, Section 3 and Section 4 individually introduce the three modules of the proposed approach specifically. Section 5 introduces the whole flow chart of the proposed method. Experimental evaluations are provided on a NAO robot and a human in Section 6. Section 7 concludes the contributions of this paper.

2. 3D Labeled Reconstruction

This section introduces the priori model for the target object. The priori model is the 3D reconstruction (stored in the form of the point cloud [16,17,18]) of the target object in its stationary status, with a SIFT feature vector attached to each cloud point. Therefore, the priori model is built through two steps. First, we use a traditional 3D reconstruction approach (for static object) to reconstruct the target object by multiple images. Second, we use the SIFT algorithm [19] to extract feature vectors from the collected images, attaching them to the corresponding 3D points on the reconstructed point cloud.

2.1. 3D Reconstruction with Multiple Images

Given a static object, we can use a single camera (whose intrinsic parameter is f) to circle around the object, reconstructing a point cloud. In order to reconstruct the object, the total number of the images captured by the camera is N. Additionally, the total number of points on the object surface is M. The orientation and position of the camera with regard to the world frame at the ith instant can be respectively represented by a matrix $R_i$ and a vector $t_i$. Denote $P_j={\left(\begin{array}{ccc}{X}_j& Y_j& Z_j\end{array}\right)}^T$ as the jth point on the object surface with regard to the world frame, $P_j^i={\left(\begin{array}{ccc}{X}_j^i& Y_j^i& Z_j^i\end{array}\right)}^T$ as the same jth point with regard to the camera frame at the ith instant, and $m_j^i={\left(\begin{array}{cc}{x}_j^i& y_j^i\end{array}\right)}^T$ as the image coordinate of jth point at the ith instant (for denotation simplification, we define $m_j^i\in \varnothing$ if $P_j$ is occluded under the observation by the camera at ith instant). The following can be given [20]:

$$\left[\begin{array}{c}{P}_j^i\\ 1\end{array}\right]=\left[\begin{array}{cc}{R}_i& t_i\\ 0& 1\end{array}\right]\left[\begin{array}{c}{P}_j\\ 1\end{array}\right],$$

(1)

$$Z_j^i\left[\begin{array}{c}{m}_j^i\\ 1\end{array}\right]=\left[\begin{array}{cc}{f}^i{I}_{2\times 2}& O_{2\times 1}\\ O_{1\times 2}& 1\end{array}\right]P_j^i.$$

(2)

Let

$$v_j^i=Z_j^i{m}_j^i-\left[\begin{array}{cc}{f}^i{I}_{2\times 2}& O_{2\times 1}\end{array}\right]\left[\begin{array}{cc}{R}_i& t_i\end{array}\right]{\left[\begin{array}{cc}{P}_j^T& 1\end{array}\right]}^T.$$

(3)

Subsequently, the desired result in this step is the 3D reconstruction ${\left\{P_j^{*}\right\}}_{j=1\dots M}$ in the following form:

$${\left\{P_j^{*}\right\}}_{j=1\dots M}=arg\underset{{\left\{P_j\right\}}_{j=1\dots M}}{min}\sum _i^N\sum _j^M∥v_j^i∥.$$

(4)

2.2. SIFT Features to Label the 3D Reconstruction

Given a two-dimensional image $I\left(x,y\right)$, the SIFT algorithm [19] is able to extract effective key points through the LoG operator. By computing the gradients in the neighborhood of each key point, a corresponded descriptor vector can be obtained to distinguish the key point. Then, we can use the SIFT algorithm to find a set of feature points (denoted as ${\left\{m_s^i\right\}}_{s=1\dots S}$) and their corresponding descriptor vectors (denoted as ${\left\{l_s^i\right\}}_{s=1\dots S}$) for the image captured at the $i^{th}$ instant, which jointly yield a two-tuple set denoted as ${\left\{\left(m_s,l_s\right)\right\}}_{s=1\dots S}$. Executing the same operation to all the images, we can finally get ${\left\{\left(m_s^i,l_s^i\right)\right\}}_{\genfrac{}{}{0pt}{}{i=1\dots N\hfill }{s=1\dots S^i\hfill }}$ (where $S^i$ indicates the total number of feature points derived from the SIFT algorithm for the image captured at the ith instant).

Subsequently, we need to attach the descriptor vectors to the corresponding 3D points on the surface of the reconstructed point cloud. It can be deduced from Equations (1) and (2) that

$$P_j=R_i^T\left[\begin{array}{cc}\frac{d_j^i}{f^i}{I}_{2\times 2}& O_{2\times 1}\\ O_{1\times 2}& d_j^i\end{array}\right]\left[\begin{array}{c}{m}_j^i\\ 1\end{array}\right]-R_i^T{t}_i.$$

(5)

By Equation (5), we can determine the 3D point $P_s^i$ on the reconstructed point cloud corresponding to the feature point $m_s^i$. Thus, we can acquire a two-tuple set ${\left\{\left(P_s^i,l_s^i\right)\right\}}_{\genfrac{}{}{0pt}{}{i=1\dots N\hfill }{s=1\dots S^i\hfill }}$. For denotation simplification, we define that any $P_j$ that is occluded from the camera view at the ith instant, or whose corresponding $m_j^i$ is not a key point, still has a descriptor vector $l_j^i=0$. Therefore, we can get ${\left\{\left(P_j^i,l_j^i\right)\right\}}_{\genfrac{}{}{0pt}{}{i=1\dots N\hfill }{j=1\dots M\hfill }}$ after the Nth instant. The required 3D labeled reconstruction is the following:

$${\left\{\left(P_j^{*},{l_j}^{*}\right)\right\}}_{j=1\dots M}={\left\{\left(P_j^{*},{\overline{l}}_j\right)\right\}}_{j=1\dots M},$$

(6)

where ${\overline{l}}_j$ represents the average vector over all the non-zero descriptor vectors (i.e., $\left\{l_j^i\left|i\le N,l_j^i\ne 0\right.\right\}$) related to the 3D point $P_j$ within the Nth sampling instant.

3. Skeleton-Based Topological Segmentation

This section introduces how to provide the reconstructed point cloud with a robust topological segmentation, so as to deal with the case where the target object is not rigid. In this way, each sub-point-cloud derived from the topological segmentation is expected to be rigid. Topological segmentation based on the surface information of the target object is inclined to getting influenced by the surface noise, causing weak robustness. Therefore, the proposed topological segmentation is executed through two steps. First, we extract the skeleton of the reconstructed point cloud [21] and segment the skeleton based on its curvature. Second, we dilate the sub-skeletons [22] to yield the sub-point-clouds, which are the results of the topological segmentation.

Skeleton extraction and segmentation. Given an object denoted as $\mathsf{\Omega }={\left\{P_j^{*}\right\}}_{j=1\dots M}$, we denote $CORE\left(\mathsf{\Omega }\right)$ as the set of all the maximally inscribed spheres in $\mathsf{\Omega }$, none of which has common points of tangency with noisy surface. Then, the skeleton of $\mathsf{\Omega }$ is denoted as $S\left(\mathsf{\Omega }\right)$.

After extracting the skeleton of the reconstructed point cloud, we segment the skeleton according to its curvature. Supposing $C\left(\mathsf{\Omega }\right)\subset S\left(\mathsf{\Omega }\right)$, an equivalence relation ${\sim }_C$ implicated by $C\left(\mathsf{\Omega }\right)$ is defined as $\forall p_1,p_2\in S\left(\mathsf{\Omega }\right)$, $p_1{\sim }_{C_1}{p}_2$ if and only if $p_1$, $p_2$ are on a curve segment whose ends are two points in $C\left(\mathsf{\Omega }\right)$ and no other points in $C\left(\mathsf{\Omega }\right)$ are on the same curve segment.

Thus, the curve segments determined by $C\left(\mathsf{\Omega }\right)$ are the equivalence classes [23] $S_C\left(\mathsf{\Omega }\right)$ defined as

$$S_C\left(\mathsf{\Omega }\right)=S\left(\mathsf{\Omega }\right)/{\sim }_C=\left\{\left\{p\in S\left(\mathsf{\Omega }\right):p{\sim }_{C}q\right\}:q\in S\left(\mathsf{\Omega }\right)\right\}.$$

(7)

It is clear that the curve segments in $S_C\left(\mathsf{\Omega }\right)$ are separated from each other. In this paper, we propose two categories of points, respectively, denoted as $C_1\left(\mathsf{\Omega }\right)$ and $C_2\left(\mathsf{\Omega }\right)$ in order that $C\left(\mathsf{\Omega }\right)=C_1\left(\mathsf{\Omega }\right)\cup C_2\left(\mathsf{\Omega }\right)$ determines ${\sim }_C$ in Equation (7).

3.1. The First Category $C_1\left(\mathsf{\Omega }\right)$

Supposing all the points on the skeleton $S\left(\mathsf{\Omega }\right)$ constitute a set ${\left\{p_k\right\}}_{k=1\dots K}$ (K is the total number of elements in $S\left(\mathsf{\Omega }\right)$). We use a set ${\left\{e_{mn}\right\}}_{\genfrac{}{}{0pt}{}{0<m\le K\hfill }{0<n\le K\hfill }}$ to represent the connectivity of each two points from $S\left(\mathsf{\Omega }\right)$. Specifically, $e_{mn}=1$ if $p_m$ and $p_n$ are adjacent to each other. Otherwise, $e_{mn}=0$. Subsequently, the first category of points $C_1\left(\mathsf{\Omega }\right)$ is defined as

$$C_1\left(\mathsf{\Omega }\right)=\left\{p_m\in {\left\{p_k\right\}}_{k=1\dots K}\left|\sum _n^K{e}_{mn}>2or\sum _n^K{e}_{mn}=1\right.\right\}.$$

(8)

3.2. The Second Category $C_2\left(\mathsf{\Omega }\right)$

Similar to $S_C\left(\mathsf{\Omega }\right)$ (Equation (7)), $S\left(\mathsf{\Omega }\right)$ segmented by $C_1\left(\mathsf{\Omega }\right)$ is

$$S_{C_1}\left(\mathsf{\Omega }\right)=S\left(\mathsf{\Omega }\right)/{\sim }_{C_1}.$$

(9)

We define the function of the cth curve segment (suppose C curve segments totally) in $S_{C_1}\left(\mathsf{\Omega }\right)$ as $r_c\left(u_c\right)$, where $u_c$ is the arc length parameter of the function $r_c$ and $u_c\in \left(0,L_c\right)$ ($L_c$ is the length of the cth curve segment). Then, the Frenet formulas [24] of the cth curve is

$$\frac{d}{d{u}_c}\left[\begin{array}{c}{r}_c\\ {\alpha }_c\\ {\beta }_c\\ {\gamma }_c\end{array}\right]=\left[\begin{array}{ccc}1& 0& 0\\ 0& {\kappa }_c& 0\\ -{\kappa }_c& 0& {\tau }_c\\ 0& -{\tau }_c& 0\end{array}\right]\left[\begin{array}{c}{\alpha }_c\\ {\beta }_c\\ {\gamma }_c\end{array}\right],$$

(10)

where ${\alpha }_c$, ${\beta }_c$ and ${\gamma }_c$ are, respectively, the unit vector tangent, normal unit vector and binormal unit vector of $r_c$; ${\kappa }_c$ and ${\tau }_c$ are the curvature and torsion.

We construct a quantity ${\delta }_c$ satisfying that ${\delta }_c\propto {\kappa }_c$ and ${\delta }_c\propto {\tau }_c$ as well as a threshold $\upsilon$. Subsequently, the second category of points $C_2\left(\mathsf{\Omega }\right)$ is defined as

$$C_2\left(\mathsf{\Omega }\right)={\left\{p=r_c\left(u_c\right)\left|{\delta }_c\right.\ge \upsilon \right\}}_{c=1\dots C}.$$

(11)

3.3. Skeleton Dilation with Constraints

Supposing B is a structuring element [20] in the form of a subset in $R^3$, the dilation of $S\left(\mathsf{\Omega }\right)$ by B after T iterations is defined as

$$B_T\left(S\left(\mathsf{\Omega }\right)\right)=\underset{t=1\dots T}{\oplus }B\oplus S\left(\mathsf{\Omega }\right).$$

(12)

The dilation operation is executed to segment the reconstructed point cloud by the sub-skeletons. Therefore, sub-point-clouds should be separated from each other. Additionally, the dilation should be stopped when reaching the surface of the reconstructed point cloud. Thus, in each iteration $t\in \left\{1\dots T\right\}$, we remove the points that violate the following two constraints:

$$B_t\left(S_{c_1}\left(\mathsf{\Omega }\right)\right)\cap B_t\left(S_{c_2}\left(\mathsf{\Omega }\right)\right)=\varnothing ,$$

(13)

$$B_t\left(S_{c_1}\left(\mathsf{\Omega }\right)\right)\cap \overline{\mathsf{\Omega }}=\varnothing ,$$

(14)

where $S_{c_1}\left(\mathsf{\Omega }\right)$ and $S_{c_2}\left(\mathsf{\Omega }\right)$ represent two distinct sub-skeletons in $S_C\left(\mathsf{\Omega }\right)$, $\overline{\mathsf{\Omega }}$ is the external space of $\mathsf{\Omega }$.

Moreover, for the dilation of each sub-skeleton $S_{c_1}\left(\mathsf{\Omega }\right)$ under constraints Equations (13) and (14), the corresponding total iteration times T satisfy that

$$B_{T+1}\left(\mathsf{\Omega }\right)=B_T\left(\mathsf{\Omega }\right),B_T\left(\mathsf{\Omega }\right)-B_{T-1}\left(\mathsf{\Omega }\right)\ne \varnothing .$$

(15)

Then, the dilation result of sub-skeletons in $S_C\left(\mathsf{\Omega }\right)$ under constraints in the form of Equations (13)–(15) forms a equivalence relation for topological segmentations $\left(\mathsf{\Omega },\mathcal{T}\right)$.

4. 3D Reconstruction at $\mathit{i}$th Time

This section introduces how to quickly reconstruct the dynamic object with a single RGB camera. For each frame by the camera, we first extract all the feature points. Through matching these feature points to those on the reconstructed point cloud, we are able to know to which sub-point-clouds these feature points individually correspond. Finally, the poses of the sub-point-clouds can be estimated by the correspondences between the feature points on the point cloud and the image feature points. This pose estimation problem can be easily handled by solving a nonlinear optimization. The reconstruction is therefore reduced to the reorganization of the sub-point-clouds with updated poses.

Denote the image captured at ith instant ($i>N$) as ${\left\{m_i^j\right\}}_{j=1\dots M}$. Through the SIFT algorithm, we can extract the feature points and their corresponding descriptor vectors. Through Section 2.2, a labeled 3D point cloud is reconstructed, to which the descriptor vectors are attached. Thus, we can use the descriptors from the captured image and from the cloud to find their correspondence, which can be denoted as a two-tuple set ${\left\{\left(m_i^{f_i},P_{f_i}^{*}\right)\right\}}_{f_i=j_1\dots j_F}$. In this set, $m_i^{f_i}$ and $P_{f_i}^{*}$ respectively represent the feature point from the image and the cloud point, whose descriptor vectors are similar. In addition, ${\left\{\left(m_i^{f_i},P_{f_i}^{*}\right)\right\}}_{f_i=j_1\dots j_F}$ yields a bijective map ${\phi }_i:{\left\{m_i^{f_i}\right\}}_{f_i=j_1\dots j_F}\to {\left\{P_{f_i}^{*}\right\}}_{f_i=j_1\dots j_F}.$

Suppose $\mathcal{T}=\left\{\left[p\right]:p\in \mathsf{\Omega }\right\}=\left\{\left\{q\in \mathsf{\Omega }:q\sim p\right\}:p\in \mathsf{\Omega }\right\}$ as the basis for the topology of space $\mathsf{\Omega }={\left\{P_j^{*}\right\}}_{j=1\dots M}$. We can get the basis for topology (denoted as ${\mathcal{T}}_i$) of the subset ${\left\{P_{f_i}^{*}\right\}}_{f_i=j_1\dots j_F}\subset {\left\{P_{f_i}^{*}\right\}}_{j=1\dots M}$ by

$${\mathcal{T}}_i=\left\{T_c\cap {\left\{P_{f_i}^{*}\right\}}_{f_i=j_1\dots j_F}\left|T_c\in \mathcal{T}\right.\right\}.$$

(16)

We can further acquire the basis for the topology (denoted as ${\mathcal{T}}_{m_i}$) of the set ${\left\{m_i^{f_i}\right\}}_{f_i=j_1\dots j_F}$ by

$${\mathcal{T}}_{m_i}=\left\{{\phi }_i^{-1}\left(T_{m_i}\right)\left|T_{m_i}\in {\mathcal{T}}_i\right.\right\}.$$

(17)

Based on our design, each element in $\mathcal{T}$ is a rigid component of $\mathsf{\Omega }$. Thus, when the object represented by $\mathsf{\Omega }$ moves stochastically, the elements in $T_c\in \mathcal{T}$ have the same rigid transformation, i.e., there exists a single pair $\left(R_i^c,t_i^c\right)$ for $T_c^i$ such that

$$T_c^i=R_i^c{T}_c+t_c^i,$$

(18)

where $R_i^c$ is a three-dimensional rotation matrix, $t_i^c$ is a three-dimensional translation vector and $R_i^c{T}_c^i+t_c^i$ is defined as

$$R_i^c{T}_c^i+t_c^i=\left\{R_i^{c}p+t_c^i\left|p\in T_c^i,\right.T_c^i\in \mathcal{T}\right\},$$

(19)

where p is denoted in the form of a column vector.

Defining $T_{c_i}=T_c\cap {\left\{P_{f_i}^{*}\right\}}_{f_i=j_1\dots j_F}$, since $T_{c_i}\subset T_c$, based on Equation (18), we can get the actual coordinates of $T_{c_i}^i$ at $i^{th}$ instant as

$$T_{c_i}^i=R_i^c{T}_{c_i}+t_c^i.$$

(20)

Then, we can get another expression of ${\phi }^{-1}\left(T_{c_i}\right)$ as

$${\phi }^{-1}\left(T_{c_i}\right)=\left\{m_i\left(p\right)\left|p\in T_{c_i}^i\right.\right\},$$

(21)

where $m_i\left(·\right)$ (based on Equation (2)) is an operator to transform a three-dimensional point to a two-dimensional coordinate related to the camera at ith instant, which is defined as

$$\forall p\in R^3,m_i\left(p\right)=\frac{f^i}{\left[\begin{array}{cc}{O}_{1\times 2}& 1\end{array}\right]p}\left[\begin{array}{cc}{I}_{2\times 2}& O_{2\times 1}\end{array}\right]p.$$

(22)

Thus, we can compute a $\left(R_i^c,t_i^c\right)$ for $T_c^i$ by solving the following optimization problem:

$$\left(R_i^c,t_i^c\right)=arg\underset{\left(R_i^c,t_i^c\right)}{min}\sum _{p\in T_{c_i}}{\left[{\phi }^{-1}\left(p\right)-m_i\left(R_i^{c}p+t_i^c\right)\right]}^2.$$

(23)

Finally, the raw result of the 3d reconstruction ${\left\{{\tilde{P}}_i^j\right\}}_{\begin{array}{l}i>N\\ j=1\dots M\end{array}}$ based on ${\left\{m_i^{f_i}\right\}}_{f_i=j_1\dots j_F}$ at ith time is

$${\left\{{\tilde{P}}_i^j\right\}}_{\begin{array}{l}i>N\\ j=1\dots M\end{array}}=\bigcup _{T_c\in \mathcal{T}}\left(R_i^c{T}_c+t_i^c\right).$$

(24)

5. Approach Overview

Specifically, we utilize a single camera to resolve the dynamic 3D object reconstruction. The problem can be formulated as: given the images ${\left\{m_i^j\right\}}_{\genfrac{}{}{0pt}{}{i=1\dots N}{j=1\dots M}}$ captured within time (the pose and position alters at each time $i\le N$ in order that static 3D reconstruction of the object can be satisfactorily achieved) and images captured at time $i>N$, the expected result is the dense reconstruction of the object ${\left\{P_i^j\right\}}_{\begin{array}{l}i>N\\ j=1\dots M\end{array}}$ at the same time $i>N$.

Accordingly, we consider to fully utilize the 3D information acquired from the foregoing frames and then reduce the dynamic object reconstruction problem to a re-organizing problem. Thus, the proposed approach mainly incorporates three steps.

In the first step, we obtain the static 3D reconstruction of the target object in its stationary status. Specifically, the point cloud of the target object is acquired in the first phase through existing static 3D reconstruction methods. Meanwhile, we obtain the SIFT features on each image (among images utilized for 3D reconstruction) and attach the feature descriptors to the corresponding points on the reconstructed point cloud.

In the second step, we find an appropriate topological segmentation for the reconstructed point cloud such that each topological part moves rigidly during the object motion. Actually, the point cloud topological segmentation is also an open problem due to the fact that it is difficult to determine the standard for a satisfactory segmentation. In this paper, we transfer the point cloud segmentation problem into the skeleton segmentation problem, i.e., the segmentation of the point cloud results from the segmentation of its skeleton. This is based on the thought that the object skeleton is much more stable toward perturbation than the object surface. Thus, we can simply segment the skeleton into several sub-skeletons based on its curvature and torsion property before dilating each sub-skeleton to determine a corresponding topological part of the point cloud.

In the third step, when capturing a new image, we extract the SIFT features and match these features to those attached to the point cloud in the first phase. These matched features implicate the correspondence between the image and the point cloud. Based on the topological segmentation executed on the point cloud, the correspondence between the image and each topological part can also be computed. Then, the pose and position of each topological part can be deduced, and the reconstruction result can thus be obtained through a simple re-organization of these topological parts. The whole flow chart of the proposed approach is illustrated in Figure 3.

Construction of rotation matrix. Since the analytic expression of rotation matrix $R_i^c$ in Equations (18), (20), (23) and (24) is required for optimization program (following Equation (23)), we utilize the quaternion to construct the rotation matrix. Specifically, a quaternion is in the form of

$$q=w+x·\overrightarrow{i}+y·\overrightarrow{j}+z·\overrightarrow{k}.$$

(25)

Then, the corresponding rotation $R_i^c$ in the optimization problem of Equation (23) is

$$R_i^c=\left[\begin{array}{ccc}1-2\left(y^2+z^2\right)& 2\left(xy-zw\right)& 2\left(xz+yw\right)\\ 2\left(xy+zw\right)& 1-2\left(x^2+z^2\right)& 2\left(yz-xw\right)\\ 2\left(xz-yw\right)& 2\left(yz+xw\right)& 1-2\left(x^2+y^2\right)\end{array}\right],$$

(26)

with a constraint as

$$w^2+x^2+y^2+z^2=1,$$

(27)

which constitutes a simple constrained nonlinear optimization problem.

6. Experiments and Discussion

This section provides the experiment results from the proposed approach. We evaluate the proposed approach by a set of experiments on a NAO robot Figure 4 and a human being.

6.1. Experiments on a NAO Robot

NAO robot is an autonomous, programmable humanoid robot developed by Aldebaran Robotics (France), with a height of 58 centimetres and 25 degrees of freedom. We use a NAO robot as the target object for 3D reconstruction. During the experiment, the NAO robot continuously changes its poses such that it is deformable. We reconstruct the NAO robot to test the depth accuracy by the proposed algorithm.

This experiment is undertaken on a single monocular camera to reconstruct the NAO robot in its dynamic status. In order to guarantee the effectiveness of the proposed method, several specific processing procedures are listed in Appendix A, Appendix B and Appendix C.

Other procedures of the proposed approach in experiments follow the descriptions and formulas in Section 2, Section 3 and Section 4.

We compare the proposed method with approaches in [14,25]. The depth estimation result by the proposed method is shown in Figure 5. Note that the proposed method only estimates the depth information of the target object while the previous approaches estimate the depth of the whole image. Therefore, we only compare the depth estimation accuracies corresponding to the image regions where the target object appears. Four accuracy metrics are used [26] as shown in

Table 1. Quantitative comparison on the aspects of accuracy and efficiency.

Metrics Names	Equations
Abs Relative Difference	$\frac{1}{T}{\displaystyle \sum _{y\in T}}\left|y-y^{*}\right|/y^{*}$
Squared Relative difference	$\frac{1}{T}{\displaystyle \sum _{y\in T}}{\left|y-y^{*}\right|}^2/y^{*}$
RMSE (linear)	$\sqrt{\frac{1}{T}{\displaystyle \sum _{y\in T}}{\left|y-y^{*}\right|}^2}$
RMSE (log)	$\sqrt{\frac{1}{T}{\displaystyle \sum _{y\in T}}{\left|logy-log{y}^{*}\right|}^2}$

.

The depth accuracies of the proposed method and other three approaches are shown in

Table 2. Quantitative comparison on the aspects of accuracy and efficiency.

Approaches	Abs Relative Difference	Squared Relative Difference	RMSE (linear)	RMSE (log)
The approach in [25]	4.8693	510.8358	452.1486	3.3712
The approach in [14]	1.1972	53.7397	182.3361	1.2484
The proposed approach	0.9987	31.7892	152.0740	1.0743

.

6.2. Experiments on a Human

Experiments are also undertaken on a human to verify the proposed approach. We produce the 3D reconstruction of the human in his still pose at the very beginning. Subsequently, the human changes his pose and the camera captures the corresponding monocular image for each pose. We use these images to test the proposed approach and the previous algorithms [14,25]. The program to implement this experiment is the same as that for the NAO robot experiment.

The depth estimation results by the proposed method and other approaches are shown in Figure 6. Additionally, the depth accuracies of the proposed method and other three approaches are shown in

Table 3. Quatitative comparison on the aspects of accuracy and efficiency.

Approaches	Abs Relative Difference	Squared Relative Difference	RMSE (linear)	RMSE (log)
The approach in [25]	5.9473	638.9376	367.0673	3.7256
The approach in [14]	2.0792	197.2335	216,7147	2.1638
The proposed approach	1.8274	83.5462	187.4539	2.0132

.

Similar to the experiments taken on NAO robot, the experiments results can validate the effectiveness of the proposed method. The proposed method uses the 3D point cloud reconstructed ahead of time as the priori model. Subsequently, it depends on the image features to estimate the pose changes of the priori model, therefore being capable of estimating the depth information accurately.

7. Conclusions

In this paper, we propose a feature-based approach to estimate the depth of a deformable object accurately via a monocular camera. The proposed approach needs to reconstruct the target object in its initial pose as a priori model. Afterwards, only one monocular image is required to to accurately estimate the depth of the target object no matter how it changes its pose. Experiments are undertaken on a NAO robot and a human to evaluate the accuracy of the proposed approach. In future work, we aim to accurately estimate depth of the same kind of deformable object, by only reconstructing a single instance of that kind as the priori model.