Reliable Fusion of Stereo Matching and Depth Sensor for High Quality Dense Depth Maps

Abstract

Depth estimation is a classical problem in computer vision, which typically relies on either a depth sensor or stereo matching alone. The depth sensor provides real-time estimates in repetitive and textureless regions where stereo matching is not effective. However, stereo matching can obtain more accurate results in rich texture regions and object boundaries where the depth sensor often fails. We fuse stereo matching and the depth sensor using their complementary characteristics to improve the depth estimation. Here, texture information is incorporated as a constraint to restrict the pixel’s scope of potential disparities and to reduce noise in repetitive and textureless regions. Furthermore, a novel pseudo-two-layer model is used to represent the relationship between disparities in different pixels and segments. It is more robust to luminance variation by treating information obtained from a depth sensor as prior knowledge. Segmentation is viewed as a soft constraint to reduce ambiguities caused by under- or over-segmentation. Compared to the average error rate 3.27% of the previous state-of-the-art methods, our method provides an average error rate of 2.61% on the Middlebury datasets, which shows that our method performs almost 20% better than other “fused” algorithms in the aspect of precision.

1. Introduction

Depth estimation is one of the most fundamental and challenging problems in computer vision. For decades, it has been important for many advanced applications, such as 3D reconstruction [1], robotic navigation [2], object recognition [3] and free viewpoint television [4]. Approaches for obtaining 3D depth estimation can be distinguished into two categories: passive and active. The goal of passive methods like stereo matching is to estimate a high-resolution dense disparity map by finding corresponding pixels in image sequences [5]. However, these methods heavily rely on how the scene is presented and contain error matchings caused by the luminance variation. Passive methods fail in textureless and repetitive regions where there is not enough visual information to obtain the correspondence. On the contrary, active methods, like depth sensors (ASUS Xtion [6] and Microsoft Kinect [7]), do not suffer from ambiguities in textureless and repetitive regions, because they emit an infrared signal. Unfortunately, sensor errors and the properties of the object surfaces mean that depth maps from a depth sensor are often noisy [8]. Additionally, their resolution is at least an order of magnitude lower than common digital single-lens reflex (DSLR) cameras, which limits many applications. Moreover, they cannot satisfactorily deal with object boundaries and a wide range of distances. Therefore, fusing different kinds of methods using their complementary characteristics undoubtedly makes the obtained depth map more robust and improves the quality. Commonly-used consumer DSLR cameras have higher resolution and can record better texture information than depth sensors. Therefore, it is reasonable to fuse the depth sensor with DSLR cameras to yield a high resolution depth map. Note that for notational clarity, all values mentioned here are disparities (considering that depth values are inversely proportional to disparities).

Figure 1. The system used in our method. It consists of two Cannon EOS 700D digital single-lens reflex (DSLR) cameras and one Xtion depth sensor. All DSLR cameras are controlled by the wireless remote controller. We used an adjustable bracket to change the angle and height of the Xtion depth sensor.

Figure 1. The system used in our method. It consists of two Cannon EOS 700D digital single-lens reflex (DSLR) cameras and one Xtion depth sensor. All DSLR cameras are controlled by the wireless remote controller. We used an adjustable bracket to change the angle and height of the Xtion depth sensor.

In this paper, we propose a novel disparity estimation method for the system shown in Figure 1. It fuses the complementary characteristics of high resolution DSLR cameras and the Xtion depth sensor to obtain an accurate disparity estimate. Compared to the average error rate 3.27% of the previous state-of-the-art methods, our method provides an average error rate of 2.61% on the Middlebury datasets. It is clear that our method performs almost 20% better than other “fused” algorithms in the aspect of precision. The proposed method views a scene with complex geometric characteristics as a set of segments in the disparity space. It assumes that the disparities of each segment have a compact distribution, which strengthens the smooth variance of the disparities in each segment. Additionally, we assume that each segment is biased towards being a 3D planar surface. The major contributions are as follows:

• We incorporated texture information as a constraint. The texture variance and gradient is used to restrict the range of the potential disparities for each pixel. In textureless and repetitive regions (which often cause ambiguities when stereo matching), we restrict the possible disparities for a neighborhood centered on each pixel to a limited range around the values suggested by the Xtion. This reduces the errors and strengthens the compact distribution of the disparities in a segment.
• We propose the multiscale pseudo-two-layer image model (MPTL; Figure 2) to represent the relationships between disparities at different pixels and segments. We consider the disparities from the Xtion as the prior knowledge and use it to increase the robustness to luminance variance and to strengthen the 3D planar surface bias. Furthermore, considering the spatial structures of segments obtained from the depth sensor, we treat the segmentation as a soft constraint to reduce matching ambiguities caused by under- and over-segmentation. Here, pixels with similar colors, but on different objects are grouped into one segment, and pixels with different colors, but on the same object are partitioned into different segments. Additionally, we only retain the disparity discontinuities that align with object boundaries from geometrically-smooth, but strong color gradient regions.

Figure 2. The illustration of the multiscale pseudo-two-layer (MPTL) model. The rectified left (a) and right (b) images from DLSRs, the segmentation; (c) as well as the depth map (d) from the depth sensor are taken as our inputs; (e) The conceptual structure of our MPTL. The MPTL captures the complementary characteristics of active and passive methods by allowing interactions between them. All interactions are defined to act in the segment-level component, the pixel-level component and the edges that connect them (Segment Pixel-edge, SP-edge). See Section 3.2 for the full details of the MPTL model and Section 4 for further results.

Figure 2. The illustration of the multiscale pseudo-two-layer (MPTL) model. The rectified left (a) and right (b) images from DLSRs, the segmentation; (c) as well as the depth map (d) from the depth sensor are taken as our inputs; (e) The conceptual structure of our MPTL. The MPTL captures the complementary characteristics of active and passive methods by allowing interactions between them. All interactions are defined to act in the segment-level component, the pixel-level component and the edges that connect them (Segment Pixel-edge, SP-edge). See Section 3.2 for the full details of the MPTL model and Section 4 for further results.

The remainder of this paper is organized as follows. Section 2 gives a summary of various methods used for disparity estimation. We present a pre-processing and some important notations of our model in Section 3.1. We discuss the details of the MPTL image model in Section 3.2, the optimization in Section 3.7 and the post-processing in Section 3.8. Section 4 contains our experiments, and Section 5 presents some conclusions with suggestions for future work.

2. Previous Work

There are many approaches to obtaining disparity estimation. They can generally be categorized into two major classes: passive and active. A passive method indirectly obtains the disparity map using image sequences captured by cameras from different viewpoints. Among the plethora of passive methods, stereo matching is probably the most well known and widely applied. Stereo matching algorithms can be divided into two categories [9]: local and global methods. Local methods [10] estimate disparity using color or intensity values in a support window centered on each pixel. However, they often fail around disparity discontinuities and low-texture regions. Global methods [11] use a Markov random field model to formulate the stereo matching as a maximum a posteriori probability energy function with explicit smoothness priors. They can significantly minimize matching ambiguities compared to local methods. However, the biggest disadvantage of them is the low computational efficiency. Segmentation-based global approaches [12,13] encode the scene as a set of non-overlapping homogeneous color segments. They are based on the hypothesis that the variance of the disparity in each segment is smooth. In other words, the segment boundaries are forced to coincide with object boundaries. Recently, the ground control point (GCP)-based methods [14] were used as prior knowledge to encode rich information on the spatial structure of the scene. Although a significant number of stereo matching methods have been proposed for obtaining dense disparity estimation, they heavily rely on radiometric variations and assumptions regarding the presentation of the scene. This means that stereo matching often fails in textureless and repetitive regions, where there is not enough visual information to obtain a correspondence. Furthermore, their accuracy is relatively low. Passive methods heavily rely on the luminance condition and how the scene is presented. They often fail in textureless and repetitive regions where there is not enough visual information to obtain the correspondence.

On the contrary, active methods like depth sensors, do not suffer from ambiguities in textureless and repetitive regions, because they emit an infrared signal. Three different kinds of equipments are used in active methods: a laser scanner device, a time-of-flight (ToF) sensor and an infrared single-based device (such as ASUS Xtion [6] and Microsoft Kinect [7]). The laser scanner device [15] can provide extremely accurate and dense depth estimation, but it is too slow to use in real time and too expensive for many applications. The ToF sensor and infrared single-based device can obtain real-time depth estimation and have recently become available from companies, such as 3DV [16] and PMD [17]. However, sensor errors and the properties of the object surfaces mean that depth maps from them are often noisy [8]. Additionally, their resolution is at least an order of magnitude lower than commonly-used DSLR cameras [18], which limits many applications. Moreover, they cannot satisfactorily deal with object boundaries.

It is clear that each disparity acquisition method is limited in some aspects where other approaches may be effective. Joint optimization methods that combine active and passive sensors have been used to make the obtained depth map more robust and to improve the quality. Zhu et al. please check throughout [19,20] fused a ToF sensor and stereo cameras to obtain better disparity maps. They improved the quality of the estimated maps for dynamic scenes by extending their fusion technology to the temporal domain. Yang et al. [21] presented a fast depth sensing system that combined the complementary properties of passive and active sensors in a synergistic framework. It relied on stereo matching in rich textured regions, while using data from depth sensors in textureless regions. Zhang et al. [22] proposed a system that addresses high resolution and high quality depth estimation by fusing stereo matching and a Kinect. A pixel-wise weighted function was used to reflect the reliabilities of the stereo camera and the Kinect. Wang et al. [23] presented a novel method that combined the initial stereo matching result and the depth data from a Kinect. Their method also considers the visibilities and pixel-wise noise of the depth data from a Kinect. Gowri et al. [24] proposed a global optimization scheme that defines the data and smoothness costs using sensor confidences and the low resolution geometry from a Kinect. They used a spatial search range to limit the scope of the potential disparities at each pixel. The smoothness prior was based on the available low resolution depth data from the Kinect, rather than the image color gradients.

Although existing disparity estimation methods have achieved remarkable results, they are typically performed using pixel-level cues, such as the smoothness of neighboring pixels, and do not consider the regional information (regarding, for example, 3D spatial structure, segmentation and texture) as a cue for the disparity estimation, which is the largest distinction between their method and ours. For example, occlusion cannot be precisely estimated using a single pixel, but a fitted plane-based filling occlusion in a segment can give good results. Additionally, if the spatial structure of neighboring segments is not known, matching ambiguities can arise at the boundaries of neighboring segments that physically belong to the same object, but have different appearances. Without texture information, we cannot be sure if the disparity from stereo matching is more confident than that from the depth sensor in textureless and repetitive regions (where stereo matching usually fails and the depth sensor performs well).

3. Method

The proposed method can be partitioned into four phases: pre-processing, problem definition, optimization and post-processing. Each phase will be discussed in detail later.

3.1. Pre-Processing

There are three camera coordinates involved in our system (Figure 1): the Xtion coordinate, the coordinates of the two DSLR cameras before the epipolar rectification and the DSLR camera coordinates after the epipolar rectification. During the pre-processing step, in order to combine the data from the Xtion and DSLR cameras, as shown in Figure 3, we firstly calibrated two DSLR cameras using the checkerboard-based method [25] and calibrated the DSLR camera pair with the Xtion sensor using the planar surfaces-based method [26], respectively. After the calibration, the depth image obtained from the Xtion is first transformed from the Xtion coordinate to the original DSLR cameras’ coordinates, then rotated and up-sampled, so that it registers with the unrectified left image. Furthermore, according to the theory of epipolar geometric constraints, the registered depth image and original left image, as well as the original right image are rectified to be row-aligned, which means there are only horizontal disparities in the row direction. We denote the seed image (Π) as the map with disparities transferred from the rectified depth map. Each pixel $p\in \Pi$ is defined as a seed pixel when it is assigned a non-zero disparity. The initial disparity maps ($D_L$ and $D_R$) of the rectified left and right images ($I_L$ and $I_R$) are computed using a local stereo matching method [27]. $I_L$ is partitioned into a set of segments using the edge-aware filter-based segmentation algorithm [28].

Figure 3. Conceptual flow diagram for the calibration and rectification phase.

Figure 3. Conceptual flow diagram for the calibration and rectification phase.

In addition, as shown in Figure 4, all pixels and segments are divided into different categories. The occlusion judgment is used to find the occluded pixels with initial disparity maps ($D_L$ and $D_R$ of $I_L$ and $I_R$, respectively) and to classify pixels into different categories: reliable and occluded. As we known, how to find occluded pixels accurately is always the challenging problem, because it often leads to error results that matching points might not even exist at all, especially in depth discontinuities. Pixels are defined as occluded when they are only visible from the left rectified view ($I_L$), but not from the right rectified view ($I_R$). Since image pairs have been rectified, we assume that occlusion only occurs in the horizontal direction. In early algorithms, cross-consistency checking is often applied to identify occluded pixels by enforcing a one-to-one correspondence between pixels. It is written as:

$$O\left(p\right)=\left\{\begin{array}{cc}0\hfill & |D_L\left(p\right)-D_R\left(q\right)|<1\hfill \\ 1\hfill & \mathrm{otherwise}\hfill \end{array}\right.p\in I_L,q\in I_R$$

(1)

$D_L\left(p\right)$ and $D_R\left(q\right)$ are the disparity of p and q, and q is the corresponding matching point of p. If p does not meet the cross-consistency checking, then it will be regard as an occluded pixel ($O\left(p\right)=0$); otherwise, p is a reliable pixel ($O\left(p\right)=1$). The cross-consistency checking states that a pixel of one image corresponds to at most one pixel of the other image. However, because of different sampling, the projection of a horizontal slant or a curved surface shows various lengths in the image pairs. Therefore, conventional cross-consistency checking that often identifies occluded pixels by enforcing a one-to-one correspondence is only suitable for a frontal parallel surface and cannot be true for a horizontal slant or curved surfaces. Considering the different sampling of image pairs, Bleyer et al. [29] proposed a new visibility constraint by extending the asymmetric occlusion model [30] that allows a one-to-many correspondence between pixels. Let $p_0$ and $p_1$ be neighboring pixels in the same horizontal line of $I_L$. Then, $p_0$ will be occluded by $p_1$ when they meet three conditions:

-

$p_0$ and $p_1$ have the same matching point in $I_R$ under their current disparity value;

-

$D_L\left(p_0\right)\le D_L\left(p_1\right)$;

-

$p_0$ and $p_1$ belong to different segments.

In this paper, for each pixel p of $I_L$, if there is only one matching point in $I_R$, the conventional cross-checking is applied to obtain the occlusion Equation (1). Otherwise, if there are more than two matching points in $I_R$, pixels in $I_L$ are marked as either reliable ($O\left(p_0\right)=0$) or occluded ($O\left(p_0\right)=1$), which satisfy or do not satisfy the Bleyer’s asymmetric occlusion model. As shown in Figure 4, each segment belongs to the reliable segment (R) if it contains a sufficient amount of reliable pixels; otherwise, it belongs to the unreliable segment ($\overline{R}$). Furthermore, each segment $f_i\in R$ is denoted as a stable segment (S) when it contains a sufficient number of seed pixels. Otherwise, $f_i$ belongs to the unstable segment ($\overline{S}$). We apply a RANSAC-based algorithm to approximate each stable segment $s_i\in S$ as a fitted plane ${\Psi }_{s_i}$ using the image coordinates and known disparities of all seed pixels belonging to $s_i$. Table 1 lists important notifications used in this paper.

Figure 4. Conceptual flow diagram for the classification phase.

Figure 4. Conceptual flow diagram for the classification phase.

Table 1. Notations.

D:	Disparity map	$D\left(p\right)$:	Disparity value of pixel p	Π:	Seed image
$D_L$:	Initial disparity map of rectified left image	$D_R$:	Initial disparity map of rectified right image	$I_L$:	Rectified left DSLR image
$I_R$:	Rectified right DSLR image	R:	Reliable segment	$\overline{R}$:	Unreliable segment
S:	Stable segment	$\overline{S}$:	Unstable segment	$f_i$:	i-th segment
$s_i$:	i-th stable segment	${\Psi }_{s_i}$:	Fitted plane of stable segment $s_i$	$f\left(p\right)$:	Segment that contains pixel p
$f_i$:	i-th segment	$d_l$:	Minimum disparity	$d_u$:	maximum disparity
ϖ:	Segment boundary pixels	${\Lambda }_l^p$:	Pixel’s potential minimum disparity	${\Lambda }_u^p$:	Pixel’s potential maximum disparity
${\Psi }_i^l$:	Minimum fitted disparity of the i-th stable segment	${\Psi }_i^u$:	Minimum fitted disparity of the i-th stable segment

Table 1. Notations.

D:	Disparity map	$D\left(p\right)$:	Disparity value of pixel p	Π:	Seed image
$D_L$:	Initial disparity map of rectified left image	$D_R$:	Initial disparity map of rectified right image	$I_L$:	Rectified left DSLR image
$I_R$:	Rectified right DSLR image	R:	Reliable segment	$\overline{R}$:	Unreliable segment
S:	Stable segment	$\overline{S}$:	Unstable segment	$f_i$:	i-th segment
$s_i$:	i-th stable segment	${\Psi }_{s_i}$:	Fitted plane of stable segment $s_i$	$f\left(p\right)$:	Segment that contains pixel p
$f_i$:	i-th segment	$d_l$:	Minimum disparity	$d_u$:	maximum disparity
ϖ:	Segment boundary pixels	${\Lambda }_l^p$:	Pixel’s potential minimum disparity	${\Lambda }_u^p$:	Pixel’s potential maximum disparity
${\Psi }_i^l$:	Minimum fitted disparity of the i-th stable segment	${\Psi }_i^u$:	Minimum fitted disparity of the i-th stable segment

3.2. Problem Formulation

In the problem formulation phase, we propose the MPTL model, which combines the complementary characteristics of stereo matching and the Xtion sensor. As shown in Figure 2, the MPTL model consists of three components:

-

The pixel-level component, which improves the robustness against the luminance variance (Section 3.3) and strengthens the smoothness of disparities between neighboring pixels and segments (Section 3.4). Nodes at this level represent reliable pixels from stable and unstable segments. The edges between reliable pixels represent different types of smoothness terms.

-

The edge that connects two level components (the SP-edge), which uses the texture variance and gradient as a guide to restrict the scope of potential disparities (Section 3.5).

-

The segment-level component, which incorporates the information from the Xtion as prior knowledge to capture the spatial structure of each stable segment and to maintain the relationship between neighboring stable segments (Section 3.6). Each node at this level represents a stable segment.

Existing global methods have achieved remarkable results, but the capability of the traditional Markov random field stereo model remains limited. To lessen the matching ambiguities, additional information is required to formulate an accurate model. In this paper, the pixel-level improved luminance consistency term ($E_l$), the pixel-level hybrid smoothness term ($E_s$) and the SP-edge texture term ($E_t$), as well as the segment-level 3D plane bias term ($E_p$) are integrated as additional regularization constraints to obtain a precise disparity estimation (D) for a scene with complex geometric characteristics. According to Bayes’ rule, the posterior probability over D given l, s, t and p is:

$$p\left(D\right|l,s,t,p)=\frac{p(l,s,t,p|D\left)p\right(D)}{p(l,s,t,p)}$$

(2)

During each optimization process, $P(l,s,t,p|D)$ is only dependent on l, s, t and p. Therefore, $P\left(D\right|l,s,t,p)$ can be rewritten as:

$$\begin{array}{cc}& p\left(D\right|l,s,t,p)\propto p(l\left|D\right)p\left(s\right|D\left)p\right(t\left|D\right)p\left(p\right|D\left)p\right(D)\hfill \end{array}$$

(3)

Because maximizing this posterior is equivalent to minimizing its negative log likelihood, our goal is to obtain the disparity map (D) that minimizes the following energy function:

$$\begin{array}{cc}& E\left(D\right)=E_l\left(D\right)+E_s\left(D\right)+E_t\left(D\right)+E_p\left(D\right)\hfill \end{array}$$

(4)

Each term will be discussed in detail in the following sections.

3.3. Improved Luminance Consistency Term

The conventional luminance consistency hypothesis is used to penalize the appearance dissimilarity between corresponding pixels in $I_L$ and $I_R$, based on the hypothesis that the surface of a 3D object is Lambertian. Because it refers to a perfectly-diffuse appearance in which pixels originating from the same 3D object have similar appearances in different views, its accuracy is heavily dependent on the lighting condition for which colors change substantially depending on the viewpoint. Furthermore, an object may appear to have different colors because different views have different sensor characteristics. In contrast, the Xtion sensor is more robust to the light condition and can be used as prior knowledge to reduce ambiguities caused by the non-Lambertian surface. Thus, the improved luminance consistency term is denoted as:

$$E_l=\sum _{p\in I_L}{\lambda }_l·(1-O\left(p\right))·[w_p^l·C(p,q)+w_p^x·X(p,q)]+O\left(p\right)·{\lambda }_o$$

(5)

where q is the matching pixel of p in the other image. $O\left(p\right)$ is the asymmetric occlusion function described in Section 3.1, and ${\lambda }_o$ is a positive penalty used to avoid maximizing the number of occluded pixels. $C(p,q)$ is defined as the pixel-wise cost function from stereo matching to measure the color dissimilarity.

$$C(p,q)=\alpha ·(1-exp(-\frac{C_{ssd}(p,q)}{r_{ssd}}))+(1-\alpha )·(1-exp(-\frac{C_g(p,q)}{r_g}))$$

(6)

where $r_{ssd}$ and $r_g$ are constant values defined by our experience. α is the scalar weight from zero to one. $C_{ssd}(p,q)$ and $C_g(p,q)$ are the color dissimilarity and gradient in three color channels as:

$$C_{ssd}(p,q)=\sqrt{\sum _{i=R,G,B}{(I_L^i\left(p\right)-I_R^i\left(q\right))}^2}$$

(7)

$$C_g(p,q)=\sqrt{\sum _{i=R,G,B}{(\nabla I_L^i\left(p\right)-\nabla I_R^i\left(q\right))}^2}$$

(8)

$X(p,q)$ is the components from the Xtion sensor, which are defined as:

$$X(p,q)=min\left\{\right|D\left(p\right)-\Pi \left(p\right)|,T_{\pi }\}$$

(9)

$T_{\pi }$ is the constant threshold, and $D\left(p\right)$ is the disparity value assigned to pixel p in each optimization. $\Pi \left(p\right)$ is the disparity of pixel $p\in \Pi$. $w_p^x$ and $w_p^l$ are pixel-wise confidence weights that are denoted as $w_p^x=1-w_p^l$. They are derived from the reliabilities of disparities obtained from stereo matching ($m_p^l$) and the Xtion ($m_p^x$) as:

$$w_p^l=\left\{\begin{array}{cc}\frac{m_p^l}{m_p^l+m_p^x}\hfill & p\in \Pi \hfill \\ 1\hfill & otherwise\hfill \end{array}\right.$$

(10)

where $m_p^l$ is similar to the attainable maximum likelihood (AML) in [31], which models the cost for each pixel using a Gaussian distribution centered at the minimum actually achieved cost value for that pixel. The reliability of Xtion data $m_p^x$ is the inverse of the normalized standard deviation of the random error [20]. The confidence of each depth value obtained from the depth sensor decreases with the increasing of the normalized standard deviation.

3.4. Hybrid Smoothness Term

The hybrid smoothness term strengthens the segmentation-based assumption that the disparity variance in each segment is smooth and reduces errors caused by under- and over-segmentation. It consists of four terms: the smoothness term for neighboring reliable pixels belonging to the unstable segment ($E_{s_0}$), the smoothness term for neighboring reliable pixels in the same stable segment ($E_{s_1}$), the smoothness term for neighboring reliable pixels in different stable segments ($E_{s_2}$) and the smoothness term for neighboring reliable pixels that belong to stable and unstable segments ($E_{s_3}$).

Because there is no prior knowledge about the spatial structure of unstable segments, we define the smoothness term $E_{s_0}$ as the conventional second-order smoothness prior Equation (11), which can produce better estimates for a scene with complex geometric characteristics [11].

$$E_{s_0}=\sum _{{\Phi }_i^0\in {\Phi }^0}{\lambda }_s^0·[1-exp(-\frac{\nabla D\left({\Phi }_i^0\right)}{{\gamma }_s})]\{p_0,p_1,p_2\}\in {\Phi }_i^0$$

(11)

where ${\gamma }_s$ and ${\lambda }_s^0$ are the geometric proximity and the positive penalty. ${\Phi }^0$ is the set of triple-cliques consisting of consecutive reliable pixels belonging to unstable segment. $\nabla D\left({\Phi }_i^0\right)$ is the second derivative of the disparity map as:

$$\nabla D\left({\Phi }_i^0\right)=D\left(p_0\right)-2D\left(p_1\right)+D\left(p_2\right).\{p_0,p_1,p_2\}\in {\Phi }_i^0$$

(12)

$E_{s_0}$ captures richer features of the local structure and permits planar surfaces without penalty by setting $\nabla D\left({\Phi }_i^0\right)=0$. However, $E_{s_0}$ only considers disparity information when representing the smoothness of neighboring pixels. This means that, in several cases, error matching can result in different disparity assignments, which correspond to the same second derivatives in the disparity map (see Figure 5b–d). Meanwhile, each stable segment can be represented as a fitted plane using the disparity data from the Xtion, which contains prior knowledge about the spatial structure of each stable segment. We can incorporate the spatial similarity weight with the prior knowledge from the Xtion into a conventional second-order smoothness prior. This term encourages constant disparity gradients for pixels in a stable segment and local spatial structures that are similar to the fitted plane of the stable segment. The smoothness term for neighboring reliable pixels in the same stable segment is as follows.

$$E_{s_1}=\sum _{{\Phi }_i^1\in {\Phi }^1}{\lambda }_s^1·[2-\delta \left({\Phi }_i^1\right)-exp(-\frac{\nabla D\left({\Phi }_i^1\right)}{{\gamma }_s})]$$

(13)

where ${\Phi }^1$ is the set of triple-cliques defined by all $3\times 1$ and $1\times 3$ consecutive reliable pixels along the coordinate direction of the rectified image coordinate in each stable segment. ${\lambda }_s^1$ is a positive value penalty. As for the spatial 3D relationship shown in Figure 5, let $s_i$ be the stable segment containing ${\Phi }_i^1$ and ${\Psi }_{s_i}$ be its corresponding fitted plane. Then, the spatial similarity weight $\delta \left({\Phi }_i^1\right)$ is denoted as:

$$\delta \left({\Phi }_i^1\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{I}:\overline{p_0{p}_1}\ne \overline{p_1{p}_2}\hfill \\ 0.25\hfill & \mathrm{II}:\overline{p_0{p}_1}=\overline{p_1{p}_2}\mathrm{and}\overline{p_0{p}_2}\cap {\Psi }_{s_i}\hfill \\ 0.5\hfill & \mathrm{III}:\overline{p_0{p}_1}=\overline{p_1{p}_2}\mathrm{and}\overline{p_0{p}_2}//{\Psi }_{s_i}\hfill \\ 1\hfill & \mathrm{IV}:\overline{p_0{p}_1}=\overline{p_1{p}_2}\mathrm{and}\overline{p_0{p}_2}\in {\Psi }_{s_i}\hfill \end{array}\right.\{p_0,p_1,p_2\}\in {\Phi }_i^1$$

(14)

-

Case I: When $\overline{p_0{p}_1}\ne \overline{p_1{p}_2}$, the disparity gradients of pixels in ${\Phi }_i^1$ are not constant ($\nabla D\left({\Phi }_i^1\right)\ne 0$). This case violates the basic segmentation assumptions that the disparity variance of neighboring pixels is smooth, so a large penalty is added to prevent it from happening in our model (see Figure 5a).

-

Cases II, III and IV: When $\overline{p_0{p}_1}=\overline{p_1{p}_2}$, the disparity gradients of pixels in ${\Phi }_i^1$ are constant ($\nabla D\left({\Phi }_i^1\right)=0$). This means that the variance of the disparities is smooth. Furthermore, our model checks the relationship between all pixels in ${\Phi }_i^1$ and ${\Psi }_{s_i}$ (see Figure 5b–d). $\delta \left({\Phi }_i^1\right)$ does not penalize the disparity assignment if all pixels in ${\Phi }_i^1$ belong to ${\Psi }_{s_i}$ (Case IV in Figure 5d), because it is reasonable to assume that the local structure of ${\Phi }_i^1$ is the same as the spatial structure of ${\Psi }_{s_i}$. Note that we impose a larger penalty to Case II than to Case III to strengthen the similarity between the spatial structure of ${\Phi }_i^1$ and ${\Psi }_{s_i}$.

Figure 5. Smoothness term for pixels in the same stable segment ($s_i$) with the spatial similarity weight as the triple-clique ${\Phi }_i^1=\{p_0,p_1,p_2\}$, under different disparity assignments. Given ${\Psi }_{s_i}$ (blue line) as the fitted plane of $s_i$. Yellow nodes are the intersect points. (a) Case I, $D\left(p0\right)\ne D\left(p1\right)\ne D\left(p2\right)$ with $\nabla D\left({\Phi }_i^1\right)\ne 0$; (b) Case II, $D\left(p0\right)=D\left(p1\right)=D\left(p2\right)$ with $\nabla D\left({\Phi }_i^1\right)=0$; (c) Case III, $D\left(p0\right)\ne D\left(p1\right)\ne D\left(p2\right)$ with $\nabla D\left({\Phi }_i^1\right)=0$; (d) Case IV, $D\left(p0\right)\ne D\left(p1\right)\ne D\left(p2\right)$ with $\nabla D\left({\Phi }_i^1\right)=0$. As shown in Cases II, III and IV, different disparity assignments correspond to the same second derivative ($\nabla D\left({\Phi }_i^1\right)=0$).

Figure 5. Smoothness term for pixels in the same stable segment ($s_i$) with the spatial similarity weight as the triple-clique ${\Phi }_i^1=\{p_0,p_1,p_2\}$, under different disparity assignments. Given ${\Psi }_{s_i}$ (blue line) as the fitted plane of $s_i$. Yellow nodes are the intersect points. (a) Case I, $D\left(p0\right)\ne D\left(p1\right)\ne D\left(p2\right)$ with $\nabla D\left({\Phi }_i^1\right)\ne 0$; (b) Case II, $D\left(p0\right)=D\left(p1\right)=D\left(p2\right)$ with $\nabla D\left({\Phi }_i^1\right)=0$; (c) Case III, $D\left(p0\right)\ne D\left(p1\right)\ne D\left(p2\right)$ with $\nabla D\left({\Phi }_i^1\right)=0$; (d) Case IV, $D\left(p0\right)\ne D\left(p1\right)\ne D\left(p2\right)$ with $\nabla D\left({\Phi }_i^1\right)=0$. As shown in Cases II, III and IV, different disparity assignments correspond to the same second derivative ($\nabla D\left({\Phi }_i^1\right)=0$).

In some segmentation-based algorithms [32], the segmentation is implemented as a hard constraint by setting ${\lambda }_s^0$ and ${\lambda }_s^1$ to be positive infinity. This does not allow any large disparity variance within a segment. In other words, each segment can only be represented as a single plane model, and the boundaries of a 3D object must be exactly aligned with segment boundaries. Unfortunately, not all segments can be accurately represented as a fitted plane, and not all 3D object boundaries coincide with segment boundaries. The accuracy of the segmentation-based algorithms is easily affected by the initial segmentation. On the one hand, the initial segmentation typically contains some under-segmented regions (where pixels from different objects, but with similar colors are grouped into one segment). As a direct consequence of under-segmentation, foreground and background boundaries are blended if they have similar colors at disparity discontinuities. To avoid this, we use segmentation as a soft constraint by setting ${\lambda }_s^0$ and ${\lambda }_s^1$ to be positive finite, so that each segment can contain arbitrary fitted planes.

On the other hand, pixels with different colors, but on the same object are over-segmented into different segments in the initial segmentation, which causes computationally inefficiency and ambiguities on segment boundaries. In this paper, we considered the spatial structure of neighboring stable segments using disparities from the Xtion. Therefore, we apply the smoothness term for neighboring pixels belonging to different stable segments ($E_{s_2}$) to avoid errors caused by the over-segmentation. Let p and q be neighboring pixels belonging to stable segments $s_i$ and $s_j$, respectively, Then, $E_{s_2}$ can be expressed as:

$$E_{s_2}=\left\{\begin{array}{cc}0\hfill & \mathrm{I}:{\Psi }_{s_i}\ne {\Psi }_{s_j}\mathrm{and}D\left(p\right)\ne D\left(q\right)\hfill \\ {\lambda }_s^2\hfill & \mathrm{II}::{\Psi }_{s_i}\ne {\Psi }_{s_j}\mathrm{and}D\left(p\right)=D\left(q\right)\hfill \\ {\lambda }_s^2\hfill & \mathrm{III}::{\Psi }_{s_i}={\Psi }_{s_j}\mathrm{and}D\left(p\right)\ne D\left(q\right)\hfill \\ 0\hfill & \mathrm{IV}:{\Psi }_{s_i}={\Psi }_{s_j}\mathrm{and}D\left(p\right)=D\left(q\right)\hfill \end{array}\right.$$

(15)

As shown in Equation (15), for Cases I and II, if ${\Psi }_{s_i}$ is not equal to ${\Psi }_{s_j}$, this means that $s_i$ and $s_j$ have different spatial structures, and the 3D object boundary coincides with the boundary between them. The disparity variance between p and q is allowed without any penalty (Case I); otherwise, a constant penalty ${\lambda }_s^2$ is added (Case II). In contrast, for Cases III and IV, if ${\Psi }_{s_i}$ is equal to ${\Psi }_{s_j}$, this means that $s_i$ and $s_j$ have different appearances, but have similar spatial structures and belong to the same 3D object. In these two cases, the disparity variance between p and q is not allowed by adding a penalty. $E_{s_2}$ reduces the ambiguities caused by over-segmentation and retains only the disparity discontinuities that are aligned with object boundaries from geometrically-smooth, but strong color gradient regions, where pixels with different colors, but from the same object are partitioned into different segments.

Because unstable segments do not have sufficient disparity information from the Xtion to regard their spatial plane models, the smoothness term for neighboring pixels that belong to the stable and unstable segments ($E_{s_3}$) encourages neighboring pixels to take the same disparity assignment. It takes the form of a standard Potts model,

$$E_{s_3}=\left\{\begin{array}{cc}0\hfill & D\left(p\right)=D\left(q\right)\hfill \\ {\lambda }_s^3\hfill & D\left(p\right)\ne D\left(q\right)\hfill \end{array}\right.p\in S,q\in \overline{S}$$

(16)

Thus, let ϖ be the set of pixels belonging to segment boundaries, the hybrid smoothness term is:

$$E_s=\left\{\begin{array}{cc}{E}_{s_0}\hfill & \{p_0,p_1,p_2\}\in {\Phi }_i\mathrm{and}{\Phi }_i\in \overline{S}\mathrm{and}{\Phi }_i\cap \varpi =\varnothing \hfill \\ E_{s_1}\hfill & \{p_0,p_1,p_2\}\in {\Phi }_i\mathrm{and}{\Phi }_i\in S\mathrm{and}{\Phi }_i\cap \varpi =\varnothing \hfill \\ E_{s_2}\hfill & \{p_0,p_1,p_2\}\in {\Phi }_i\mathrm{and}{\Phi }_i\cap \varpi \ne \varnothing \mathrm{and}\{p_0,p_1,p_2\}\in S\hfill \\ E_{s_3}\hfill & \{p_0,p_1,p_2\}\in {\Phi }_i\mathrm{and}{\Phi }_i\cap \varpi \ne \varnothing \mathrm{and}\{p_0,p_1\}\in S,\left\{p_2\right\}\in \overline{S}\hfill \end{array}\right.$$

(17)

3.5. Texture Term

Stereo matching often fails in textureless and repetitive regions, because there is not enough visual information to obtain a correspondence. However, the Xtion does not suffer from ambiguities in these regions. Therefore, the disparities from the Xtion are more reliable than those obtained from stereo matching on textureless and repetitive regions and should be closer to the range of potential disparities for pixels in these regions. In contrast, the disparities from the Xtion are susceptible to noise and problems caused by rich texture regions and have poor performance in preserving object boundaries. Therefore, the disparities obtained from stereo matching are more reliable than that of the Xtion and should be used to define the scope of potential disparities of pixels in those regions. Considering the complementary characteristics of stereo matching and the Xtion sensor, texture information can be used as a useful guide for disparities.

Figure 6. Surrounding neighborhood patch, $N_p$, for: (a) pixel p and (b) its corresponding sub-regions.

Figure 6. Surrounding neighborhood patch, $N_p$, for: (a) pixel p and (b) its corresponding sub-regions.

The texture variance and gradient are used as a cue to restrict the scope of potential disparities for pixels. This reduces errors caused by noise or outliers and makes the distribution of the disparity more compact. To do this, we first define a surrounding neighborhood patch $N_p$ (with a radius of, for example, 20 pixels) centered at each pixel $p\in I_L$, as shown in Figure 6. Considering that the annular spatial histogram is translation and rotation invariant [33], $N_p$ is evenly partitioned into four annular sub-regions. For each sub-region $N_p^i(i=0\cdots 3)$, we compute its normalized intensity 16-bin gray histogram $H_p^i=\{h_p^{(i,j)},j=0\cdots 15\}$ to represent the annular distribution density of $N_p$ as a 64-dimensional feature vector.

Finally, let $L_p$ be a 1D line segment ranging from ($p-10$) to ($p+10$) in the same row of p in $I_L$. The texture variance and gradient of p is determined by the texture dissimilarity ${\Gamma }_p$ Equation (18), using the Hamming distance Equation (19) between the annular distribution densities of p and its neighboring pixel q in $L_p$. That is,

$${\Gamma }_p=mi{n}_{(q\in L_p,q\ne p)}\sum _{i=0}^3\sum _{j=0}^{15}H(h_p^{(i,j)},h_q^{(i,j)})$$

(18)

$$H(h_p^{(i,j)},h_q^{(i,j)})=\left\{\begin{array}{c}1|h_p^{(i,j)}-h_q^{(i,j)}|\ge T_H\hfill \\ 0\mathrm{otherwise}\hfill \end{array}\right.$$

(19)

Each pixel’s disparity variance buffer (${\Omega }_p$) can be denoted as:

$${\Omega }_p=1+[1-exp(-\frac{{\Gamma }_p}{{\gamma }_H})]·\xi \xi =0.2·(d_u-d_l)$$

(20)

$d_l$ and $d_u$ are the minimum and maximum disparities. ${\Gamma }_p$ is small in the textureless and repetitive regions and is large in the rich texture regions or object boundaries. The scope of each pixel’s potential disparities [${\Lambda }_l^p$, ${\Lambda }_u^p$] is denoted as:

$${{\rm Y}}_l=max\{({\Psi }_{f\left(p\right)}^l-{\Omega }_p),d_l)\}$$

(21)

$${{\rm Y}}_u=min\{({\Psi }_{f\left(p\right)}^l+{\Omega }_p),d_u)\}$$

(22)

$$\begin{array}{cc}& {\Lambda }_l^p=\left\{\begin{array}{cc}max\{({\Theta }_l^p-{\Omega }_p),{{\rm Y}}_l\}\hfill & {\chi }_p\ge T_{\Lambda }\mathrm{and}f\left(p\right)\in S\hfill \\ {{\rm Y}}_l\hfill & {\chi }_p<T_{\Lambda }\mathrm{and}f\left(p\right)\in S\hfill \\ d_l\hfill & f\left(p\right)\in \overline{S}\hfill \end{array}\right.\hfill \end{array}$$

(23)

$$\begin{array}{cc}& {\Lambda }_u^p=\left\{\begin{array}{cc}min\{({\Theta }_u^p+{\Omega }_p),{{\rm Y}}_u\}\hfill & {\chi }_p\ge T_{\Lambda }\mathrm{and}f\left(p\right)\in S\hfill \\ {{\rm Y}}_u\hfill & {\chi }_p<T_{\Lambda }\mathrm{and}f\left(p\right)\in S\hfill \\ d_u\hfill & f\left(p\right)\in \overline{S}\hfill \end{array}\right.\hfill \end{array}$$

(24)

where ${\Theta }_l^p$ and ${\Theta }_u^p$ are the minimum and maximum disparities from the Xtion in the region centered at p in $I_L$. $f\left(p\right)$ is the segment that contains p. ${\Psi }_{f\left(p\right)}^l$ and ${\Psi }_{f\left(p\right)}^u$ are the minimum and maximum fitted disparities of $f\left(p\right)$. ${\chi }_p$ is the number of seed pixels in the region centered at p. $T_{\Lambda }$ is a positive value. As described in Equation (23), there are three cases for the definition of ${\Lambda }_l^p$:

-

When $f\left(p\right)$ is a stable segment ($f\left(p\right)\in S$) and contains sufficient seed pixels (${\chi }_p>T_{\Lambda }$), ${\Lambda }_l^p$ is equal to $max\{({\Theta }_l^p-{\Omega }_p),{{\rm Y}}_l\}$. In this case, there are enough seed pixels from the Xtion to denote a guide for the variance of disparities of p. If p is in the textureless or repetitive region, ${\Omega }_p$ is small. This indicates that stereo matching may fail in these regions, and a small search range should be used around disparities from the Xtion. In contrast, if p is in the rich textured region or object boundaries, ${\Omega }_p$ is large. This indicates that disparities from the Xtion may be susceptible to noise and problems caused by rich texture regions where disparities obtained from stereo matching are more reliable. Then, a broader search range should be used, so that we can extract better results not observed by the Xtion.

-

When $f\left(p\right)$ is a stable segment ($f\left(p\right)\in S$), but there are not enough seed pixels around p (${\chi }_p\le T_{\Lambda }$), ${\Lambda }_l^p$ is equal to ${{\rm Y}}_l$. In this case, although there are some seed pixels from the Xtion, they are not enough to represent the disparity variance around p. On the other hand, because each stable segment is viewed as a 3D fitted plane, the search range for the potential disparities is limited by the fitted disparity of $f\left(p\right)$ and the disparity variance buffer (${\Omega }_p$).

-

When $f\left(p\right)$ is an unstable segment ($f\left(p\right)\in \overline{S}$), ${\Lambda }_l^p$ is the minimum disparity ($d_l$).

Similarly, ${\Lambda }_u^p$ can be obtained in the same way. Then, the SP-edge term (which defines the scope of pixel’s potential disparities) is:

$$E_t=\left\{\begin{array}{c}0{\Lambda }_l^p\le D\left(p\right)\le {\Lambda }_u^p\hfill \\ {\lambda }_t\mathrm{otherwise}\hfill \end{array}\right.$$

(25)

3.6. 3D Plane Bias Term

This 3D plane bias term focuses on strengthening the assumption that each stable segment has a 3D plane bias. It is denoted as:

$$E_p=\sum _{s_i\in S}\sum _{p\in s_i}{\lambda }_p·min\left\{\right|D\left(p\right)-{\Psi }_{s_i}\left(p\right)|,T_p\}$$

(26)

where $D\left(p\right)$ is the assigned value of pixel p in $I_L$. ${\Psi }_{s_i}\left(p\right)$ is the plane fitted value, and $T_p$ is a threshold value. Note that for notation clarity, the traditional 3D bias assumption is a hard constraint that forbids any distinctive between $D\left(p\right)$ and ${\Psi }_{s_i}\left(p\right)$ by setting ${\lambda }_p$ to be infinite. On the contrary, our 3D plane bias term is a soft constraint that a certain distinctive between $D\left(p\right)$ and ${\Psi }_{s_i}\left(p\right)$ is allowed by setting ${\lambda }_p$ to be a finite positive value.

3.7. Optimization

The energy function defined in Equation (4) is a function of the real discrete disparity map. In this section, we describe how to optimize Equation (4) using the fusion move algorithm to obtain the disparity map $D^{*}$:

$$D^{*}=argmi{n}_{D}E\left(D\right)$$

(27)

The fusion move approach [34] is an extended approach of the $\alpha -expansion$ algorithm [35], which allows arbitrary values for each pixel in the proposed disparity map. It generates a new result by fusing the current and proposed disparity maps with the energy either decreasing or remaining constant. Let $D^c$ and $D^p$ be the current and proposed disparity maps of $I_L$. Our goal is to optimally “fuse” $D^c$ and $D^p$ to generate a new depth map $D^n$, so that the energy $E\left(D^n\right)$ is lower than $E\left(D^c\right)$. This fusion move is achieved by taking each pixel in $D^n$ from either $D^c$ or $D^p$, according to a binary indicator map B. B is the result of the graph cut-based fusion move Markov random field optimization technique. During each optimization, each pixel either keeps its current disparity value ($B\left(p\right)=0$) or changes it to proposed disparity value ($B\left(p\right)=1$). That is,

$$D^n=(1-B)·D^c+B·D^p$$

(28)

However, the fusion move is limited to optimizing the submodular binary fusion-energy functions that consist of unary and pairwise potentials. Because of the hybrid smoothness term, our binary fusion-energy functions are not submodular and cannot be directly solved using the fusion move [36]. Using the quadratic pseudo-Boolean optimization (QPBO) algorithm [37], we can obtain a partial solution for the non-submodular binary fusion-energy function by assigning either zero or one to partial pixels, and leaving the rest unassigned. The partial solution is a part of the global minimum solution, and its energy is not higher than that of the original solution. Because of the given lowest average number of unlabeled pixels, we used Quadratic Pseudo Boolean Optimization with Probing (QPBO-P) [38] and Quadratic Pseudo Boolean Optimization with Improving (QPBO-I) [39] as our fusion strategies. During the optimization, the pixel-level improved luminance consistency term ($E_l$), the SP-edge texture term ($E_t$) and the segment-level 3D plane bias term ($E_p$) are expressed as unary terms, respectively. We tackle the transformation problem of the pixel-level hybrid smoothness term ($E_s$) that contains triple-cliques using the decomposition method called Excludable Local Configuration (ELC) [40]. The essence of the ELC method is a QPBO-based transformation of a general higher-order Markov random field with binary labels into a first-order one that has the same minima as the original. It combines a new reduction with the fusion move and QPBO to approximately minimize higher-order multi-label energies. Furthermore, the new reduction technique is along the lines of the Kolmogorov-Zabih reduction that can reduce any higher-order minimization problem of Markov random fields with binary labels into an equivalent first-order problem. Each triple clique in $E_s$ is decomposed into a set of unary or pairwise terms by ELC without introducing any new variables.

The choice of the proposed disparity maps in the fusion move approach is another crucial factor for the successful use and efficiency of the fusion move. Because there is not an algorithm that can be applied to all situations, our goal is to expect all proposed disparity maps to be correct in some parts and under some parameter setting. Here, we use the following schemes to obtain all proposed disparity maps:

-

Proposal A: Uniform value-based proposal. All disparities in the proposal are assigned to a discrete disparity, in the range of $d_l$ to $d_u$.

-

Proposal B: The hierarchical belief propagation-based algorithm [41] is applied to generate proposals with different segmentation maps.

-

Proposal C: The joint disparity map and color consistency estimation method [42], which combines mutual information, a SIFT descriptor and segment-based plane-fitting techniques.

During each optimization, the result of the current fusion move is used as the initial disparity map of the next iteration.

3.8. Post-Processing

The post-processing is composed of two steps: filling occlusions and refinement. Given that p is a occluded pixel in $I_L$, a two-step method is implemented to estimate disparities of occluded pixels. If $f\left(p\right)\in S$, the fitted plane value ${\Psi }_{f\left(p\right)}\left(p\right)$ is assigned as p’s disparity. Otherwise, the disparity of p is the smaller disparity of its closet left and right seed pixels that belongs to the background.

After filling occlusions, in order to obtain an accurate disparity map and to remove ambiguities at object boundaries, the weighted joint bilateral filter with the slope depth compensation filter [43] is applied to refine the disparity map.

4. Results and Discussion

Here, a series of evaluations were performed to verify the effectiveness and accuracy of the proposed method. Results were composed of qualitative and quantitative analyses. The segmentation parameters for all experiment are the same: spatial bandwidth $=7$, color bandwidth $=6.5$, minimum region $=20$. Other parameters are presented in Table 2. They were kept constant for all experiments and were typically empirically based.

Table 2. Parameter settings for all experiments.

$T_H$	$T_{\Lambda }$	$T_{\pi }$	$T_p$	${\gamma }_H$	${\gamma }_s$	${\lambda }_t$	${\lambda }_l$	${\lambda }_o$	${\lambda }_s^0$	${\lambda }_s^1$	${\lambda }_s^2$	${\lambda }_s^3$	${\lambda }_p$
0.1	50	3	5	35	1.5	200	12	200	40	40	15	15	10

Table 2. Parameter settings for all experiments.

$T_H$	$T_{\Lambda }$	$T_{\pi }$	$T_p$	${\gamma }_H$	${\gamma }_s$	${\lambda }_t$	${\lambda }_l$	${\lambda }_o$	${\lambda }_s^0$	${\lambda }_s^1$	${\lambda }_s^2$	${\lambda }_s^3$	${\lambda }_p$
0.1	50	3	5	35	1.5	200	12	200	40	40	15	15	10

4.1. Qualitative Evaluation Using the Real-World Datasets

We performed qualitative analyses of the proposed method using real-world datasets. In all evaluations, we captured the image pairs using the system in Figure 1 and regarded the left DSLR cameras as the target to be estimated by the disparity map. Notice that all scenes contain weakly-textured and repetitive regions, as well as a non-Lambertian surface.

In order to illustrate that our method combines the complementary characteristics of the various disparity estimation methods and outperforms using the conventional stereo matching or depth sensor alone, we evaluated the qualitative quality of the disparity estimates from three stereo matching methods, the Xtion depth sensor and the proposed method using several complex indoor scenes in Figure 7. As show in Figure 7c, although the local stereo matching with fast cost volume filtering (FCVF [44]) performed well by recovering the object boundaries using a color image as a guide, it was very fragile for noise and textureless regions (such as the uniformly-colored board in the yellow rectangle of Figure 7a). In contrast, the depth values obtained from the active sensor are more accurate (see the same regions in Figure 7f). Therefore, our method overcame these problems with the improved luminance consistency term and texture term by incorporating the prior depth information from the depth sensor. As shown in Figure 7d, segmentation-based global stereo matching with second order smoothness prior (SOSP [11]) overcame some of the problems caused by the noise and outliers, but did not solve the problems caused by the over-segmentation, which led to ambiguous matching when segment boundaries did not correspond to object boundaries (green rectangle in Figure 7a). Segment tree-based stereo matching (ST [13]) blended the foreground and background in the under-segmented region (as shown in Figure 7e), where different objects with similar appearances (such as the red rectangle in Figure 7b) were grouped into a segment. Comparing to our result and those of segmentation-based methods, it is clear that the proposed hybrid smoothness term helps reduce matching ambiguities causing by over-segmentation and under-segmentation with the indication of the depth sensor. The raw data from the depth sensor were noisy and had poor performance in preserving object boundaries (blue rectangle in Figure 7f); our result is more robust in this situation and can be used to improve the performance of the depth sensor by considering the color and segmentation information from stereo matching. Based on the above, we can safely draw the conclusion that the proposed method obtains accurate depth estimation by combining the complementary characteristics of stereo matching and the depth sensor. We also tested the proposed method on other real-world scenes to verify its robustness (see Figure 8).

Figure 7. Results of the different methods applied to the real-world scenes: Dragon, Book, Plant and Tablecloth. Each column from up to down is: (a) the rectified left image; (b) the segmentation result, (c) the disparity map of FCVF [44]; (d) the disparity map of SOSP [11]; (e) the disparity map of segment tree (ST) [13]; (f) the seed image transformed from the Xtion data and (g) our result.

Figure 7. Results of the different methods applied to the real-world scenes: Dragon, Book, Plant and Tablecloth. Each column from up to down is: (a) the rectified left image; (b) the segmentation result, (c) the disparity map of FCVF [44]; (d) the disparity map of SOSP [11]; (e) the disparity map of segment tree (ST) [13]; (f) the seed image transformed from the Xtion data and (g) our result.

Figure 8. Comparative results for different real-world scenes: (a) Board; (b) Box; (c) Kola; (d) Vase; (e) Dragon and Kola; (f) Piggy. All scenes were approximately $0.5–1.5$ m from the cameras, and the maximum disparity was 107 pixels. Each scene from left to right contains the rectified left image and its associated result of our method.

Figure 8. Comparative results for different real-world scenes: (a) Board; (b) Box; (c) Kola; (d) Vase; (e) Dragon and Kola; (f) Piggy. All scenes were approximately $0.5–1.5$ m from the cameras, and the maximum disparity was 107 pixels. Each scene from left to right contains the rectified left image and its associated result of our method.

Furthermore, we implemented the post-processing processing introduced in Section 3.8 to assign valid disparities to pixels in the black regions of seed images. The seed image after assignment can be treated as the up-sampling disparity map of the target image captured by the Xtion alone. Then, we evaluated the quality of the 3D reconstruction from our method and that using only Xtion data (see Figure 9). The 3D point cloud reconstructions consist of the pixels’ image coordinates and their associated disparities in disparity space. The blue rectangles highlight some regions where our method performed well. For example, the proposed method was more effective at retaining the boundaries of Piggy and Plant (Figure 9a,c) and correctly recovered the top of the head and beard of the dragon (Figure 9b). These comparisons illustrate that the stereo matching using the the depth sensor as the prior knowledge is more effective and accurate than using stereo matching or the depth sensor alone.

Figure 9. Comparative results for 3D reconstructions. (a) Book; (b) Dragon; (c) Plant; (d) Tablecloth; (e) Box; (f) Piggy; (g) Vase; (h) Dragon and Kola. Each scene from top to bottom contains the reconstruction using our method and the result using the up-sampled disparity map captured by the Xtion depth sensor.

Figure 9. Comparative results for 3D reconstructions. (a) Book; (b) Dragon; (c) Plant; (d) Tablecloth; (e) Box; (f) Piggy; (g) Vase; (h) Dragon and Kola. Each scene from top to bottom contains the reconstruction using our method and the result using the up-sampled disparity map captured by the Xtion depth sensor.

4.2. Quantitative Evaluation Using the Middlebury Datasets

To quantitatively illustrate the validity of the proposed method, we also conducted evaluations on the Middlebury datasets [9,45] and focused on recovering the disparity map of the left image in each dataset. The evaluation is made by third-size resolution Views 1 and 5 of all image pairs. However, because there is nothing about the scanning depth information of this dataset, we used the method described in [14] to simulate the seed image transformed from the Xtion projected to View 1. This technique is based on a voting strategy and simply requires some disparity maps produced using several stereo methods [46,47,48]. Each pixel was labeled as a seed if its disparity in different maps was consistent (varied by less than a fixed threshold and was not near the intensity edge). Results on these datasets and their corresponding errors (compared with the ground truth) in non-occlusion regions are shown in Figure 10. As shown in Figure 11, our method ranks first among approximately 164 methods listed on the website [49]. It performs especially well on the Tsukuba image pairs, with minimum errors in non-occluded regions and near depth discontinuities.

Figure 10. Evaluation results on the Middlebury standard data. (a) Tsukuba; (b) Venus; (c) Teddy; (d) Cones. Each row contains (from left to right): the left image, our results, the error map (error matching pixels whose absolute disparity errors are larger than one in non-occlusion and occlusion regions are marked in black and gray), the occlusion map (occluded pixels are marked black) and the ground truth map.

Figure 10. Evaluation results on the Middlebury standard data. (a) Tsukuba; (b) Venus; (c) Teddy; (d) Cones. Each row contains (from left to right): the left image, our results, the error map (error matching pixels whose absolute disparity errors are larger than one in non-occlusion and occlusion regions are marked in black and gray), the occlusion map (occluded pixels are marked black) and the ground truth map.

Figure 11. Middlebury results of our method. All numbers are the percentage of error pixels whose absolute disparity error is larger than one. The blue number is the ranking in every column. Our method outperforms the conventional stereo matching algorithms and ranks first among approximately 164 methods according to the average of the sum of the rankings in every column (up to 20 April 2015).

Figure 11. Middlebury results of our method. All numbers are the percentage of error pixels whose absolute disparity error is larger than one. The blue number is the ranking in every column. Our method outperforms the conventional stereo matching algorithms and ranks first among approximately 164 methods according to the average of the sum of the rankings in every column (up to 20 April 2015).

On the other hand, as shown in Figure 12, we presented some evaluation results of the Middlebury extension datasets [50,51] to illustrate the robustness of the proposed method. Meanwhile, we also show the quality of 3D reconstruction of Middlebury datasets using the pixels’ image coordinate and their corresponding disparities in disparity space (see Figure 13). The evaluation results in Figure 12 and Figure 13 illustrate that our method is robust to different types of scenes and outperforms in slanted and highly curved surfaces.

Besides, we also compared our results with those produced by other “fused” schemes [20,23,52,53,54,55,56], and the compared results are listed in the Table 3. Our method provides an error rate of 2.61% on the Middlebury datasets, compared to the average error rate 3.27% of the previous state-of-the-art “fused” methods. It is clear that our method performs almost 20% better than other “fused” scheme-based algorithms in the aspect of precision. Furthermore, As shown in Figure 14 and Figure 15, our method achieves comparable results in the following aspects:

-

Noise and outliers are significantly reduced, mainly because of the improved luminance consistency term and the texture term.

-

The method obtains precise disparities for slanted or highly-curved surfaces of objects with complex geometric characteristics, mainly because of the 3D plane bias term.

-

Ambiguous matchings caused by over-segmentation or under-segmentation are overcome and disparity variances become smoother, mainly because of the hybrid smoothness term.

Figure 12. Evaluation results on the Middlebury extension datasets. (a) Wood1; (b) Cloth4; (c) Reindeer; (d) Dolls. Each row contains (left to right): the left image, our results, the error map (from top to bottom, the percentages of error pixels with absolute disparity error larger than one in non-occlusion regions are: 4.35%, 1.03%, 4.31%, 4.78%; error pixels are marked red), the occlusion map (occluded pixels are marked black) and the ground truth map.

Figure 12. Evaluation results on the Middlebury extension datasets. (a) Wood1; (b) Cloth4; (c) Reindeer; (d) Dolls. Each row contains (left to right): the left image, our results, the error map (from top to bottom, the percentages of error pixels with absolute disparity error larger than one in non-occlusion regions are: 4.35%, 1.03%, 4.31%, 4.78%; error pixels are marked red), the occlusion map (occluded pixels are marked black) and the ground truth map.

Figure 13. The results of 3D reconstructions. (a) Tsukuba; (b) Venus; (c) Teddy; (d) Cones; (e) Wood1; (f) Cloth4; (g) Reindeer; (h) Dolls.

Figure 13. The results of 3D reconstructions. (a) Tsukuba; (b) Venus; (c) Teddy; (d) Cones; (e) Wood1; (f) Cloth4; (g) Reindeer; (h) Dolls.

Table 3. The percentages of error pixels (absolute disparity error larger than 1 in non-occlusion regions) of our method and other “fused” methods on the Middlebury datasets. “Averages” are the average percentages of error pixels over all images. Compared to the average error rate 3.27% of the previous state-of-the-art “fused” methods, our method provides a lower average error rate of 2.61% on the Middlebury datasets. It performs almost $20\%$ better than other “fused” methods in the aspect of precision.

	The Percentages of Error Pixels (%)
	Tsukuba	Venus	Teddy	Conse	Wood1	Colth4	Reimdeor	Dools		Averages
Zhu et al. [20]	1.16	0.14	2.83	3.47	5.38	3.74	5.83	5.46		3.50
Wang et al. [23]	0.89	0.12	6.39	2.14	4.05	3.81	3.55	2.71		2.96
Yang et al. [52]	0.94	0.26	5.65	7.18	1.76	2.60	4.43	4.13		3.37
Jaesik et al. [53,55]	2.38	0.56	5.59	6.28	3.72	2.88	4.04	4.69		3.77
James et al. [54]	2.90	0.29	2.12	2.83	2.74	2.32	5.02	4.02		2.78
Ours	0.79	0.10	3.49	2.06	4.35	1.03	4.31	4.78		2.61

Table 3. The percentages of error pixels (absolute disparity error larger than 1 in non-occlusion regions) of our method and other “fused” methods on the Middlebury datasets. “Averages” are the average percentages of error pixels over all images. Compared to the average error rate 3.27% of the previous state-of-the-art “fused” methods, our method provides a lower average error rate of 2.61% on the Middlebury datasets. It performs almost $20\%$ better than other “fused” methods in the aspect of precision.

	The Percentages of Error Pixels (%)
	Tsukuba	Venus	Teddy	Conse	Wood1	Colth4	Reimdeor	Dools		Averages
Zhu et al. [20]	1.16	0.14	2.83	3.47	5.38	3.74	5.83	5.46		3.50
Wang et al. [23]	0.89	0.12	6.39	2.14	4.05	3.81	3.55	2.71		2.96
Yang et al. [52]	0.94	0.26	5.65	7.18	1.76	2.60	4.43	4.13		3.37
Jaesik et al. [53,55]	2.38	0.56	5.59	6.28	3.72	2.88	4.04	4.69		3.77
James et al. [54]	2.90	0.29	2.12	2.83	2.74	2.32	5.02	4.02		2.78
Ours	0.79	0.10	3.49	2.06	4.35	1.03	4.31	4.78		2.61

Figure 14. Evaluation results with the state-of-the-art “fused” scheme-based algorithms on the Middlebury datasets. (a) Tsukuba; (b) Venus; (c) Teddy; (d) Cones. Each column from top to bottom is the results obtained from: Zhu et al. [20], Wang et al. [23], Yang et al. [52], Jaesik et al. [53,55], James et al. [54] and our method. Error pixels with absolute disparity error larger than one in non-occlusion regions are marked red. The percentages of error pixels are listed in Table 3.

Figure 14. Evaluation results with the state-of-the-art “fused” scheme-based algorithms on the Middlebury datasets. (a) Tsukuba; (b) Venus; (c) Teddy; (d) Cones. Each column from top to bottom is the results obtained from: Zhu et al. [20], Wang et al. [23], Yang et al. [52], Jaesik et al. [53,55], James et al. [54] and our method. Error pixels with absolute disparity error larger than one in non-occlusion regions are marked red. The percentages of error pixels are listed in Table 3.

Figure 15. Evaluation results with the state-of-the-art “fused” scheme-based algorithms on the Middlebury extension datasets. (a) Wood1; (b) Cloth4; (c) Reindeer; (d) Dolls. Each column from top to bottom is the results obtained from: Zhu et al. [20], Wang et al. [23], Yang et al. [52], Jaesik et al. [53,55], James et al. [54] and our method. Error pixels with absolute disparity error larger than one in non-occlusion regions are marked red. The percentages of error pixels are listed in Table 3.

Figure 15. Evaluation results with the state-of-the-art “fused” scheme-based algorithms on the Middlebury extension datasets. (a) Wood1; (b) Cloth4; (c) Reindeer; (d) Dolls. Each column from top to bottom is the results obtained from: Zhu et al. [20], Wang et al. [23], Yang et al. [52], Jaesik et al. [53,55], James et al. [54] and our method. Error pixels with absolute disparity error larger than one in non-occlusion regions are marked red. The percentages of error pixels are listed in Table 3.

4.3. Evaluation Results for Each Term

We conducted evaluations to analyze the effect of the individual terms in Equation (4). In each experiment, one term was turned off and the others remained on. First, the texture term was turned off, which meant that the range of the potential disparities for each pixel was no longer restricted by the texture variance and gradient. Ambiguities occurred in textureless and repetitive texture regions without the prior restriction from the data of the depth sensor (see the yellow rectangle in Figure 16b). The average error rate of all images in non-occlusion regions sharply increased to 2.77%. Furthermore, the improved luminance consistency term was turned off by setting $w_p^x:=0$. Then, this term can be viewed as the conventional one that is easily affected by light variation and causes error matching on the non-Lambertian surface and rich texture regions (see the green rectangle regions in Figure 16c–e). The corresponding average error rate of all images in the non-occlusion regions is 2.37%. Thirdly, the hybrid smoothness term was turned out by replacing by the usual second-order smoothness term [11]. Some artifacts in the red rectangle in Figure 16f were caused by over-segmentation and under-segmentation. Its average error rate sharply increased to 3.02%. Finally, the 3D plane bias term was turned off by setting ${\lambda }_p$:=0. In that case, all 3D object surfaces are assumed as the frontal parallel ones, and the depth map is rather noisy, which makes it difficult to preserve the details at the boundary of objects (see the blue rectangle region in Figure 16g). Its average error rate is 2.14%. The corresponding error statistic analysis on the Middlebury datasets is listed in Table 4. It is clear that our method can obtain the lowest average error rate when all terms turn on (average error rate of 1.61% in non-occlusion regions).

Figure 16. Evaluation results when turning off some terms. (a) Our result; (b) result without the texture term; (c) result without the improved luminance consistency term; (d) the detail with an enlarged scale in the green region of (c); (e) our result detail with enlarged scale in the same green region of (c); (f) result without the hybrid smoothness term; (g) result without the 3D plane bias term. Nonocc: non-occlusion regions.

Figure 16. Evaluation results when turning off some terms. (a) Our result; (b) result without the texture term; (c) result without the improved luminance consistency term; (d) the detail with an enlarged scale in the green region of (c); (e) our result detail with enlarged scale in the same green region of (c); (f) result without the hybrid smoothness term; (g) result without the 3D plane bias term. Nonocc: non-occlusion regions.

Table 4. Error statistic for the Middlebury datasets with different constraint terms turned off. “Averages” are the average percentages of error pixels over all images in different regions.

	Tsukuba				Venus				Teddy				Cones				Averages
	nonocc	all	disc		nonocc	all	disc		nonocc	all	disc		nonocc	all	disc		nonocc	all	disc
Texture term off	2.04	2.12	5.78		0.81	1.03	3.09		5.43	10.2	13.25		2.79	8.26	6.35		2.77	5.40	7.12
Luminance term off	1.01	1.65	4.78		0.11	0.25	1.54		5.49	11.20	14.92		2.88	8.47	7.74		2.37	5.39	7.25
Smoothness term off	1.39	2.14	4.94		0.85	0.93	2.02		6.57	13.01	14.80		3.28	7.50	7.13		3.02	5.90	7.22
Plane bias term off	0.88	1.49	4.86		0.23	0.65	2.27		4.53	9.30	10.63		2.90	7.96	8.96		2.14	4.76	6.68
All terms on	0.79	1.21	4.30		0.10	0.21	1.27		3.49	9.04	10.90		2.06	7.05	5.80		1.61	4.37	5.56

Table 4. Error statistic for the Middlebury datasets with different constraint terms turned off. “Averages” are the average percentages of error pixels over all images in different regions.

	Tsukuba				Venus				Teddy				Cones				Averages
	nonocc	all	disc		nonocc	all	disc		nonocc	all	disc		nonocc	all	disc		nonocc	all	disc
Texture term off	2.04	2.12	5.78		0.81	1.03	3.09		5.43	10.2	13.25		2.79	8.26	6.35		2.77	5.40	7.12
Luminance term off	1.01	1.65	4.78		0.11	0.25	1.54		5.49	11.20	14.92		2.88	8.47	7.74		2.37	5.39	7.25
Smoothness term off	1.39	2.14	4.94		0.85	0.93	2.02		6.57	13.01	14.80		3.28	7.50	7.13		3.02	5.90	7.22
Plane bias term off	0.88	1.49	4.86		0.23	0.65	2.27		4.53	9.30	10.63		2.90	7.96	8.96		2.14	4.76	6.68
All terms on	0.79	1.21	4.30		0.10	0.21	1.27		3.49	9.04	10.90		2.06	7.05	5.80		1.61	4.37	5.56

4.4. Computational Time Analyses

The proposed method was implemented on a PC with Core i5-2500 3.30 GHZ CPU and 4 GB RAM. Table 5 and Table 6 list the running time of the proposed method for all experiments. It is obvious that the computational time is proportional to the image resolution and the scope of potential disparities. For example, it took approximately 1–9 mins to obtain results on Middlebury data and 19–25 mins on the real-world scene datasets. In the future, we aim to implement our method on a GPU to achieve a good balance between accuracy and efficiency.

Table 5. Running times for the real-world datasets. The disparity map resolution of all real-world datasets is $1024\times 960$. The corresponding maximum disparity is 107.

	Dragon	Book	Plant	Tableclotd	Board	Box	Kola	Vase	Piggy	Dragon and Piggy
Running Time (m):	22.15	20.54	22.16	24.31	21.51	24.26	23.43	22.30	19.44	23.04

Table 5. Running times for the real-world datasets. The disparity map resolution of all real-world datasets is $1024\times 960$. The corresponding maximum disparity is 107.

	Dragon	Book	Plant	Tableclotd	Board	Box	Kola	Vase	Piggy	Dragon and Piggy
Running Time (m):	22.15	20.54	22.16	24.31	21.51	24.26	23.43	22.30	19.44	23.04

Table 6. Running times for the Middlebury datasets.

	Tsukuba	Venus	Teddy	Cones	Wood1	Cloth4	Reindeer	Dolls
Running Time (m):	1.03	1.19	4.08	4.57	8.18	7.44	7.02	8.31
Disparity Map Resolution:	$384\times 288$	$434\times 383$	$450\times 375$	$450\times 375$	$457\times 370$	$433\times 375$	$447\times 370$	$463\times 375$
Maximum Disparity:	15	19	59	59	71	69	67	73

Table 6. Running times for the Middlebury datasets.

	Tsukuba	Venus	Teddy	Cones	Wood1	Cloth4	Reindeer	Dolls
Running Time (m):	1.03	1.19	4.08	4.57	8.18	7.44	7.02	8.31
Disparity Map Resolution:	$384\times 288$	$434\times 383$	$450\times 375$	$450\times 375$	$457\times 370$	$433\times 375$	$447\times 370$	$463\times 375$
Maximum Disparity:	15	19	59	59	71	69	67	73

5. Conclusions

In this paper, we present an accurate disparity estimation fusion model that “fused” the advantages of the complementary nature of active and passive sensors. Our main contributions are the texture information constraint and the multiscale pseudo two-layer image model. The comparison results show that our method can reduce the error estimate caused by under- or over- segmentation and has good performance in keeping object boundaries compared to using the conventional stereo matching or the depth sensor alone. Furthermore, the proposed method provides an error rate of 2.61% on the Middlebury datasets, compared to the average error rate 3.27% of the previous state-of-the-art “fused” methods. It is clear that our method performs almost 20% better than other “fused” scheme-based algorithms in the aspect of precision. In the future, we will investigate a more accurate method for estimating the disparities of occluded pixels. We also intend to transform our method to a parallel GPU implementation.