Simplified High-Performance Cost Aggregation for Stereo Matching

Featured Application

The guidance of uncrewed aerial/ground vehicles, high-end security, surveillance, and various 3D manipulation, inspection, and measurements.

Abstract

Applying edge preservation filters for cost aggregation has been a leading technique in generating dense disparity maps. However, traditional approaches usually require intensive calculations, and their design parameters must be tuned for different scenarios to obtain the best performance. This paper shows that a simple texture-independent aggregation approach can achieve similar high performance. The proposed algorithm is equivalent to a sequence of matrix multiplications involving two weighting matrices and a primary matching cost. Notably, the weighting matrices are constant for image pairs with the same resolution. For higher matching accuracy, we integrate the algorithm with a multi-scale scheme to fully exploit the spatial distribution of textures in the image pairs. The resultant hybrid approach is efficient and accurate enough to surpass most existing approaches in stereo matching. The performance of the proposed approach is verified by extensive simulation results using the Middlebury (3rd Edition) benchmark stereo database.

1. Introduction

Binocular stereo matching aims to restore 3D information based on a pair of rectified 2D images obtained from the same scene. Due to its passive and low-cost sensing characteristics, the acquired depth information may play a vital role in the guidance of uncrewed aerial/ground vehicles, high-end security, surveillance, and various 3D manipulation, inspection, and measurement applications.

Traditional stereo matching algorithms can be categorized into global and local algorithms [1], depending on the extent of information used for matching evaluation. Local methods consider only neighboring pixels for each candidate pixel, while the other methods exploit the entire image information. Traditional matching algorithms normally begin with formulating a stereo matching cost that estimates the matching degree between reference patches and target patches.

Global stereo matching algorithms require computationally demanding optimization algorithms, such as graph cuts [2] and dynamic programming [3,4], to find the disparity of each pixel. To facilitate the algorithms in real time, [5] implemented an integrated scheme combining a dynamic programming algorithm and a local algorithm using a graphics processing unit (GPU), while [6] implemented semi-global stereo matching using a field-programmable gate array (FPGA).

Unlike global algorithms, the stereo matching cost of local and non-local/semi-global algorithms is an aggregation of primary matching costs. The aggregation is normally conducted as a filtering procedure, and the resultant disparity map is obtained through the winner-take-all (WTA) strategy [1]. Since the computational complexities of local algorithms are usually smaller than that of other algorithms, they are widely used in practical applications.

Among the early efforts of using filters for cost volume aggregation in local algorithms, the matching performance of [7] was constrained by a fixed supporting window. This shortage was alleviated in [8] by introducing an adaptive window-size approach. Later, the Guided Image Filtering (GIF) model [9] was successfully implemented in [10], which demonstrates an edge-preserving advantage. Based on [9,11] further proposed a weighted guided image filter (WGIF) scheme to avoid halo artifacts and was used in [12] for disparity estimation. As the matching performance of GIF depends on the size of kernel windows, [13] proposed an adaptive guided filtering method to exclude pixels that do not belong to the same region. Besides, an iterative guided filtering approach [14] and an adaptive support weight version [15] were created to improve matching accuracy. More recently, [16] proposed weights according to structural features and filtered the matching cost volume by using the adaptive guided filtering method proposed by [13]. These are typical local stereo matching algorithms that do not aggregate matching costs outside the supporting window.

In [17], matching costs were aggregated according to tree structures individually derived from the entire image pair. Similarly, matching algorithms based on the so-called permeability filter [18] and pervasive guided image filtering [19] can effectively aggregate matching costs based on the whole image. In addition [20], integrated multi-scale information into the scheme of [19], significantly improving the matching performance. These algorithms use the full window for aggregation and are called non-local stereo matching algorithms. To solve the problem of matching ambiguity in low-texture areas and high sensitivity in high-texture areas [21], proposed the use of both the local support window and the whole image.

In addition to these traditional algorithms, disparity maps can also be computed using deep learning-based methods. These algorithms have the advantage of high matching accuracy. For instance [22], developed an automatic encoder to generate feature maps for semi-global stereo matching, and [23] implemented an unsupervised disparity estimation neural network based on the principle of disparity consistency. Besides [24], proposed a simplified independent component correlation algorithm (ICA)-based local similarity stereo matching algorithm to further improve matching accuracy in non-texture areas and boundaries. Among the deep learning-based methods, both [25,26] are typical end-to-end stereo matching networks, which realize the aggregation of matching costs through 3D convolution operations.

However, as pointed out in [27], compared with traditional stereo matching algorithms, deep learning-based stereo matching algorithms still suffer from insufficient generalization ability. Additionally, they typically require GPU-based computing resources. The current characteristics of deep learning-based methods justify continued research on traditional methods.

In most traditional local and non-local stereo matching algorithms, the weights for the matching cost aggregation depend on the texture of the image pair. This dependency restricts the possibility of sharing weights for different scenes. To improve the computational efficiency of stereo matching, this paper proposes a texture-independent aggregation method.

The main contributions of this paper are as follows:

(1)

We propose an aggregation algorithm for stereo matching that significantly simplifies computation without sacrificing matching performance. The aggregation weights can be shared between different scene images with the same resolution.

(2)

To provide a higher matching accuracy, we integrate the algorithm with a multi-scale scheme to exploit the spatial distribution of texture that can achieve improved performance with a minor increase in computational efforts.

2. Methods

2.1. Traditional Local and Non-Local Stereo Matching Algorithms

In this section, we examine two representative stereo matching algorithms proposed in [9,17], which are classified as local and non-local, respectively.

Traditional local and non-local stereo matching algorithms start with calculating a primary matching cost that reflects the degree of match between corresponding pixels on an image pair. Many measurements can be used for computing the primary matching cost. The implementation presented in this paper uses a linear combination of the truncated absolute difference of the gradient, denoted as ${\mathit{C}}_{TGD},$ and the Hamming distance of the census transform:

$$\begin{array}{l}{\mathit{C}}_{TGD}(q,d)=\min (|{\nabla }_x{\mathit{I}}_{\mathit{L}}(q)-{\nabla }_x{\mathit{I}}_{\mathit{R}}(q+d)|,2)+\min (|{\nabla }_y{\mathit{I}}_{\mathit{L}}(q)-{\nabla }_y{\mathit{I}}_{\mathit{R}}(q+d)|,2)\\ {\mathit{C}}_{Census}(q,d)=\mathrm{Ham}(T_{Census}({\mathit{I}}_{\mathit{L}}(q)),T_{Census}({\mathit{I}}_{\mathit{R}}(q+d))),\end{array}$$

(1)

where q is the location of a pixel, d is the estimated disparity of this pixel, ${\mathit{I}}_{\mathit{L}}$ and ${\mathit{I}}_{\mathit{R}}$ are the left and right images, respectively, ${\nabla }_x$ and ${\nabla }_y$ are the gradients of this pixel in the horizontal and vertical directions, respectively, $T_{Census}$ represents the Census transformation [28], and Ham is the Hamming Distance computation. Left and right images often appear different at the same location due to lighting conditions and image sensing inhomogeneity. These two matching costs are adopted because they can significantly reduce the misleading effects of pixel-level intensity variations.

To consider both matching costs, we make a linear combination of ${\mathit{C}}_{TGD}$ and ${\mathit{C}}_{Census}$ to obtain the primary matching cost, ${\mathit{C}}_0$:

$${\mathit{C}}_0(q,d)=\alpha \cdot {\mathit{C}}_{TGD}(q,d)+(1-\alpha )\cdot {\mathit{C}}_{Census}(q,d),$$

(2)

where $\alpha$ is a weighting constant. It is set to be 0.95 in the following investigations.

The primary matching cost can be further aggregated to include information from surrounding pixels. The aggregation of the primary matching cost is the most crucial step that affects the stereo matching performance. A general form of the aggregated matching cost, denoted as $\overline{\mathit{C}}(p,d)$, can be written as

$$\overline{\mathit{C}}(p,d)={\displaystyle \sum _{q\in \mathsf{\Omega }(p)}{W}_0(p,q)\cdot {\mathit{C}}_0(q,d)},$$

(3)

where $\mathsf{\Omega }(p)$ is the supporting region centered at p, and $W_0(p,q)$ is the aggregation weight. For the well-celebrated algorithms based on the weighted guided image filter, such as those proposed in [10,12,16], the aggregation weight, denoted as $W_{G\mathit{I}F}(p,q)$, can be represented in the following form:

$$W_0=W_{G\mathit{I}F}(p,q)={\displaystyle \sum _{k\in \mathsf{\Omega }(p)\cap \mathsf{\Omega }(q)}\frac{1}{|\mathsf{\Omega }(p)|\cdot |\mathsf{\Omega }(k)|}\left(1+\frac{({\mathit{I}}_{\mathit{L}}(p)-\mu ({\mathit{I}}_{\mathit{L}}(k)))\cdot ({\mathit{I}}_{\mathit{L}}(q)-\mu ({\mathit{I}}_{\mathit{L}}(k)))}{{\sigma }^2({\mathit{I}}_{\mathit{L}}(k))+\epsilon }\right)},$$

(4)

where $\mu$ and ${\sigma }^2$ represent the mean value and variance operations, respectively, and $\epsilon$ is a small constant introduced to avoid division by zero. From (4), we know that $W_{G\mathit{I}F}$ is closely related to the patterns on ${\mathit{I}}_{\mathit{L}}$. Besides, $W_{G\mathit{I}F}$ depends on the spatial distribution of the image since its value is a function of the interaction between two regions which are centered at p and q, denoted as $\mathsf{\Omega }(p)$ and $\mathsf{\Omega }(q)$.

In [17], the matching cost values are aggregated based on the minimum spanning tree (MST) being derived from the guidance image, where the aggregation weight, denoted as $W_{MST}$, is calculated as

$$W_0=W_{MST}(p,q)={\displaystyle \prod _{(u,v) \in L(p,q)}\exp (-\frac{|{\mathit{I}}_{\mathit{L}}(u)-{\mathit{I}}_{\mathit{L}}(v)|}{\beta })},$$

(5)

where $L(p,q)$ is a path determined by the MST between a pixel pair located at p and q, u and v are the coordinates of pixels in the path, and $\beta$ is a shaping parameter. It is clear from (5) that $W_{MST}$ also depends on the patterns on ${\mathit{I}}_{\mathit{L}}$.

Once we have the aggregated matching cost, the estimated disparity map, denoted as $\widehat{\mathit{D}}(p)$, can be obtained by the winner-take-all computation:

$$\widehat{\mathit{D}}(p)=\arg \underset{d}{\min } \overline{\mathit{C}}(p,d), d\in \{d_{\min },\dots ,d_{\max }\},$$

(6)

where $d_{\min }$ and $d_{\max }$ are the lower and upper bounds of disparity, assumed to be known beforehand.

Based on the above examination and derivation, we know that the aggregation weights of [9,17] are related to the texture and spatial distance of the participating pixels on an image pair. Different scene images have different aggregation weights, making the weights not sharable. This shortcoming is typical for most local and non-local stereo matching algorithms.

2.2. The Proposed Aggregation Method

We propose conducting a texture-independent cost aggregation scheme in three steps. In the first step, we consider the matching cost continuity in the horizontal direction. Then we enforce the vertical continuity of the cost in the second step. In each of these two steps, the aggregation is derived based on the minimization of an objective function. In the third step, we integrate the proposed algorithm with a cross-scale scheme to provide higher matching accuracy and efficient computation simultaneously.

2.2.1. The First Step: Horizontal Aggregation

In the first step, the proposed aggregation cost assumes the least squared difference from the primary matching cost. In addition, the cost has fewer differences with the horizontal 2 neighbors of each pixel. Based on these assumptions, we compose an objective function $J_h$ to be minimized:

$$\begin{array}{l}{J}_h={\displaystyle \sum _{p\in {\mathit{I}}_{\mathit{L}}}{[{\overline{\mathit{C}}}_1(p,d)-{\mathit{C}}_0(p,d)]}^2}+\lambda \cdot {\displaystyle \sum _{p\in {\mathit{I}}_{\mathit{L}}}{\displaystyle \sum _{q\in {\mathsf{{\rm N}}}_h(p)}{[{\overline{\mathit{C}}}_1(p,d)-{\overline{\mathit{C}}}_1(q,d)]}^2}}\\ ={\displaystyle \sum _{x=1}^M{\displaystyle \sum _{y=1}^N{[{\overline{\mathit{C}}}_1(x,y,d)-{\mathit{C}}_0(x,y,d)]}^2}}+\lambda \cdot {\displaystyle \sum _{x=1}^M{\displaystyle \sum _{y=1}^N{\displaystyle \sum _{(i,j)\in {\mathsf{{\rm N}}}_h(x,y)}{[{\overline{\mathit{C}}}_1(x,y,d)-{\overline{\mathit{C}}}_1(i,j,d)]}^2}}}\\ \hspace{1em} ={\displaystyle \sum _{x=1}^M({\displaystyle \sum _{y=1}^N{[{\overline{\mathit{C}}}_1(x,y,d)-{\mathit{C}}_0(x,y,d)]}^2}}\\ \hspace{1em} +2\lambda \cdot {\displaystyle \sum _{y=2}^{N-1}\{{[{\overline{\mathit{C}}}_1(x,y,d)-{\overline{\mathit{C}}}_1(x,y-1,d)]}^2+{[{\overline{\mathit{C}}}_1(x,y,d)-{\overline{\mathit{C}}}_1(x,y+1,d)]}^2\}}),\end{array}$$

(7)

where ${\overline{\mathit{C}}}_1$ is the proposed aggregation cost to be defined in the first step, M and N are the vertical and horizontal resolutions of ${\mathit{I}}_{\mathit{L}}$, $\lambda$ is a normalization factor, and ${\mathsf{{\rm N}}}_h$ is a set of 2 horizontal neighbors of the pixel under consideration, as depicted in Figure 1. In the figure, the coordinates of pixels are expended as $p=(x,y)$ and $q=(i,j).$ This 2-neighbor regularization term limits the abrupt changes in the horizontal direction of the proposed aggregation cost.

The aggregated matching cost in this step is constrained only in the horizontal direction, being independent in the vertical direction. We assume the objective function $J_H$ can be decomposed into M independent objective functions of the rows:

$$J_h={\displaystyle \sum _{x=1}^M{H}_x},$$

(8)

where

$$\begin{array}{l}{H}_x={\displaystyle \sum _{y=1}^N{[{\overline{\mathit{C}}}_1(x,y,d)-{\mathit{C}}_0(x,y,d)]}^2}\\ \hspace{1em} +2\lambda \cdot {\displaystyle \sum _{y=2}^{N-1}\{{[{\overline{\mathit{C}}}_1(x,y,d)-{\overline{\mathit{C}}}_1(x,y-1,d)]}^2+{[{\overline{\mathit{C}}}_1(x,y,d)-{\overline{\mathit{C}}}_1(x,y+1,d)]}^2\}.}\end{array}$$

(9)

In minimizing each of the objective function $H_x$, we may take partial derivative of it with respect to $\overline{\mathit{C}}’(x,y,d)$:

$$\frac{\partial H_x}{\partial {\overline{\mathit{C}}}_1(x,y,d)}=0,\hspace{1em} y\in \{1,2,\dots ,N\}$$

(10)

This results in the following N equations for different y:

$$\{\begin{array}{l}(1+2\lambda )\cdot {\overline{\mathit{C}}}_1(x,1,d)-2\lambda \cdot {\overline{\mathit{C}}}_1(x,2,d)={\mathit{C}}_0(x,1,d)\\ -2\lambda \cdot {\overline{\mathit{C}}}_1(x,y-1,d)+(1+4\lambda )\cdot {\overline{\mathit{C}}}_1(x,y,d)-2\lambda \cdot {\overline{\mathit{C}}}_1(x,y+1,d)={\mathit{C}}_0(x,y,d),\hspace{1em} y\in \{2,3,\dots ,N-1\}\\ -2\lambda \cdot {\overline{\mathit{C}}}_1(x,N-1,d)+(1+2\lambda )\cdot {\overline{\mathit{C}}}_1(x,N,d)={\mathit{C}}_0(x,N,d).\end{array}$$

(11)

If we define

$${\mathbf{A}}_h=\left[\begin{array}{ccccc}1+2\lambda & -2\lambda & & & 0\\ -2\lambda & 1+4\lambda & -2\lambda & & \\ & -2\lambda & \ddots & \ddots & \\ & & \ddots & 1+4\lambda & -2\lambda \\ 0& & & -2\lambda & 1+2\lambda \end{array}\right], B(x)=\left[\begin{array}{c}{\overline{\mathit{C}}}_1(x,1,d)\\ ⋮\\ {\overline{\mathit{C}}}_1(x,N,d)\end{array}\right], D(x)=\left[\begin{array}{c}{\mathit{C}}_0(x,1,d)\\ ⋮\\ {\mathit{C}}_0(x,N,d)\end{array}\right],$$

(12)

Equation (11) can be written in a matrix form as

$$A_h\cdot B(x)=D(x)$$

(13)

Besides, with (12) we observe that:

$$\begin{array}{l}{\overline{\mathit{C}}}_1=\left[\begin{array}{cccc}{\overline{\mathit{C}}}_1(1,1,d)& {\overline{\mathit{C}}}_1(1,2,d)& \cdots & {\overline{\mathit{C}}}_1(1,N,d)\\ {\overline{\mathit{C}}}_1(2,1,d)& {\overline{\mathit{C}}}_1(2,2,d)& \cdots & {\overline{\mathit{C}}}_1(2,N,d)\\ ⋮& ⋮& ⋮& ⋮\\ {\overline{\mathit{C}}}_1(M,1,d)& {\overline{\mathit{C}}}_1(M,2,d)& \cdots & {\overline{\mathit{C}}}_1(M,N,d)\end{array}\right]=\left[\begin{array}{c}B{(1)}^T\\ B{(2)}^T\\ ⋮\\ B{(M)}^T\end{array}\right],\hspace{1em}\\ \\ {\mathit{C}}_0=\left[\begin{array}{cccc}{\mathit{C}}_0(1,1,d)& {\mathit{C}}_0(1,2,d)& \cdots & {\mathit{C}}_0(1,N,d)\\ {\mathit{C}}_0(2,1,d)& {\mathit{C}}_0(2,2,d)& \cdots & {\mathit{C}}_0(2,N,d)\\ ⋮& ⋮& ⋮& ⋮\\ {\mathit{C}}_0(M,1,d)& {\mathit{C}}_0(M,2,d)& \cdots & {\mathit{C}}_0(M,N,d)\end{array}\right]=\left[\begin{array}{c}D{(1)}^T\\ D{(2)}^T\\ ⋮\\ D{(M)}^T\end{array}\right],\end{array}$$

(14)

Taking the whole image into consideration, (13) becomes

$${\overline{\mathit{C}}}_1\cdot A_h^T={\mathit{C}}_0$$

(15)

The matrix $A_h$ is tridiagonal, symmetric, and sparse. This N-by-N matrix is invertible for positive $\lambda$ [29]. From (15), we may obtain the proposed aggregated matching cost, ${\overline{\mathit{C}}}_1$, by inverting $A_h$:

$${\overline{\mathit{C}}}_1^T=A_h^{-1}\cdot {\mathit{C}}_0^T$$

(16)

This procedure has the advantage that the matching cost can be calculated using a simple linear combination with constant weights while the resultant matching cost has a small difference with the horizontal neighborhoods, and only a smooth transition in this direction is possible.

2.2.2. The Second Step: Vertical Aggregation

Similar to the first step, we can create an objective function, denoted as $J_v$, for an aggregated matching cost, ${\overline{\mathit{C}}}_2.$ The function applies constraints in the vertical direction while enforces its least squared difference from the matching cost aggregated in the first step, ${\overline{\mathit{C}}}_1$:

$$\begin{array}{l}{J}_v={\displaystyle \sum _{y=1}^N\{{\displaystyle \sum _{x=1}^M{[{\overline{\mathit{C}}}_2(x,y,d)-{\overline{\mathit{C}}}_1(x,y,d)]}^2}}\\ \hspace{1em} +2\lambda \cdot {\displaystyle \sum _{x=2}^{M-1}\{{[{\overline{\mathit{C}}}_2(x,y,d)-{\overline{\mathit{C}}}_2(x-1,y,d)]}^2+{[{\overline{\mathit{C}}}_2(x,y,d)-{\overline{\mathit{C}}}_2(x+1,y,d)]}^2\}}\}\\ \hspace{1em} \equiv {\displaystyle \sum _{y=1}^N{V}_y}.\end{array}$$

(17)

We assume the objective function is composed of sub-functions $V_y$, which are mutually independent in the horizontal direction. ${\overline{\mathit{C}}}_2$ can be solved by taking a partial derivative of $V_y$ with respect to ${\overline{\mathit{C}}}_2$ in each column. By accumulating the results through the whole image, we have

$${\overline{\mathit{C}}}_2=A_v^{-1}\cdot {\overline{\mathit{C}}}_1$$

(18)

where

$$A_v=\left[\begin{array}{ccccc}1+2\lambda & -2\lambda & & & 0\\ -2\lambda & 1+4\lambda & -2\lambda & & \\ & -2\lambda & \ddots & \ddots & \\ & & \ddots & 1+4\lambda & -2\lambda \\ 0& & & -2\lambda & 1+2\lambda \end{array}\right]$$

(19)

Again, the matrix $A_v$ is an M-by-M invertible tridiagonal sparse matrix for positive $\lambda$ [29]. Combining (18) and (16), and considering that $A_h$ is symmetric, we have

$${\overline{\mathit{C}}}_2=A_v^{-1}\cdot {\overline{\mathit{C}}}_1=A_v^{-1}\cdot {\mathit{C}}_0\cdot {(A_h^{-1})}^T=A_v^{-1}\cdot {\mathit{C}}_0\cdot A_h^{-1}.$$

(20)

From (20), we know that the aggregation of the primary matching cost can be simplified as a series of matrix multiplication. The horizontal aggregation weight, $A_h$, and the vertical aggregation weight, $A_v$, are constant tridiagonal matrices of the parameter $\lambda$. Their inverses can be calculated beforehand once the image sizes, M and N, are known. Under this condition, the computational complexity of the proposed algorithm is much simpler than most aggregation approaches.

Taking M = 480, N = 720, and $\lambda =7$ as an example, Figure 2 and Figure 3 show visualized representations of $A_v^{-1}$ in two and three dimensions. It can be seen from these figures that the values of the matrix are mostly concentrated near the diagonal. It can be clearly seen that the values of the points on both sides of the diagonal gradually decrease as the distance from the diagonal increases. The behavior of $A_h^{-1}$ is similar to that of $A_v^{-1}$.

As $A_v^{-1}$ is symmetric and sparse, its element values in the row and column directions follow the same pattern. For a fixed image size, $A_v^{-1}$ is completely defined by the only parameter $\lambda$. Figure 4 demonstrates the element values of $A_v^{-1}$ as a function of $\lambda$. The image size is also 480-by-720, and only elements on row 240 are depicted.

Also, as observed in Figure 4, the distribution of the sparse matrix becomes smoother when $\lambda$ is larger, which implies a larger effective range of the aggregation. When $\lambda$ is smaller, the distribution of the sparse matrix is sharper, and the effective range of the matching cost aggregation is smaller. This observation is consistent with (7) and (17) that, when the value of $\lambda$ is larger, the objective functions, $J_h$ and $J_v$, have stronger constraints on the aggregated matching costs, ${\overline{\mathit{C}}}_1$ and ${\overline{\mathit{C}}}_2$.

From the above analysis, we have gathered that the effective range of $A_h^{-1}$ and $A_v^{-1}$ is similar to the size of the aggregation window or the length of an aggregation path in the local and non-local algorithms. For images with larger resolutions, a larger value of $\lambda$ is required to effectively aggregate the matching cost in a broader range. For images with smaller resolutions, a smaller value of $\lambda$ is more adequate.

2.2.3. The Proposed Scheme: Integration of the Texture-Independent Scheme with a Cross-Scale Cost Aggregation Algorithm

To enhance the matching accuracy of the cost aggregation algorithm derived in the last section, we present its integration with a cross-scale scheme of [20,30] in this section. This hybrid approach is recommended since higher matching accuracy can be achieved with only a slight increase in computation. Similarly, other algorithms can be integrated with the proposed scheme.

The cross-scale scheme of [20,30], denoted as CS, considers the primary matching costs of various scales. The scheme begins with down-sampling an image pair into K pairs. For each image pair, a matching cost is calculated according to (20). We denote the matching cost of the k-th pair as ${\tilde{\mathit{C}}}_k(p,d)$, $k\in \{0,1, \dots ,K\}$, where K is the highest pair with the lowest resolution. The framework is derived by introducing an objective function for the cross-scale scheme of [20,30]:

$$J_{\mathit{C}S}={\displaystyle \sum _{k=0}^K{\displaystyle \sum _{q\in \mathsf{\Omega }(p)}{[{\widehat{\mathit{C}}}_k(p,d)-{\tilde{\mathit{C}}}_k(q,d)]}^2}}+{\displaystyle \sum _{z=1}^K{\gamma }^z\cdot {[{\widehat{\mathit{C}}}_k(p,d)-{\widehat{\mathit{C}}}_{k-1}(p,d)]}^2}$$

(21)

where $\gamma$ is a constant constraining factor and $\mathsf{\Omega }(p)$ is the aggregation window centered at p. For K = 2 and $\gamma$ = 1.5, minimizing (21) with respect to ${\widehat{\mathit{C}}}_k$ results in a simple representation that the aggregated matching cost of the zero-layer is

$${\widehat{\mathit{C}}}_0(q,d)=0.56\cdot \frac{{\displaystyle \sum _{q\in \mathsf{\Omega }(p)}{\tilde{\mathit{C}}}_0(q,d)}}{|\mathsf{\Omega }(p)|}+0.26\cdot \frac{{\displaystyle \sum _{q\in \mathsf{\Omega }(p)}{\tilde{\mathit{C}}}_1(q,d)}}{|\mathsf{\Omega }(p)|}+0.18\cdot \frac{{\displaystyle \sum _{q\in \mathsf{\Omega }(p)}{\tilde{\mathit{C}}}_2(q,d)}}{|\mathsf{\Omega }(p)|}$$

(22)

where $|\mathsf{\Omega }(p)|$ is the number of pixels within the aggregation window $\mathsf{\Omega }(p)$. As can be observed in the original scheme [20,30], the larger the value of K, the more multi-scale features the aggregated matching cost contains. The larger the value of $\gamma$, the larger the proportion of down-sampled information in the aggregation matching cost, and vice versa.

A primary disparity map $\widehat{\mathit{D}}$ can be obtained using the winner-take-all principle:

$$\widehat{\mathit{D}}(x,y)=\arg \underset{d\in \{d_{\min },d_{\max }\}}{\min }{\widehat{\mathit{C}}}_0(x,y,d).$$

(23)

To reduce the noise effects on $\widehat{\mathit{D}}$, we create an updated cost volume based on the primary disparity map of (23):

$$\overline{\mathit{C}}’(x,y,d)= \{\begin{array}{l}|\widehat{\mathit{D}}(x,y)-d|, if (x,y)\notin S_p\\ 0, if (x,y)\in S_p\end{array} ,$$

(24)

where $S_p$ is the erroneous region detected via the left-right consistency examination. Implementation details of (21), (22), and (24) can be found in [20]. Finally, we aggregate the updated cost volume using the algorithm of (20):

$$\overline{\mathit{C}}’’=A_v^{-1}\cdot \overline{\mathit{C}}’\cdot A_h^{-1}$$

(25)

This is the final aggregated matching cost. We can then obtain the final disparity map by applying the winner-take-all computational principle to $\overline{\mathit{C}}’’$ once again.

3. Results

To investigate the matching performance of the proposed scheme, performance comparisons have been made between eight representative stereo matching schemes and the proposed scheme:

• End-to-end real-time stereo matching network proposed in [25], denoted as RTSMNet.
• Matching algorithm based on a combination of the adaptive support weight with iterative guided filter and the sum of gradient matching [15], denoted as ISM.
• Sparse representation over a learned discriminative dictionary for stereo matching [4], denoted as DDL.
• Stereo matching algorithm based on two-phase adaptive optimization of ad-census and gradient fusion [8], denoted as TPAO.
• Stereo matching algorithm based on per pixel difference adjustment, iterative guided filter, and graph segmentation [14], denoted as IGF.
• Local stereo matching using adaptive cross-region-based guided image filtering with orthogonal weights [16], denoted as ACR-GIF-OW.
• Hierarchical guided-image-filtering for stereo matching [20], denoted as HGIF.
• Stereo matching with fusing adaptive support weights [21], denoted as FASW.
• The proposed scheme, which is an integration with a cross-scale cost aggregation algorithm as described in the last section.

We selected several stereo image pairs from the Middlebury version 3 [31] and the KITTI Vision Benchmark Suite [32] datasets for demonstration. The “trainingQ” of the Middlebury version 3 [31] dataset is composed of 15 groups of pictures from “Adirondack” to “Vintage”. As summarized in

Table 1. Resolution of the images for stereo matching.

Resolution	Adirondack	ArtL	Jadeplant	Motorcycle	MotorcycleE	Piano	PianoL	Pipes
M	496	277	497	497	497	481	481	485
N	718	347	659	741	741	707	707	735
Resolution	Playroom	Playtable	PlaytableP	Recycle	Shelves	Teddy	Vintage
M	476	463	462	486	497	450	480
N	699	680	681	720	738	375	722

, resolutions of the images are around 480-by-720.

According to the previous section’s discussion, the design parameter is positively related to the image resolution under matching. In the following demonstrations, the values of $\lambda$ are simply assigned as

$$\lambda =\frac{M}{480}\cdot \frac{N}{720}\cdot 6$$

(26)

Figure 5 shows the disparity maps obtained by these stereo matching algorithms on four of the image sets.

Table 2. Comparison of the error rates in the non-occluded region without refinements using different matching algorithms (%). In calculating the error rates, the error threshold = 1.0.

Test Sets	RTSMNet	ISM	DDL	TPAO	IGF	ACR-GIF-OW	HGIF	FASW	Proposed
Adirondack	4.5	15.5	6.5	7.1	14.9	7.9	5.7	6.8	4.8
ArtL	11.5	16.5	13.1	13.2	15.9	10.1	10.9	10.5	9.7
Jadeplant	43.0	25.1	18.8	16.1	24.2	17.4	19.7	19.0	17.4
Motorcycle	8.9	10.0	9.0	6.8	9.6	6.8	8.5	6.8	7.6
MotorcycleE	8.9	11.0	7.7	5.9	10.5	6.8	8.5	6.3	6.7
Piano	13.3	23.5	14.8	17.9	24.1	19.1	14.0	13.8	14.3
PianoL	25.2	32.6	26.6	26.7	32.1	34.2	29.3	25.8	27.2
Pipes	13.8	15.5	11.6	10.1	15.1	9.4	9.9	9.4	9.1
Playroom	18.0	21.6	16.2	17.6	21.5	18.7	15.2	16.7	16.3
Playtable	10.6	39.6	18.4	27.3	39.6	24.3	16.7	20.8	16.6
PlaytableP	6.7	20.9	10.6	12.5	20.9	11.4	11.0	10.2	10.3
Recycle	6.8	15.0	9.1	9.1	14.4	9.5	8.3	8.1	8.7
Shelves	22.2	34.4	37.8	38.6	36.0	38.4	31.6	32.6	33.6
Teddy	9.4	7.2	6.2	6.2	7.0	6.1	5.6	5.2	6.3
Vintage	39.3	34.4	27.3	27.5	34.6	25.6	27.0	23.9	26.5
Weighted Average	14.7	19.3	13.6	13.9	19.1	14.0	12.9	12.5	12.4

The best score for each image set is marked in bold.

and

Table 3. Comparison of the error rates in the all-region without refinement (%). In calculating the error rates, the error threshold = 1.0. The best score for each image set is highlighted in bold.

Test Sets	RTSMNet	ISM	DDL	TPAO	IGF	ACR-GIF-OW	HGIF	FASW	Proposed
Adirondack	6.2	18.4	8.8	11.7	17.6	12.5	8.0	8.7	7.1
ArtL	15.5	25.6	22.8	24.5	24.8	23.4	21.2	20.8	20.7
Jadeplant	47.8	36.8	32.4	28.4	35.9	31.3	33.0	32.3	31.8
Motorcycle	11.7	14.9	13.4	14.1	14.5	14.9	12.6	10.9	11.9
MotorcycleE	11.7	15.9	11.8	13.1	15.3	14.6	12.9	10.5	11.2
Piano	16.8	27.6	19.2	22.9	28.1	23.8	18.1	18.2	18.4
PianoL	28.4	36.3	30.5	30.9	35.7	37.9	33.1	29.5	30.7
Pipes	21.7	26.2	22.7	22.8	25.7	21.8	21.5	20.2	20.7
Playroom	25.0	29.7	25.1	26.7	29.4	28.4	23.8	25.5	25.0
Playtable	13.6	42.9	23.3	33.1	42.9	30.5	21.2	25.0	22.6
PlaytableP	8.8	25.9	14.3	19.2	25.9	18.5	14.3	13.4	14.4
Recycle	7.8	17.5	11.4	12.7	17.0	13.6	10.3	10.2	11.2
Shelves	23.7	35.5	39.1	39.9	37.0	40.3	32.6	34.0	34.7
Teddy	11.2	12.0	11.8	12.7	11.8	12.3	11.2	10.9	11.7
Vintage	41.1	37.7	31.5	32.2	37.9	30.1	31.0	28.7	30.8
Weighted Average	18.0	24.9	19.5	21.1	24.6	21.6	18.7	18.2	18.4

The best score for each image set is marked in bold.

present the error rates and weighted error rates of the algorithms using the complete “trainingQ” of the Middlebury version 3 [31] dataset. In the tables, the experimental results of the four algorithms for comparison are obtained from the original literature. For ease of viewing, the graphical representations of Table 2 and Table 3 are shown in Figure 6 and Figure 7.

The performance of FASW [21] and the proposed algorithm are more prominent for stereo matching on the dataset. Specifically, in the non-occluded region, the proposed algorithm has the lowest mismatch rate, followed by FASW [21] and the HGIF [20] algorithm. In the all-region, the RTSMNet [25] algorithm has the lowest mismatch rate, followed by FASW [21] and the proposed algorithm.

However, we can observe a significant performance deterioration in using RTSMNet [25] to predict the disparity maps of the “Jadeplant” and “Vintage” image sets. RTSMNet [25] is a deep learning-based method, its generalization ability depends on the quality and richness of the training dataset. Similar scenes may be rare in its training set. This behavior is not observed in traditional algorithms because their performance is less sensitive to scene type.

The time required for the matching of a rectified stereo image pair for each algorithm is summarized in

Table 4. Comparison of the running time of each algorithm for a pair of rectified images (s). These computations were executed in MATLAB 2017b using an Intel Core I5 8300H and 16 GB RAM.

Test Sets	ACR-GIF-OW	HGIF	Proposed
Adirondack	215	31	10
ArtL	34	7	1
Jadeplant	374	87	21
Motorcycle	191	37	10
MotorcycleE	192	36	10
Piano	169	27	7
PianoL	171	26	7
Pipes	197	37	11
Playroom	211	31	11
Playtable	169	29	9
PlaytableP	167	30	9
Recycle	199	30	8
Shelves	195	29	8
Teddy	68	14	4
Vintage	499	98	25

The best score for each image set is marked in bold.

. These computations were executed in MATLAB 2017b using an Intel Core I5 8300 H and 16 GB RAM. Notably, the proposed algorithm takes the least computational time. If we consider both matching accuracy and algorithmic complexity, the proposed algorithm is close to the best accuracy while requiring the least amount of computation.

To verify the stereo matching performance of the proposed algorithm in real scenes, the autonomous driving training dataset of KITTI Vision Benchmark Suite [32] is used for further demonstration. The dataset contains 194 image pairs with corresponding ground truth disparity maps. Performance comparisons to be presented are between the following stereo matching schemes:

• Stereo matching based on adaptive guided filtering [13], denoted as AGF.
• Adaptive stereo matching using tree filtering [17], denoted as MST.
• The ISM algorithm [15].
• The HGIF algorithm [20].
• The FASW algorithm [21].
• The proposed scheme.

Figure 8 shows the disparity maps generated by these algorithms on the KITTI suite [32]. Each algorithm performs stereo matching on 194 sets of stereo image pairs. Due to space constraints, we only select three stereo-image pairs for visual comparison.

Table 5. Comparison of the effects of each algorithm using the KITTI image dataset.

Algorithms	Non-Occ (%)	All (%)	Non-Occ (Pixels)	All (Pixels)
ISM	8.88	10.01	1.87	2.07
AGF	8.59	9.73	1.77	1.99
MST	23.27	24.41	3.47	4.15
HGIF	6.57	7.78	1.35	1.61
FASW	6.89	8.12	1.31	1.45
Proposed	6.30	7.48	1.30	1.58

Non-Occ (%): Percentage of erroneous pixels in non-occluded areas, All (%): Percentage of erroneous pixels in all regions, Non-Occ (pixels): Average disparity error in terms of pixel numbers in non-occluded areas, and All (pixels): Average disparity error in terms of pixel numbers in all regions. The best score for each image set is marked in bold.

shows the quantitative matching performance of these algorithms. Four scores are compared: the percentage of erroneous pixels in non-occluded areas, denoted as Non-Occ (%); the percentage of erroneous pixels in all regions, denoted as All (%); the average disparity error in terms of pixel numbers in non-occluded areas, denoted as Non-Occ (pixels); and the average disparity error in terms of pixel numbers in all regions, denoted as All (pixels). The scores used for comparison are quoted from [21].

As seen from Figure 8 and Table 5, compared with the five representative aggregation methods, the proposed algorithm outperforms other algorithms in items Non-Occ (%), All (%), and Non-Occ (pixels), with scores of 6.30%, 7.48%, and 1.3 pixels, respectively. Its performance is only slightly inferior to [21] on All (pixels), with a score of 1.58, which is higher than [21]’s 1.45.

Figure 9 shows three disparity maps generated by the proposed algorithm using three sets of multi-view remote sensing images (pictures from https://github.com/whuwuteng/benchmark_ISPRS2021, accessed on 15 February 2019). These additional disparity maps demonstrate the effectiveness of the proposed approach.

4. Discussion

Traditional local and non-local stereo matching algorithms require an efficient aggregation procedure, which is the most critical stage for generating accurate dense disparity maps. The aggregation weights in most stereo matching algorithms in the literature, unfortunately, are scenario specific.

We re-examined the procedure of cost aggregation from the perspective of matrix operations and treated the aggregation as constraining the degree of difference between adjacent costs. By decoupling the aggregation procedure in the horizontal and vertical directions, we propose a new aggregation algorithm to effectively calculate a cost volume for stereo matching. This process is equivalent to multiplying the initial cost by two constant matrices.

The aggregation algorithm requires two constant weight matrices that are only related to the image resolution and can be calculated beforehand. These matrices are mathematically proven to be independent of image texture and thus can be applied to different scene images of the same resolution. This algorithm can be used for integration with other schemes to provide both computational efficiency and stereo matching accuracy. We demonstrate its integration with the cross-scale scheme of [20,30].

Through numerical experiments using indoor and outdoor benchmark stereo image datasets, we demonstrate that the integrated scheme is not only computationally efficient but also provides disparity maps that are highly comparable to the most accurate algorithms in the literature.

In the future, we will focus on real-time implementation of the proposed algorithm by using, for instance, GPU-based systems. Preprocessing techniques, such as the algorithm proposed in [33] for detecting and removing shadows, can be employed to further improve the accuracy of dense disparity maps.