A Global Calibration Method for Widely Distributed Cameras Based on Vanishing Features

Abstract

This paper presents a global calibration method for widely distributed vision sensors in ring-topologies. Planar target with two mutually orthogonal groups of parallel lines is needed for each camera. Firstly, the relative pose of each camera and its corresponding target is found from the vanishing points and lines. Next, an auxiliary camera is used to find the relative poses between neighboring pairs of calibration targets. Then the relative pose from each target to the reference target is initialized by the chain of transformations, followed by nonlinear optimization based on the constraint of ring-topologies. Lastly, the relative poses between the cameras are found from the relative poses of calibration targets. Synthetic data, simulation images and real experiments all demonstrate that the proposed method is reliable and accurate. The accumulated error due to multiple coordinate transformations can be adjusted effectively by the proposed method. In real experiment, eight targets are located in an area about 1200 mm × 1200 mm. The accuracy of the proposed method is about 0.465 mm when the times of coordinate transformations reach a maximum. The proposed method is simple and can be applied to different camera configurations.

1. Introduction

Vision sensors have advantages of flexibility and high-precision. Distributed vision sensors (DVS) are always used due to their wider fields of view (FOVs). Calibration is an important step for most camera applications. Calibration of a DVS typically has two stages: the intrinsic calibration which can be done separately and the global calibration which calculates the relative poses of the camera frames and the global coordinate frame (GCF). Then the information extracted from each camera can be integrated into the GCF. Generally, the coordinate frame of the reference camera is selected as GCF. However, vision sensors are usually widely distributed to have a better coverage. As two adjacent cameras usually have small or no overlapping FOV, the global calibration of DVS becomes of prime importance.

DVS can be calibrated by high-precision 3D measurement equipment. Lu et al. [1] constructed a measurement system and achieved calibration of non-overlapping DVS using two theodolites and a planar target. Calibration methods without expensive equipment have also been investigated. Peng et al. [2] proposed an approach omitting translation vectors between cameras, due to the loss of depth information during the camera projection [3]. It assumes that all the cameras have approximately the same optical center. However, this assumption is not appropriate when the relative distances between cameras are not small in comparison with the distances to the captured scene. In addition, feature detection and matching, such as scale invariant feature transform (SIFT) [4] is not reliable in insufficient textural environments due to lack of enough distinctive feature points [2].

Global calibration methods for DVS with overlapping FOV cannot be applied in the case of non-overlapping FOV. Most of the global calibration methods for DVS with non-overlapping FOV are based on mirror reflections [5,6,7], rigidity constraints of calibration targets [8,9,10], movements of the platform [11] and the auxiliary camera [12]. For general distributed vision sensors, it is hard to make sure that each camera has a clear sight of the targets through mirror reflections, especially in complex environments. Liu et al. [9] proposed a global calibration method by placing multiple targets in front of the vision sensors at least four times. Bosch et al. [10] use a poster to determine the relative poses of multiple cameras in two steps. It requires that different parts of the poster are observed by at least two cameras at the same time, so that SIFT features can be utilized. However, it is not flexible to use a long one-dimensional target [8], rigidly-connected targets [9] or a large area poster [10] for the calibration of widely distributed cameras. Pagel [11] achieved extrinsic calibration of a multi-camera rig with non-overlapping FOV by moving the platform. However, it required that at least two adjacent targets be visible and two cameras can see the target at the same time. Sun et al. [12] used an auxiliary camera to observe all the sphere targets. However, all the targets can hardly been observed by one camera at the same time due to widely distribution of the vision sensors.

Structure from motion [13] solves similar problems as that in the global calibration of DVS. The difference is that the global calibration transforms local coordinate frames into GCF, while structure from motion estimates the locations of 3D points from multiple images [14]. Fitzagibbon et al. [15] recover the 3D scene structure together with 3D camera positions from a closed image sequence. Compared with open sequences, the closed image sequence contains additional constraints. Zhang et al. [16] propose an incremental motion estimation algorithm to deal with long image sequences.

Generally, one calibration target is selected as the reference target. Compared with employing the auxiliary camera to capture all the targets in one image, capturing neighboring pairs of targets is more suitable for widely distributed cameras. The relative poses between neighboring pairs of the targets can be solved separately. Then the relative poses between each target and the reference target can be achieved by chainwise coordinate transformations. However, the error accumulates with increasing times of transformations. When dealing with DVS that provides a vision of the surrounding scene just as [2,11], vision sensors are always configured in ring-topologies to have a better coverage of the surroundings. The first sensor adjoins the end to form a closed chain. Thus a closed image sequence of neighboring targets can be acquired by the auxiliary camera.

Line features are more stable than point features in detection and matching [17]. The principle of perspective projection indicates that an infinite scene line is mapped onto an image plane as a line terminating in a vanishing point. Vanishing points and vanishing lines are the distinguish features of perspective projection [18]. Xu et al. [19] proposed a pose estimation method based on vanishing lines of a T-shaped target. Wang [20] used a target with three equally spaced parallel lines to estimate the rotation matrix by moving the target into at least three different positions. Wei et al. [21,22] calibrated a line-structured vision sensor and a binocular vision sensor by a planar target with several parallel lines. Two mutually orthogonal groups of parallel lines are common in urban environments, such as crossroads, facades of architectures and so on. They can be used as the calibration targets. Even if they are absent in the scene, targets with two mutually orthogonal groups of parallel lines can be employed.

In this paper, we focus on the calibration of widely distributed cameras in ring-topologies. A planar target with two mutually orthogonal groups of parallel lines is allocated to each camera. The vanishing line of the target plane is obtained from two vanishing points. Then the relative pose between each camera and its corresponding target is initialized and refined based on vanishing features and the known line length. A closed image sequence of neighboring pairs of calibration targets is acquired by repeated operations of the auxiliary camera. Then the relative poses between two adjacent targets can be obtained and the transformation matrix from each target to the reference target is initialized in a chainwise manner. In order to adjust the accumulated error due to the chain of transformations, a global calibration method is proposed to optimize relative poses of the targets based on the constraint of the ring-type structure. Finally, using the targets as media, the optimal relative poses between each camera and the reference camera are obtained.

The rest of the paper is organized as follows: preliminary work is introduced in Section 2. The proposed global calibration method is described in Section 3. Accuracy analysis of different factors’ effects is given in Section 4. Synthetic data, simulation images and real data experiments are carried out in Section 5. The conclusions are given in Section 6.

2. Preliminaries

2.1. Coordinate Frame Definition

In this paper, the camera coordinate frame is used as the vision sensor coordinate frame. Assuming DVS consists of M cameras, C_kCF (1 ≤ k ≤ M) denotes the coordinate frame of camera k. A_iCF (1 ≤ i ≤ M) denotes the coordinate frame of the auxiliary camera that captures two adjacent targets (i, j). The origins of CCF and ACF are fixed at the optical centers, respectively. I_kCF (1 ≤ k ≤ M) denotes the image coordinate frame of camera k in pixels. The origin of ICF is fixed at the center of the image plane.

As shown in Figure 1, the target is constructed of two mutually orthogonal groups of parallel lines with a known length L₁ and the distance L₂. $P_m^k$ and $l_m^k$ denote the mth corner point and the mth feature line of target k, 1 ≤ m ≤ 6. T_kCF (1 ≤ k ≤ M) denotes the coordinate frame of target k. $l_6^k$ and $l_1^k$ coincide with the x-axis and the z-axis of the target, respectively. The y-axis is decided by the right hand rule. ECF represents the ground coordinate frame. The origin of ECF is fixed on the ground. Plane x_eo_ez_e lies in the ground plane. The y-axis of ECF is decided by the right hand rule.

2.2. Measurement Model

In this paper, a two-dimensional image point is denoted by $p={\left[u,v\right]}^T$, a three-dimensional spatial point by $P={\left[X,Y,Z\right]}^T$. $\tilde{p}$ and $\tilde{P}$ are the corresponding homogeneous points, $\tilde{p}={[p^T,1]}^T$, $\tilde{P}={[P^T,1]}^T.$ The projection of a spatial point in TCF onto the image plane is described as:

$$s\tilde{p}=\left[{\rm K}|0_{3\times 1}\right]T\tilde{P},K=\left[\begin{array}{ccc}{f}_x& 0& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right],T=\left[\begin{array}{cc}{R}_{3\times 3}& t_{3\times 1}\\ 0_{1\times 3}& 1\end{array}\right],R=\left[\begin{array}{ccc}{R}_{11}& R_{12}& R_{13}\\ R_{21}& R_{22}& R_{23}\\ R_{31}& R_{32}& R_{33}\end{array}\right]$$

(1)

where K is the intrinsic parameter matrix, f_x and f_y are the equivalent focal length in horizontal and vertical directions, respectively. (u₀, v₀) is the principal point. T denotes the transformation matrix between targets and cameras. R is a 3 × 3 rotation matrix, t is a 3 × 1 translation vector. The rotation matrix can be expressed in terms of Y-X-Z Euler angles: yaw angle φ, pitch angle θ and roll angle ϕ:

$$R\left(\mathsf{\phi },\mathsf{\theta },\mathsf{\varphi }\right)=\left[\begin{array}{ccc}cos\mathsf{\varphi }cos\mathsf{\phi }+sin\mathsf{\theta }sin\mathsf{\varphi }sin\mathsf{\phi }& cos\mathsf{\theta }sin\mathsf{\varphi }& cos\mathsf{\phi }sin\mathsf{\theta }sin\mathsf{\varphi }-cos\mathsf{\varphi }sin\mathsf{\phi }\\ cos\mathsf{\varphi }sin\mathsf{\theta }sin\mathsf{\phi }-cos\mathsf{\phi }sin\mathsf{\varphi }& cos\mathsf{\theta }cos\mathsf{\varphi }& sin\mathsf{\varphi }sin\mathsf{\phi }+cos\mathsf{\varphi }cos\mathsf{\phi }sin\mathsf{\theta }\\ cos\mathsf{\theta }sin\mathsf{\phi }& -sin\mathsf{\theta }& cos\mathsf{\theta }cos\mathsf{\phi }\end{array}\right]$$

(2)

In this paper, definitions of the transformation matrices are shown in

Table 1. Definition of the transformation matrices.

$T_k^{\mathrm{tc}}$	The transformation matrix from TkCF to CkCF
$T_{ii}^{\mathrm{ta}}$	The transformation matrix from TiCF to AiCF
$T_{ji}^{\mathrm{ta}}$	The transformation matrix from TjCF to AiCF
$T_{ij}^{\mathrm{tt}}$	The transformation matrix from TiCF to TjCF
$T_{ij}^{\mathrm{cc}}$	The transformation matrix from CiCF to CjCF

.

3. The Principle of Global Calibration

The principle of global calibration is shown in Figure 2. In this paper, we choose camera 1 as the reference camera as well as target 1 as the reference target. The main process of the proposed global calibration method works as follows:

• Intrinsic calibration is done separately for each camera using the J. Bouguet Camera Calibration Toolbox based on Zhang’s calibration method [23,24].The intrinsic parameters are treated as fixed and the cameras’ poses are unchangeable during the calibration.
• Place the planar targets in each camera’s FOV. The symmetry axis of each target is set to approximately orient to its corresponding camera. Image I_k denotes target k captured by camera k. An image sequence I = {I_k|1 ≤ k ≤ M} is obtained.
• Use the auxiliary camera to capture neighboring pairs of the targets. As shown in Figure 2a, image ${\tilde{I}}_i$ denotes two adjacent targets (i, j) captured by the auxiliary camera. A closed image sequence $\tilde{I}$ is acquired, $\tilde{I}=\left\{{\tilde{I}}_i|1\le i\le M\right\};$ 1 ≤ i ≤ M; j = i +1 if i < M, j = 1 if i = M.
• All the images are rectified to compensate for cameras’ distortion based on the intrinsic calibration results. The linear equation of parallel lines in the image plane can be obtained from the feature points extracted by Steger’s method [25].
• Compute the transformation matrix $T_k^{\mathrm{tc}}$ based on the undistorted image sequence I.
• Compute the transformation matrix $T_{ij}^{\mathrm{tt}}$ of two adjacent targets based on the undistorted image sequence $\tilde{I}.$
• Calculate the initial value of $T_{k1}^{\mathrm{tt}}$ (2 ≤ k ≤ M) by multiple coordinate transformations, as shown in Figure 2b. Then $T_{k1}^{\mathrm{tt}}$ (2 ≤ k ≤ M) is refined by the global nonlinear optimization.
• Compute the transformation matrix $T_{k1}^{\mathrm{cc}}$ (2 ≤ k ≤ M).The calibration is completed.

3.1. Sloving $T_k^{\mathrm{tc}}$

3.1.1. Feature Extraction

The feature points on a line can be extracted by Steger’s method [25], denoted by p_i = [u_i, v_i]^T, where 1 ≤ i ≤ s. s is the number of feature points. The projection of a line onto the image plane is also a straight line. The equation of a line on the image can be expressed as au + bv + c = 0.

Let $A={[\begin{array}{cccc}{p}_1& p_2& \dots & p_s\end{array}]}^T,w={\left[\begin{array}{cccc}-1& -1& \dots & -1\end{array}\right]}^T$, $A$ is a $s\times 2$ matrix, $w$ is a $s\times 1$ vector. Thus the relation of $a,b$ and $c$ can be obtained by the least squares method:

$$\left[\begin{array}{c}a\\ b\end{array}\right]=c{\left[A^{T}A\right]}^{-1}\left[A^{T}w\right]$$

(3)

Thus, the linear equation of line $l_m^k$ can be found by the above method, where 1 ≤ m ≤ 6. Then the coordinates of ${\tilde{p}}_m^k$ are obtained from line intersections. As shown in Figure 1b, two vanishing points v₁ and v₂ can be found by lines $l_1^k$, $l_3^k$ and lines $l_5^k$, $l_6^k$, respectively. Then the linear equation of the vanishing line is obtained, denoted by:

$$\tilde{a}u+\tilde{b}v+\tilde{c}=0$$

(4)

3.1.2. Computing the Vanishing Line

As shown in Figure 1b, two groups of parallel lines converge at vanishing points v₁ and v₂ in the image plane, respectively. The line crossing v₁ and v₂ is the vanishing line. The equations of two non-parallel lines in T_kCF are:

$$a_{i}x+c_{i}z+d_i=0,y=0,\left(i=1,2\right)$$

(5)

where $a_1{c}_2-a_2{c}_1\ne 0.$

Let ${\tilde{V}}_1\text{ and }{\tilde{V}}_2$ be the infinite points in the two lines, ${\tilde{V}}_i={\left[-c_i,0,a_i,0\right]}^T,i=1,2$ (the elaboration is given in Appendix A). We have:

$$s_i{\tilde{v}}_i=\left[{\rm K}|0_{3\times 1}\right]T_k^{\text{tc}}{\tilde{V}}_i\left(i=1,2\right)$$

(6)

where K denotes the intrinsic parameter matrix of camera k.

Combining Equations (1) and (6), we have:

$${\tilde{v}}_i={\left[f_x\frac{-c_i{R}_{11}+a_i{R}_{13}}{-c_i{R}_{31}+a_i{R}_{33}}+u_0,f_y\frac{-c_i{R}_{21}+a_i{R}_{23}}{-c_i{R}_{31}+a_i{R}_{33}}+v_0,1\right]}^T$$

(7)

Vanishing line l can be computed by $l={\tilde{v}}_1\times {\tilde{v}}_2$. With Equation (7), we have:

$$l=\left[\begin{array}{l}{f}_y(R_{23}{R}_{31}-R_{21}{R}_{33})\\ f_x(R_{11}{R}_{33}-R_{13}{R}_{31})\\ f_x{f}_y\left(R_{21}{R}_{13}-R_{11}{R}_{23}\right)+f_x{v}_0\left(R_{13}{R}_{31}-R_{11}{R}_{33}\right)+f_y{u}_0\left(R_{21}{R}_{33}-R_{23}{R}_{31}\right)\end{array}\right]$$

(8)

Combining Equations (2) and (8), the linear equation of the vanishing line is expressed as:

$$\frac{\sin \mathsf{\varphi }}{f_x}\left(u-u_0\right)+\frac{\cos \mathsf{\varphi }}{f_y}\left(v-v_0\right)-\tan \mathsf{\theta }=0$$

(9)

3.1.3. Computing the Rotation Matrix of $T_k^{\mathrm{tc}}$

Combining Equations (4) and (9), the roll angle ϕ and pitch angle θ can be obtained:

$$\{\begin{array}{l}\mathsf{\varphi }={\tan }^{-1}\left[f_x\tilde{a}/\left(f_y\tilde{b}\right)\right]\\ \mathsf{\theta }={\tan }^{-1}\left(-\frac{\tilde{c}}{\tilde{a}}\frac{\sin \mathsf{\varphi }}{f_x}-\frac{\sin \mathsf{\varphi }}{f_x}{u}_0-\frac{\cos \mathsf{\varphi }}{f_y}{v}_0\right)\end{array}$$

(10)

Vanishing points are determined by the directions of the parallel lines [18], we have:

$${\tilde{v}}_i=K{d}_i\left(i=1,2\right)$$

(11)

where d_i is the 3 × 1 direction vector of the line in CCF.

$l_1^k$ and $l_6^k$ coincide with the z-axis and the x-axis of T_kCF, respectively. Thus:

$$\{\begin{array}{l}{d}_1/\Vert d_1\Vert =R\cdot {\left[\begin{array}{ccc}0& 0& 1\end{array}\right]}^T\\ d_2/\Vert d_2\Vert =R\cdot {\left[\begin{array}{ccc}1& 0& 0\end{array}\right]}^T\end{array}$$

(12)

According to the orthogonal constraint of a Rotation matrix, the rotation matrix R of $T_k^{\mathrm{tc}}$ can be obtained:

$$R=\left[\begin{array}{ccc}\frac{d_2}{\Vert d_2\Vert }& \frac{d_1\times d_2}{\Vert d_1\Vert \cdot \Vert d_2\Vert }& \frac{d_1}{\Vert d_1\Vert }\end{array}\right]$$

(13)

3.1.4. Computing Translation Vector of $T_k^{\mathrm{tc}}$

$P_7^k$ is a virtual point in the target plane. As vector $\overrightarrow{P_1^k{P}_2^k}\perp d_2,$ the projections of $\overrightarrow{P_1^k{P}_7^k}$ and $\overrightarrow{P_2^k{P}_7^k}$ onto the vector $d_2$ are equal. We have:

$$d_2^T\overrightarrow{P_1^k{P}_7^k}=d_2^T\overrightarrow{P_2^k{P}_7^k}$$

(14)

Combining Equations (1) and (14), we have:

$$\frac{z_1}{z_2}=\frac{d_2^T{K}^{-1}{\tilde{p}}_2^k}{d_2^T{K}^{-1}{\tilde{p}}_1^k}$$

(15)

where $z_1$ and $z_2$ are the z coordinates of $P_1^k$ and $P_2^k$ in ${\mathrm{C}}_k\text{CF}$, respectively. ${\tilde{p}}_1^k$ and ${\tilde{p}}_2^k$ are known coordinates of the corner points.

Besides, the length of $P_1^1{P}_2^k$ is known:

$$\Vert z_1{K}^{-1}{\tilde{p}}_1^k-z_2{K}^{-1}{\tilde{p}}_2^k\Vert =L_1$$

(16)

Combining Equations (15) and (16), $z_1$ and $z_2$ can be found. Thus, the coordinate of $P_1^k$ in ${\mathrm{C}}_k\text{CF}$ is obtained. In addition, $P_1^k$ is the origin of ${\mathrm{T}}_k\text{CF}$, the translation vector $t$ of $T_k^{\text{tc}}$ can be obtained:

$$t=z_1{K}^{-1}{\tilde{p}}_1^k$$

(17)

3.1.5. Nonlinear Optimization

Let ${\tilde{P}}_m^k\left(1\le m\le 6\right)$ be the homogeneous coordinate of $P_m^k$ in ${\mathrm{T}}_k\text{CF}$. Let ${\tilde{p}}_m^k$ be the corresponding coordinate in the image $I_k$. We have:

$$s_m^k{\tilde{p}}_m^k=\left[{\rm K}|0_{3\times 1}\right]T_k^{\text{tc}}{\tilde{P}}_m^k$$

(18)

Assuming that image points are corrupted by independently and identically distributed Gaussian noise, the maximum likelihood estimation is obtained by minimizing the sum of squared distances between the observed feature lines and the re-projected corner points. $T_k^{\mathrm{tc}}$ (1 ≤ k ≤ M) are refined separately by minimizing the following function using Levenberg-Marquardt algorithm [26]:

$$f\left(\mathsf{\Omega }\right)={\displaystyle \sum _{m=1}^6\left[d^2\left({\tilde{p}}_m^k,l_m^k\right)+d^2\left({\tilde{p}}_m^k,l_n^k\right)\right]}$$

(19)

$$n=\{\begin{array}{l}m-1,\text{\hspace{1em}if }m\ge 2\\ 6,\text{ \hspace{1em}\hspace{1em}\hspace{1em}if }m=1\end{array}$$

(20)

where Ω = $T_k^{\mathrm{tc}}$. $l_m^k$ and $l_n^k$ denotes the projections of $l_m^k$ and $l_n^k$ onto the image I_k, respectively. d(·) denotes distances between points and lines. R of$T_k^{\mathrm{tc}}$ is parameterized using the Rodrigues’ formula [27].

3.2. Initializing $T_{k1}^{\mathrm{tt}}$

Generally, target pair (i, j) is visible in the image ${\tilde{I}}_i$, that:

$$j=\{\begin{array}{l}i+1,\text{ \hspace{1em}if }i\le M-1\\ 1,\text{ \hspace{1em}\hspace{1em}if }i\text{ = }M\end{array}$$

(21)

As shown in Figure 2a, $T_{ii}^{\mathrm{ta}}$ and $T_{ji}^{\mathrm{ta}}$ are the transformation matrices from target i and target j to the auxiliary camera, respectively. $T_{ii}^{\mathrm{ta}}$ and $T_{ji}^{\mathrm{ta}}$ can be initialized and refined separately by the methods described in Section 3.1. Then the initial value of $T_{ij}^{\mathrm{tt}}$ can be calculated by:

$$T_{ij}^{\text{tt}}={\left(T_{ji}^{\text{ta}}\right)}^{-1}{T}_{ii}^{\text{ta}}$$

(22)

The initial value of $T_{k1}^{\mathrm{tt}}$ (2 ≤ k ≤ M) can be obtained by the minimum times of chainwise coordinate transformations:

$$T_{k1}^{\text{tt}}=\{\begin{array}{l}{\left[T_{k-1,k}^{\text{tt}}{T}_{k-2,k-1}^{\text{tt}}\cdots T_{2,3}^{\text{tt}}{T}_{1,2}^{\text{tt}}\right]}^{-1},\text{ \hspace{1em}\hspace{1em}if }k\le \left(M/2\right)\\ T_{M,1}^{\text{tt}}{T}_{M-1,M}^{\text{tt}}\cdots T_{k+1,k+2}^{\text{tt}}{T}_{k,k+1}^{\text{tt}},\text{ \hspace{1em}\hspace{1em}if }k>\left(M/2\right)\end{array}$$

(23)

3.3. Global Calibration of the Targets

According to the camera model, we have:

$$\{\begin{array}{l}{s}_m^{ii}{\tilde{p}}_m^{ii}=\left[{\rm K}|0_{3\times 1}\right]T_{ji}^{\text{ta}}{\left(T_{j1}^{\text{tt}}\right)}^{-1}{T}_{i1}^{\text{tt}}{\tilde{P}}_m^i\\ s_m^{ji}{\tilde{p}}_m^{ji}=\left[{\rm K}|0_{3\times 1}\right]T_{ii}^{\text{ta}}{\left(T_{i1}^{\text{tt}}\right)}^{-1}{T}_{j1}^{\text{tt}}{\tilde{P}}_m^j\end{array}$$

(24)

where $K$denotes the intrinsic matrix of the auxiliary camera; ${\tilde{p}}_m^{ii}$ and ${\tilde{p}}_m^{ji}$ denote the reprojections of $P_m^i$ and $P_m^j$ onto the image ${\tilde{I}}_i$, respectively.

Assuming image points are corrupted by independent and identical Gaussian noise, $T_{k1}^{\mathrm{tt}}$ (2 ≤ k ≤ M) can be optimized by minimizing the following function using Levenberg-Marquardt algorithm [26]:

$$f\left(\mathsf{\Omega }\right)={\displaystyle \sum _{i=1}^M{\displaystyle \sum _{m=1}^6\left[d^2\left({\tilde{p}}_m^{ii},l_m^{ii}\right)+d^2\left({\tilde{p}}_m^{ii},l_n^{ii}\right)+d^2\left({\tilde{p}}_m^{ji},l_m^{ji}\right)+d^2\left({\tilde{p}}_m^{ji},l_n^{ji}\right)\right]}}$$

(25)

where $\mathsf{\Omega }=\left(T_{2,1}^{tt},\dots , T_{M1}^{tt}\right), T_{1,1}^{tt}=I_{4\times 4}$ $l_m^{ii}$ and $l_m^{ji}$ denote the projections of line $l_m^i$ and $l_m^j$ onto the image ${\tilde{I}}_i$, respectively. $R\text{ of }{T}_{k1}^{\text{tt}}\left(2\le k\le M\right)$ are parameterized by the Rodrigues formula. A good starting point of the optimization is provided by Equations (22) and (23). $\left(m,n\right)$ and $\left(i,j\right)$ subject to Equations (20) and (21), respectively.

3.4. Solving $T_{k1}^{\text{cc}}$

After the global calibration of the targets, the transformation matrix from each camera to the reference camera can be found:

$$T_{k1}^{\text{cc}}=T_1^{\text{tc}}{T}_{k1}^{\text{tt}}{\left(T_k^{\text{tc}}\right)}^{-1}\left(2\le k\le M\right)$$

(26)

where $T_k^{\mathrm{tc}}$ (1 ≤ k ≤ M) are the results of Equation (19), $T_{k1}^{\mathrm{tt}}$ (2 ≤ k ≤ M) are the optimization results of Equation (25).

4. Accuracy Analysis of Different Factors’ Effects

In this section, analysis of several factors’ effects on the accuracy of the proposed method is performed by synthetic data experiments. The auxiliary camera’s intrinsic parameters are f_x = f_y = 512, u₀ = 512, v₀ = 384. The image resolution is 1024 pixel × 768 pixel.

The cameras’ positions are represented by the coordinates of the cameras’ origins in ECF. The cameras’ orientations are denoted by the Euler angles (φ, θ, ϕ) from ECF to CCF. Targets are placed on the ground for convenience. The targets’ positions are represented by the coordinates of the targets’ origins in ECF. The targets’ orientations are denoted by the yaw angle φ from positive z-axis of ECF to the symmetry axis of the target.

dR and dt denote the 2-norm of rotation vector and translation vector differences between the calculation results and the real data. The RMS errors of dR, dt are used to evaluate the accuracy. The number of points that emulate feature lines is equal to the line length in pixels. Gaussian noise with zero mean and different noise levels is added to the image coordinates of the points of feature lines. Analysis of the factors’ effects is shown as follows.

4.1. Accuracy vs. the Pitch Angle of Camera Relative to the Target

The image sequence $\tilde{I}$ is acquired by the auxiliary camera. The pitch angle of the auxiliary camera relative to the target plane is one of the factors affecting the calibration accuracy. In this experiment, two adjacent targets are captured by the auxiliary camera at different pitch angles. The targets’ positions in ECF are [−450, 0, 300]^T and [450, 0, 300]^T, respectively. The yaw angles of the targets relative to ECF are −18° and 18°, respectively. Two targets are symmetric about the plane y_eo_ez_e. The optical axis of the auxiliary camera lies in the symmetry plane y_eo_ez_e. The error of $T_{1,2}^{\mathrm{tt}}$ obtained by Equation (22) is used to evaluate the effect of the pitch angle. Gaussian noise with σ = 0.2 pixel is added. L₁ = 500 mm, L₂ = 200 mm. For each level of pitch angle θ, 100 independent trials are performed.

From Figure 3, the RMS errors of rotation and translation are roughly U-shape. When θ→−90°, the optical axis of the auxiliary camera is perpendicular to the target plane. The vanishing points approximate to infinity, which leads to higher errors. When θ→0°, the number of the extracted feature points decreases, also leads to higher errors. It is ideal to capture pair targets when θ = −40°.

4.2. Accuracy vs. the Yaw Angle Difference between Two Adjacent Targets

In this experiment, ∆φ denotes the yaw angle difference between the symmetry axes of two adjacent targets. ∆φ varies according to the cameras’ distribution. We also use the error of $T_{1,2}^{\mathrm{tt}}$ calculated by Equation (22) to evaluate the effect of ∆φ.

The positions of the two targets are same with those in Section 4.1, while ∆φ varies from 0 to 85°. The two targets remain symmetric about the plane y_eo_ez_e and the auxiliary camera lies in the symmetry plane. Gaussian noise with σ = 0.2 pixel is added. L₁ = 500 mm, L₂ = 200 mm. For each level, 100 independent trials are performed.

From Figure 4, both the error of rotation and translation rise with the increasing of ∆φ. When ∆φ > 80°, the errors increase sharply. This is because when ∆φ→90°, a group of parallel lines of each target are parallel to the image plane, thus the vanishing points approximate to infinity, which leads to great errors, so it is necessary to avoid ∆φ→90° during the calibration.

4.3. Accuracy vs. the Distance of Parallel Lines

In this experiment, we also use the error of $T_{1,2}^{\mathrm{tt}}$ obtained by Equation (22) to evaluate the effect of the parallel line distances. The poses of the targets are same with those in Section 4.1. The pitch angle of the auxiliary camera relative to the target plane is set to −40°. L₁ = 500 mm, L₂ varies from 100 mm to 400 mm. Gaussian noise with different levels is added to the image points. For each distance level, 100 independent trials are performed.

From Figure 5, it can be seen that the error increases linearly with the noise level and decreases with the increasing distance of parallel lines. This is because the difference among slopes of intersecting lines goes up with the increasing distance of parallel lines. Calculation error of vanishing points is inversely proportional to the difference among the slopes of intersecting lines, which have been proven in [21].

5. Experimental Results

5.1. Experiment of a Use-Case

Numerous situations require a system that provides a real-time view of the surroundings [28]. One of the typical cases is the operations on aerial vehicles. In this experiment, eight cameras are used to simulate a DVS mounted on an unmanned aerial vehicle (UAV), as shown in Figure 6. The proposed method is compared with other typical methods by both synthetic data and images simulated by 3ds Max software.

The intrinsic parameters of eight cameras are f_x = f_y = 796.44, u₀ = 512, v₀ = 384. The intrinsic parameters of the auxiliary camera and the image resolution are same with those in Section 4. The positions and orientations of the cameras are listed in

Table 2. The positions and orientations of the cameras.

Camera ID	1	2	3	4	5	6	7	8
x (mm)	100	600	600	200	−200	−600	−600	−100
y (mm)	−150	−150	−150	−150	−150	−150	−150	−150
z (mm)	0	−400	−500	−1000	−1000	−500	−400	0
φ (deg)	28	64	125	158	202	235	296	332
θ (deg)	0	0	0	0	0	0	0	0
ϕ (deg)	0	0	0	0	0	0	0	0

. Each target is placed on the ground in its corresponding camera’s FOV. All the targets have the same size, L₁ = 500 mm, L₂ = 200 mm.

5.1.1. Description of the Calibration Methods

There are many calibration methods for multiple cameras. Here five typical methods are described as follows and summarized in

Table 3. Summary of the calibration methods

Methods	Calibration Targets	Auxiliary Camera Capture
The proposed method	Line-feature targets	Neighboring target pairs
Method 1	Point-feature targets	Neighboring target pairs
Method 2	Checkerboards	Neighboring target pairs
Method 3	Line-feature targets	All targets in one image
Method 4	Point-feature targets	All targets in one image
Method 5	Checkerboards	All targets in one image

:

Method 1: This method is similar to the proposed method, except that corner points ${\tilde{p}}_m^k$ are extracted by the corner extraction engine of the J. Bouguet Camera Calibration Toolbox [23], rather than the intersections of feature lines.

Method 2: This method is similar to the proposed method, except that planar checkerboards with 12 × 12 grids are used as the calibration targets. The side length of each square is 50 mm.

Method 3: The calibration targets and the extraction of corner points ${\tilde{p}}_m^k$ are same with the proposed method. Instead of capturing neighboring target pairs, the auxiliary camera captures all the targets in one image frame, so the relative poses between targets can be computed directly.

Method 4: This method is similar to Method 3, except that corner points ${\tilde{p}}_m^k$ are obtained by the corner extraction method used in Method 1.

Method 5: This method is similar to Method 3, except that planar checkerboards of Method 2 are used as the calibration targets.

In order to illustrate the effect of the global calibration, there are two sub-methods called chainwise calibration method and global calibration method. The only difference between the two sub-methods is whether to use the global optimization in Section 3.3 or not. $T_{k1}^{\mathrm{tt}}$ (2 ≤ k ≤ M) of the chainwise method are obtained directly from multiple coordinate transformations by Equation (23), while the global calibration method is based on an additional global optimization by Equation (25).

5.1.2. Synthetic Data Experiment

In this experiment, the RMS error of $T_{k1}^{\mathrm{cc}}$ (2 ≤ k ≤ 8) is used to evaluate the accuracy. Gaussian noise with σ = 0.2 pixel is added. For each method, 100 independent trials are performed. From Figure 7, the proposed method outperforms Methods 1–5. Figure 7a,b show that the error accumulates with the coordinate transformations, and peaks at camera 5, due to the maximum times of transformations. The methods based on the constraint of ring-topologies can effectively reduce the accumulated error, especially for cameras which are far away from the reference.

Methods 3–5 do not suffer from the accumulated error issue because all the targets are visible in one image frame. However, due to limitations of image resolutions, the accuracy of the pose estimation decreases with the increasing number of the targets observed in one image.

Compared with lines-feature algorithms, points-feature algorithms are more sensitive to the image noise. Figure 7 shows that Methods 1 and 4 are worse than other methods, respectively. Further discussion is given in Section 5.3.

5.1.3. Accuracy vs. the Image Noise Level

In this experiment, the RMS error of $T_{k1}^{\mathrm{cc}}$ (2 ≤ k ≤ 8) is used to evaluate the effect of the noise level. Synthetic data is same with those in Section 5.1.2. Gaussian noise from 0.0 to 1.0 pixel is added. For each noise level, 100 independent trials are performed.

From Figure 8, the RMS error increases linearly with the noise level. It also shows that the proposed method is superior to other methods. If the noise level of the real DVS is less than 0.5 pixels, the RMS errors of rotation and translation of all the cameras are less than 0.05 deg and 1.1 mm, respectively, which is acceptable for common applications.

5.1.4. Experiments Based on Simulation Images

As shown in Figure 9, we use the 3ds Max software to simulate image sequences. The parameters of the cameras and the targets are same with those in Section 5.1.2. The feature lines are obtained based on feature points extracted by Steger’s method [25]. The error of $T_{k1}^{\mathrm{cc}}$ (2 ≤ k ≤ 8) is used to evaluate the accuracy.

Figure 10 shows that errors of rotation and translation accumulate with the increasing times of coordinate transformations. The proposed method can reduce the accumulated error due to multiple coordinate transformations. Further discussion is given in Section 5.3.

5.2. Real Data Experiment

As shown in Figure 11a, eight targets are located in an area about 1200 mm × 1200 mm. As the relative poses of the cameras with non-overlapping FOV are mainly determined by the relative poses of the targets, the RMS errors of point pair distances between eight targets are used as calibration errors in the real experiments.

The distance of point pair $p_m^k$ and $p_m^l$ is computed according to the calibration result, and is called measurement distances, d_m. The targets are also calibrated similarly by a calibrated Canon 60D digital camera. The distances of the same point pairs can be obtained in the same way and are used as the ground truth d_t, due to its relatively high accuracy.

Distance error can be computed by $\mathsf{\Delta }d=d_m-d_t$. For the proposed method, Methods 1, 3 and 4, RMS error of $\mathsf{\Delta }d\left(P_1^1{P}_1^k\right),\mathsf{\Delta }d\left(P_2^1{P}_2^k\right)$ and $\mathsf{\Delta }d\left(P_6^1{P}_6^k\right)$ is used to evaluate the accuracy. For Methods 2 and 5, five point pairs are randomly selected and the RMS error of distance error is computed as the calibration error.

The auxiliary camera is a 1/3-in Sony CCD image sensor (ICX673) with a 3.6 mm lens. The image resolution is 720 pixel × 432 pixel. Target parameters are $L_1=135\text{ mm},L_2=70\text{ mm}.$ The image resolution of the Canon device is 1920 pixel × 1280 pixel. The intrinsic parameters of the sensors are calibrated using Bouguet’s calibration toolbox [23], as shown in

Table 4. Intrinsic parameters of the vision sensors

	fx	fy	u0	v0	k1	k2	p1	P2
Sony	592.73225	460.22694	343.24159	225.49825	−0.40175	0.15735	−0.00009	0.00111
Canon	1561.29373	1560.15359	972.49782	597.06160	−0.19178	0.13171	−0.00190	−0.00129

.

Figure 12 also shows that the proposed method achieves the best accuracy. The RMS error of point pair distances between target 5 and the reference target of the proposed method and Methods 1–5 are 0.465 mm, 0.828 mm, 3.94 mm, 3.83 mm, 1.92 mm and 21.6 mm, respectively. The proposed method is superior to other methods.

5.3. Discussion

Due to the restriction of image resolutions, the accuracy of the pose estimation decreases with the increasing number of the targets observed in one image. There exists a trade-off between the available features of target projections and the accumulated error from the chain of transformations. The experimental results show that the accumulated error can be effectively adjusted from the constraint of ring-topologies. For the vision sensor such as the Sony CCD sensor, capturing all the targets is not a wise choice. The benefits of direct calculation of the relative poses of the targets are cancelled out by the rise of feature extraction errors.

Moreover, it is not convenient to capture all the targets in some applications, because the auxiliary camera should be far away from the widely distributed targets. As shown in Figure 11b, in order to capture all the checkerboards in one image frame, the targets are pasted on the wall.

The results of simulation images show that the accuracy of Methods 2 and 5 is close to or even better than the proposed method. However, Methods 2 and 5 achieve the worst accuracy in the real experiments. Figure 13 shows that simulation images are very sharp and clear, which greatly benefit the corner extraction of checkerboards. However, real images could not be so ideal.

These results indicate that the proposed method is accurate and robust, especially when dealing with real images. Methods 2 and 5 are not stable against the image quality. However, there is a gap among the results of synthetic data, simulation images and real experiments. There may be some reasons for this.

Firstly, there is a measurement error during the feature extraction. In our method, the line extraction algorithm is a common-used method with acceptable accuracy and good generality. Line extraction algorithms with higher accuracy contribute to improve the accuracy, which will be further studied in the future. Secondly, the targets used in real experiments are printed on paper. They may not be strictly planar, which also leads to measurement errors.

In addition, the Sony CCD vision sensor is not a professional high-precision vision sensor, which is usually used for security cameras and radio controlled vehicles. High-resolution vision sensors can be used to improve the accuracy.

6. Conclusions

In this article, we have developed a new global calibration method for vision sensors in ring-topologies. Line-based calibration targets are placed in each camera’s FOV. Firstly, the relative poses of cameras and targets are initialized and refined based on the principle of vanishing features and the known line length. Next, in order to overcome small or no overlapping FOV between adjacent cameras, an auxiliary camera is used to capture neighboring targets. The relative poses of the targets is initialized in a chainwise manner, followed by nonlinear optimization to minimizing the squared distances between the observed feature lines and the re-projected corner points. Then the transformation matrix between each camera and the reference camera is determined.

The factors that affect the calibration accuracy are analyzed by synthetic data experiments. Synthetic data, simulation images and real data experiments all demonstrate that the proposed method is accurate and robust to image noise. The accumulated error can be adjusted effectively based on the constraint of ring-topologies. Real data experiments indicates that the measurement accuracy of the farthest camera by the proposed method is about 0.465 mm in an area about 1200 mm × 1200 mm.

The poses of targets need not be known previously and can be adjusted according to the distribution of cameras. It does not need to place the targets into different positions, one placement is enough. Our method is simple and flexible and can be applied to different configurations of multiple cameras. It is well suited for the on-site calibration of widely distributed cameras.

In this paper, we focus on the calibration of DVS in ring-topologies, which contains additional constraint. When dealing with DVS in open-topologies, accumulated errors cannot be adjusted. In addition, vanishing points approximate to infinity when feature lines are parallel to the image plane, which leads to higher errors, so the angle between parallel lines and the image plane should be in a certain range to avoid vanishing points approximate to infinity.

Restricted by hardware conditions, experiments using eight sensors mounted on an UAV are temporarily lacking. We plan to apply our method for the calibration of multiple vision sensors mounted on the vehicle in the future. Methods based on the feature lines in both indoor and outdoor environments instead of planar targets will also be investigated.