A New Camera Calibration Technique for Serious Distortion

Abstract

A new camera calibration technique based on serious distortion is proposed, which only requires the camera to observe the plane pattern in an arbitrary azimuth. It uses the geometrical imaging principle and radial distortion model to acquire radial lens distortion coefficient and the image coordinate (u₀, v₀), and then solves the linear equation aiming at the other parameters of the camera. This method has the following characteristics: Firstly, the position of the camera and the plane is arbitrary, and the technique needs only a single observation for plane pattern. Secondly, it is suitable for camera calibration with serious distortion. Thirdly, it does not need expensive ancillary equipment, accurate movement, or lots of photos observed from different orientations. Having been authenticated by computer emulation and actual experiment, the results of the proposed technique have proved to be satisfactory. The research has also paved a new way in camera calibration for further studies.

1. Introduction

Camera calibration is a necessary step in photogrammetry and computer vision, which plays an important role in many spheres such as 3D-measurement [1], 3D object reconstruction [2], robot navigation [3], visual surveillance [4], and industrial inspection [5]. Many studies have been conducted in this respect. And the techniques used can be classified into three categories: traditional camera calibration, active vision camera calibration, and camera self-calibration.

Traditional calibration. This kind of method uses information to gain parameters of the camera from the known landscape structure, which requires a calibration block of high precision. In 1986, Faugeras and Toscani [6] proposed a camera calibration method using some known points in the space to solve linear equations. Tsai [7] used the two-stage calibration technique to achieve the efficient computation of a camera’s external position and orientation. Further studies on two-stage calibration were carried out by Wen [8] and Gao [9]. These methods are always accurate for camera calibration; however, they do not fit in every situation.

Active vision camera calibration. This is completed based on the known information of the precise movement of a camera. Ma [10] exploited a sequence of specially designed motions, which were two groups of translational motions, to calibrate the camera. On this foundation, Yang [11] developed a new active vision calibration technique from four or five groups of the orthogonal motions. In recent years, active vision calibration has been developed further [12,13,14], which can easily solve problems and obtain parameters. This type of camera calibration method usually requires very sophisticated equipment and therefore calibration equipment is relatively expensive.

Camera self-calibration. This category only depends on the corresponding relationship among images, and does not need a calibration target, a known landscape structure, or the precise movement of the camera. There are two quadratic nonlinear constraints between two of the images. These constraints are composed of Kruppa equations and are used to obtain the intrinsic parameters of the camera [15,16]. Faced with the difficulty of obtaining nonlinear equations, scholars have proposed various methods for self-calibration of cameras [17,18,19]. The biggest advantage of the camera self-calibration technique is its excellent flexibility. Yet the robustness of it is lower and the nonlinear equations have to be solved.

Besides this, there are some other calibration methods which are calibrated by double planar mirrors [20], nonlinear optimization [21], feature Points Motions [22], the plane at infinity [23], invariance of cross-ratio [24], parallel circles [25], image information of different orientations from plane [26], coplanar camera calibration [27], planar mirrors from silhouettes [28], and turntable sequences from silhouettes [29].

Currently, camera calibration techniques are mainly focused on industrial cameras (they do not have severe camera distortion). However, wide-angle cameras (they have very bad camera distortion) are widely used in a number of specific fields due to the large field of view image information that can be obtained. An example is fruit tree branch-pruning robots [30], which require the acquisition of a complete image of a branch at close range and the identification of the branch to be cut from the overall branch image. Using a normal camera to capture images often requires multiple image stitches, which is not only time-consuming but also has a significant impact on stability and accuracy. Therefore, the use of a wide-angle camera to capture a complete image of the branch is a good option. Wide-angle camera lenses usually have severe distortion, and using traditional camera calibration methods to achieve wide-angle camera calibration usually requires human guidance and a lot of iterative processing to complete the calibration, which seriously affects the calibration efficiency and accuracy. To address these issues, this paper proposes a simple, fast, and intelligent wide-angle camera calibration method. The method is based on a planar target to complete the calibration of the camera. Compared with other methods, this method is simple to operate, has low equipment requirements, and only needs to acquire the target image once (traditional calibration based on a flat target requires multiple image acquisitions), has a small number of iterations (usually only ten), has a short computation time, and is suitable for the calibration of cameras with severe distortion.

The rest of the paper is organized as follows: Section 2 and Section 3 present a method of obtaining the image coordinate (u₀, v₀) and correcting the image distortion. In Section 4, we build the camera model, which includes four parameters, and we obtain all parameters of the camera. In Section 5, the experimental results are given. Finally, the results are analyzed and discussed in Section 6.

2. Obtaining the Image Coordinate (u₀, v₀) and Distortion Coefficients k₁ and k₂

In this paper, the image coordinate (u₀, v₀) is the key to the camera calibration, which can be gained exactly according to the following method.

2.1. Geometrical Deduction of Imaging

In a camera coordinate system, we assume there is an arbitrary line that has four points marked A, B, C, and D. Their imaging points are marked A′, B′, C′, D′ (see Figure 1a). In plane x_co_cz_c, H₁, H₂, H₃, H₄, P₁, P₂, P_3, and P₄ are the projection points of A′, B′, C′, D′, A, B, C, and D (see Figure 1b).

As the spatial line AB is in the planar target, we can measure the distance between any two points in the line AB. Therefore, we can easily obtain the following ratios:

$$\frac{BC}{AB}=m_1$$

(1)

$$\frac{BD}{AB}=m_2$$

(2)

For the same line, the ratio of any two line segments does not change in the projection surface. According to (1) and (2), we have:

$$\frac{P_2{P}_3}{P_1{P}_2}=m_1$$

(3)

$$\frac{P_2{P}_4}{P_1{P}_2}=m_2$$

(4)

In Figure 1b, Z_Pi (i = 1, 2, 3, 4) and Z_Hi (i = 1, 2, 3, 4) are used to represent abscissa, while X_Pi (i = 1, 2, 3, 4) and X_Hi (i = 1, 2, 3, 4) are used to represent ordinate. Then we have:

$$\{\begin{array}{c}\frac{Z_{P3}-Z_{P2}}{Z_{P2}-Z_{P1}}=m_1\\ \frac{Z_{P4}-Z_{P2}}{Z_{P2}-Z_{P1}}=m_2\end{array}$$

(5)

The line P₁P₄ can be expressed as L₁: X = kZ + b. Therefore, the coordinate of P_i (i = 1, 2, 3, 4) is represented as (Z_Pi, kZ_Pi + b). To line H_iP_i, we have the equation of the straight line:

$$X=\frac{{\mathit{kZ}}_{Pi}+b}{Z_{Pi}}Z(i=1,2,3,4)$$

(6)

The line H₁H₄ and X-axis are parallel. Therefore, the line H₁H₄ can be expressed as L₂: Z = n. According to (6), the ordinate of H_i (i = 1, 2, 3, 4) is (0, X_i).

$$X_i=\frac{{\mathit{kZ}}_{Pi}+b}{Z_{Pi}}n(i=1,2,3,4)$$

(7)

From (5)–(7), we have

$$\{\begin{array}{c}\frac{H_2{H}_3}{H_1{H}_2}=\frac{Z_{P1}{m}_1}{(Z_{P2}-Z_{P1})m_1+Z_{P2}}\\ \frac{H_2{H}_4}{H_1{H}_2}=\frac{Z_{P1}{m}_2}{(Z_{P2}-Z_{P1})m_2+Z_{P2}}\end{array}$$

(8)

As can be seen from (8), the length ratio in the image is different from the actual one in the Euclidean Space, and it is related to the coordinate of P₁, P₂. For the same line, the projection won’t change the ratio between the segments. So B′C′/A′B′ = H₂H₃/H₁H₂, B′D′/A′B′ = H₂H₄/H₁H₂.

The following are transform relations of imaging point and image point.

$$\{\begin{array}{c}u-u_0={\alpha }_{x}x\\ v-v_0={\alpha }_{y}y\end{array}$$

(9)

From (8) and (9), we can find:

$$\{\begin{array}{c}\frac{u_C-u_B}{u_B-u_A}=\frac{v_C-v_B}{v_B-v_A}=\frac{Z_{P1}{m}_1}{(Z_{P2}-Z_{P1})m_1+Z_{P2}}\\ \frac{u_D-u_B}{u_B-u_A}=\frac{v_D-v_B}{v_B-v_A}=\frac{Z_{P1}{m}_2}{(Z_{P2}-Z_{P1})m_2+Z_{P2}}\end{array}$$

(10)

2.2. Calculation of the Ideal Image Coordinates without Distortion

In this paper, the actual image coordinates have serious distortion. However, the ideal image coordinates are very close to the actual image coordinates in the vicinity of (u₀, v₀), which is located near the center of the image. We try to make these points (A, B, C) near the center of the image when taking pictures. Point D is far away from point C and closest to the edge of the image. The image coordinates of A, B, and C are regarded as the ideal image coordinates without distortion, and are used in calculations. Because of the restriction in (10), we can find the relationship between Z_P1 and Z_P2.

$$Z_{P2}=\lambda Z_{p1}\mathrm{with}\lambda =\frac{(u_C-u_A)m_1}{(u_C-u_B)(m_1+1)}$$

(11)

From (10) and (11), we can obtain the provisional value of the ideal and undistorted image coordinates of point D.

$$\{\begin{array}{c}{u}_D^{\prime }=\frac{(u_B-u_A)m_2}{(\lambda -1)m_2+\lambda }+u_B\\ v_D^{\prime }=\frac{(v_B-v_A)m_2}{(\lambda -1)m_2+\lambda }+v_B\end{array}$$

(12)

Therefore, by using the above method, we can use some (at least three) line segments to obtain the ideal and undistorted image coordinates, which are on the edge of the image, marked (u′_i, v′_i) (i = 1, 2, 3, …).

2.3. Solving the Coordinate (u₀, v₀) and Distortion Coefficients k₁ and k₂

Camera lens aberration is mainly radial aberration. It to the optical axis centre point image coordinates as a reference point, and with the image point to the distance of the reference point into a non-linear relationship. Assuming that the camera in the image horizontal and vertical axis direction of the same aberration, the camera lens radial aberration model is as follows.

$$\{\begin{array}{c}u-u_0=(u^{\prime }-u_0)(1+k_1{r}^2+k_2{r}^4)\\ v-v_0=(v^{\prime }-v_0)(1+k_1{r}^2+k_2{r}^4)\end{array}$$

(13)

The (u, v) is an actual image coordinate, the (u₀, v₀) is the image coordinate of the central point of the optical axis, and the (u′, v′) is an ideal and undistorted image coordinate. $k_1, k_{2 }$ are the second and fourth order radial distortion coefficients, respectively. Besides, r² = (u′ − u₀)² + (v′ − v₀)². Using Equation (13), a number of distortion-free ideal image coordinates can be found. Substituting any two distortion-free ideal image coordinates $(u_{j4}^{\prime },v_{j4}^{\prime })$, $(u_{k4}^{\prime },v_{k4}^{\prime })$ and the corresponding actual image coordinates $(u_{j4},v_{j4})$, $(u_{k4},v_{k4})$ into Equations (4)–(7), where j, k = 1, 2, 3, …

$$\{\begin{array}{l}{u}_{j4}-u_0=(u_{j4}^{\prime }-u_0)(1+k_1{r}_{j4}^2+k_2{r}_{j4}^4)\\ v_{j4}-v_0=(v_{j4}^{\prime }-v_0)(1+k_1{r}_{j4}^2+k_2{r}_{j4}^4)\\ u_{k4}-u_0=(u_{k4}^{\prime }-u_0)(1+k_1{r}_{k4}^2+k_2{r}_{k4}^4)\\ v_{k4}-v_0=(v_{k4}^{\prime }-v_0)(1+k_1{r}_{k4}^2+k_2{r}_{k4}^4)\end{array}$$

(14)

In Equation (14), $r_{j4}=\sqrt{{(u_{j4}^{\prime }-u_0)}^2+{(v_{j4}^{\prime }-v_0)}^2}$, $r_{k4}=\sqrt{{(u_{k4}^{\prime }-u_0)}^2+{(v_{k4}^{\prime }-v_0)}^2}$, Solving for these yields.

$$\{\begin{array}{l}{u}_0=-\frac{(u_{j4}{v}_{j4}^{\prime }-v_{j4}{u}_{j4}^{\prime })(u_{k4}^{\prime }-u_{k4})-(u_{j4}^{\prime }-u_{j4})(u_{k4}{v}_{k4}^{\prime }-u_{k4}^{\prime }{v}_{k4})}{(v_{j4}-v_{j4}^{\prime })(u_{k4}^{\prime }-u_{k4})-(u_{j4}^{\prime }-u_{j4})(v_{k4}-v_{k4}^{\prime })}\\ v_0=-\frac{(u_{j4}{v}_{j4}^{\prime }-v_{j4}{u}_{j4}^{\prime })(v_{k4}-v_{k4}^{\prime })-(v_{j4}-v_{j4}^{\prime })(u_{k4}{v}_{k4}^{\prime }-u_{k4}^{\prime }{v}_{k4})}{(u_{j4}^{\prime }-u_{j4})(v_{k4}-v_{k4}^{\prime })-(v_{j4}-v_{j4}^{\prime })(u_{k4}^{\prime }-u_{k4})}\\ k_1=\left[\frac{(u_{j4}-u_{j4}^{\prime })r_{k4}^2}{(u_{j4}^{\prime }-u_0)r_{j4}^2}-\frac{(u_{k4}-u_{k4}^{\prime })r_{j4}^2}{(u_{k4}^{\prime }-u_0)r_{k4}^2}\right]\frac{1}{(r_{k4}^2-r_{j4}^2)}\\ k_2=\left[\frac{u_{j4}-u_{j4}^{\prime }}{(u_{j4}^{\prime }-u_0)r_{j4}^2}-\frac{u_{k4}-u_{k4}^{\prime }}{(u_{k4}^{\prime }-u_0)r_{k4}^2}\right]\frac{1}{(r_{j4}^2-r_{k4}^2)}\end{array}$$

(15)

These values of the parameters are only preliminary. However, putting them into (13), we can rectify the ideal and undistorted image coordinates, which is near the center of the image and considered identical with the actual image coordinates. The ideal and undistorted image coordinates of the points away from the central are obtained again, and are used for the calculation of the parameters.

In order to obtain more accurate image coordinates of the optical axis center point and aberration coefficient, more ideal image points without aberration need to be used, by combining two and two, and then using Equation (15) to find the parameters respectively. In order to ensure the stability of the parameters, the parameters can be obtained several times using different combinations and eventually obtaining the mean value of each parameter. The mean value of each parameter is obtained as follows.

$$\{\begin{array}{c}{\overline{u}}_0=\frac{1}{n}\sum _{q=1}^n{u}_{0q}\\ {\overline{v}}_0=\frac{1}{n}\sum _{q=1}^n{v}_{0q}\\ {\overline{k}}_1=\frac{1}{n}\sum _{q=1}^n{k}_{1q}\\ {\overline{k}}_2=\frac{1}{n}\sum _{q=1}^n{k}_{2q}\end{array}$$

(16)

In Equation (16), $u_{0q}$, $v_{0q}$, $k_{1q}$, and $k_{2q}$ denote the value of the parameter for the qth combination, while ${\overline{u}}_0$, ${\overline{v}}_0$, ${\overline{k}}_1$, and ${\overline{k}}_2$ denote the average value of each parameter. Since there is a difference between the actual image coordinates of the image point in the middle of the image and the ideal image coordinates without distortion, according to Equation (12), the actual image coordinates of the image point in the middle of the image need to be corrected, and the corrected image coordinates are used to find the ideal image coordinates without distortion away from the image center. Therefore, the actual image coordinates of ${\overline{u}}_0$, ${\overline{v}}_0$, ${\overline{k}}_1$, ${\overline{k}}_2$ and the actual image coordinates of the image point in the middle of the image $(u_A,v_A)$, $(u_B,v_B)$ and $(u_C,v_C)$ are substituted into Equation (13) to find the distortion-free ideal image coordinates $(u_A^{\prime },v_A^{\prime })$, $(u_B^{\prime },v_B^{\prime })$, and $(u_C^{\prime },v_C^{\prime }).$ The above distortion-free ideal image coordinates are then used to obtain the coordinates of the distortion-free ideal image points $(u_D^{\prime },v_D^{\prime })$ away from the center of the image again. The average parameter values are obtained again using Equations (15) and (16). In order to obtain the exact parameters, several cycles are required until the parameters converge.

3. Image Distortion Correction

Because of the image distortion, it is difficult to make exact image information analysis. In view of this, we believe image correction is very necessary. From (13), we know that it is a dual higher order equation. Therefore, solving the ideal undistorted coordinates (u, v) directly is very difficult. According to u₀, v₀, k₁, k_2, and the image resolution ratio, we can estimate the coordinate range of the ideal and undistorted image points. In this range, an actual point corresponds to one or more ideal points. Thus, (13) is rewritten as follows.

$$\{\begin{array}{c}u=(u^{\prime }-u_0)(1+k_1{r}^2+k_2{r}^4)+u_0\\ v=(v^{\prime }-v_0)(1+k_1{r}^2+k_2{r}^4)+v_0\end{array}$$

(17)

The ideal and undistorted image points in estimated coordinate range are put into (17) in a certain order. Thus, we can obtain a series of real image coordinates. The ideal and undistorted image coordinates (1, 1), (1, 2) … (m, n) are used to calculate in (17), so the real image coordinates (u₁, v₁), (u₂, v₂) … (u_m, v_n) are obtained. In other words, the corresponding ideal and undistorted image points of the real image coordinates (u₁, v₁), (u₂, v₂) … (u_m, v_n) are found. In this way, one or more ideal and undistorted image points can correspond with a real image point. Therefore, this method is very simple without any additional processing.

4. Getting Other Parameters of Camera

An image point is denoted by [u, v]^T, and the corresponding spatial point in the camera coordinate system is expressed as [x_c, y_c, z_c]^T. M_in is the matrix of camera intrinsic parameters. Then we have

$$z_c\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]={\mathit{M}}_{in}\left[\begin{array}{c}{x}_c\\ y_c\\ z_c\end{array}\right]\mathrm{with}{\mathit{M}}_{in}=\left[\begin{array}{ccc}{k}_x& 0& u_0\\ 0& k_y& v_0\\ 0& 0& 1\end{array}\right]$$

(18)

In the world coordinate system, the 3D point can be expressed as [x_w, y_w, z_w]^T. Because of the planar target, we assume: z_w = 0. The ^cM_w is the matrix of camera external parameters.

$$z_c\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]={\mathit{M}}_{in}{}^c\mathit{M}_w\left[\begin{array}{c}{x}_w\\ x_y\\ 1\end{array}\right]\mathrm{with}{}^c\mathit{M}_w=\left[\begin{array}{ccc}{n}_x& o_x& p_x\\ n_y& o_y& p_y\\ n_z& o_z& p_z\end{array}\right]$$

(19)

From (19), the results of calculation are as follows:

$$z_c\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\mathit{M}\left[\begin{array}{c}{x}_w\\ x_y\\ 1\end{array}\right]\mathrm{with}\mathit{M}=\left[\begin{array}{ccc}{k}_x{n}_x+n_z{u}_0& k_x{o}_x+o_z{u}_0& k_x{p}_x+p_z{u}_0\\ k_y{n}_y+n_z{v}_0& k_y{o}_y+o_z{v}_0& k_y{p}_y+p_z{v}_0\\ n_z& o_z& p_z\end{array}\right]$$

(20)

We assume:

$$\mathit{M}=\left[\begin{array}{ccc}{m}_{11}& m_{12}& m_{13}\\ m_{21}& m_{22}& m_{23}\\ m_{31}& m_{32}& m_{33}\end{array}\right]$$

(21)

According to (20) and (21), we can obtain the following equations:

$$\{\begin{array}{c}{z}_{c}u=m_{11}{x}_w+m_{12}{y}_w+m_{13}\\ z_{c}v=m_{21}{x}_w+m_{22}{y}_w+m_{23}\\ z_c=m_{31}{x}_w+m_{32}{y}_w+m_{33}\end{array}$$

(22)

From (22), through the elimination of z_c, we have

$$\{\begin{array}{c}{m}_{11}{x}_w+m_{12}{y}_w+m_{13}-u{m}_{31}{x}_w-u{m}_{32}{y}_w=u{m}_{33}\\ m_{21}{x}_w+m_{22}{y}_w+m_{23}-v{m}_{31}{x}_w-v{m}_{32}{y}_w=v{m}_{33}\end{array}$$

(23)

We can use (x_wi, y_wi, 0) and (u_i, v_i) to represent the scene point in the world coordinate and image coordinate, and each of these points satisfies the Equation (23). Therefore, if there are n points, we will get the following equations set, which is composed of 2 × n equations:

$$\mathit{A}\mathit{m}=m_{33}\mathit{B}$$

(24)

In (24), with $\mathit{A}=\left[\begin{array}{cccccccc}{x}_{w1}& y_{w1}& 1& 0& 0& 0& -u_1{x}_{w1}& -u_1{y}_{w1}\\ 0& 0& 0& x_{w1}& y_{w1}& 1& -v_1{x}_{w1}& -v_1{x}_{w1}\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ x_{wn}& y_{wn}& 1& 0& 0& 0& -u_n{x}_{wn}& -u_n{y}_{wn}\\ 0& 0& 0& x_{wn}& y_{wn}& 1& -v_n{x}_{wn}& -v_n{x}_{wn}\end{array}\right]$,

$\mathit{B}={[\begin{array}{ccccc}{u}_1& v_1& \cdots & u_n& v_n\end{array}]}^T$, $\mathit{m}={\left[\begin{array}{cccccccc}{m}_{11}& m_{12}& m_{13}& m_{21}& m_{22}& m_{23}& m_{31}& m_{32}\end{array}\right]}^T$

When M′ = m/m₃₃, (24) can be changed into (25).

$$\mathit{A}{\mathit{M}}^{\prime }=\mathit{B}$$

(25)

Using the least square method, we have

$${\mathit{M}}^{\prime }={({\mathit{A}}^{\mathit{T}}\mathit{A})}^{-1}{\mathit{A}}^{\mathit{T}}\mathit{B}$$

(26)

The a_ij (i, j = 1, 2, 3) is used to represent the elements in the matrix M′. We have: a_ij = m_ij/m₃₃. Using the relationship between (20) and (21), we can find a_ij = m_ij/p_z and obtain the following equations.

$$\{\begin{array}{c}\frac{n_x}{p_z}=\frac{a_{11}-a_{31}{u}_0}{k_x}\\ \frac{n_y}{p_z}=\frac{a_{21}-a_{31}{v}_0}{k_y}\\ \frac{o_x}{p_z}=\frac{a_{12}-a_{32}{u}_0}{k_x}\\ \frac{o_y}{p_z}=\frac{a_{22}-a_{32}{v}_0}{k_y}\end{array}$$

(27)

Because the vectors [n_x, n_y, n_z]^T and [o_x, o_y, o_z]^T are unit vectors and orthogonal to each other. We have:

$$\{\begin{array}{c}{n}_x^2+n_y^2+n_z^2=1\\ o_x^2+o_y^2+o_z^2=1\\ n_x{o}_x+n_y{o}_y+n_z{o}_z=0\end{array}$$

(28)

From (28), we can make an appropriate deformation.

$$\{\begin{array}{c}{(\frac{n_x}{p_z})}^2+{(\frac{n_y}{p_z})}^2+{(\frac{n_z}{p_z})}^2={(\frac{1}{p_z})}^2\\ {(\frac{o_x}{p_z})}^2+{(\frac{o_y}{p_z})}^2+{(\frac{o_z}{p_z})}^2={(\frac{1}{p_z})}^2\\ (\frac{n_x}{p_z})(\frac{o_x}{p_z})+(\frac{n_y}{p_z})(\frac{n_y}{p_z})+(\frac{n_z}{p_z})(\frac{o_z}{p_z})=0\end{array}$$

(29)

Putting (27) into (29), we have

$$\{\begin{array}{c}{A}_1{}^{2}x+B_1{}^{2}y-z+a_{31}^2=0\\ A_2{}^{2}x+B_2{}^{2}y-z+a_{32}^2=0\\ A_1{A}_{2}x+B_1{B}_{2}y+a_{31}{a}_{32}=0\end{array}$$

(30)

In (30), x = (1/k_x)², y = (1/k_y)², z = (1/p_z)², A₁ = a₁₁ − a₃₁u₀, B₁ = a₂₁ − a₃₁v₀, A₂ = a₁₂ − a₃₂u₀, B₂ = a₂₂ − a₃₂v₀. Because |D| ≠ 0, we have:

$$\left(\begin{array}{c}x\\ y\\ z\end{array}\right)=-{\mathit{D}}^{-1}\left(\begin{array}{c}{a}_{31}^2\\ a_{32}^2\\ a_{31}{a}_{32}\end{array}\right)\mathrm{with}\mathit{D}=\left(\begin{array}{ccc}{A}_1^2& B_1^2& -1\\ A_2^2& B_2^2& -1\\ A_1{A}_2& B_1{B}_2& 0\end{array}\right)$$

(31)

As the values of u₀, v₀ and M′ have been calculated in (15) and (26), putting these values into (31), we can obtain the values of x, y, and z. According to the camera imaging characteristics, we know the values of k_x, k_y, and p_z are greater than zero. Therefore, it is not difficult to obtain the values of k_x, k_y, and p_z by (30).

$$\{\begin{array}{c}{k}_x=1/\sqrt{x}\\ k_y=1/\sqrt{y}\\ p_z=1/\sqrt{z}\end{array}$$

(32)

From (19) and (20), we can get the following relationship.

$${\mathit{M}}_{in}{}^c\mathit{M}_w=\mathit{M}$$

(33)

The matrix M_in and matrix ^cM_w are intrinsic and extrinsic parameter matrixes of the camera. The matrix M_in contains four parameters, which are obtained in (15) and (32). Because the matrix M_in has an inverse matrix, the intrinsic matrix ^cM_w can be expressed as follows:

$${}^c\mathit{M}_w={\mathit{M}}_{in}^{-1}\mathit{M}$$

(34)

Based on the above derivation, the method described in this paper focuses on obtaining the ideal coordinates without distortion by means of the central reference point of the image and the projection principle, thus obtaining the parameters u₀, v₀, k₁, k₂ and ultimately the other parameters of the camera. This is shown in Figure 2.

5. Experiments

5.1. Experimental Equipment

The main equipment for the ultra-wide-angle camera calibration experiments consisted of a Dell Inspion N4030 laptop, an ultra-wide-angle HD camera from Ryota Technology, a multi-square flat target calibration board, a fill light, a correction board, etc., as shown in the Figure 3. Figure 3a shows the laptop and the super wide angle camera, which are used for computer calibration processing and image acquisition, respectively. Figure 3b shows the multi-square flat target calibration board, which is the flat target for super wide angle camera calibration. Figure 3c shows the fill light, which is used for fill light processing when the target image is acquired. Figure 3d shows the aberration correction board, which is used for aberration correction experiments.

5.2. Ultra-Wide-Angle Camera Calibration Process

The ultra-wide-angle camera calibration experiment consisted of the following processes.

(1)

Prepare a flat board of targets with equally spaced black squares.

(2)

The image of the target is captured at 1280 × 1024 pixels, as shown in Figure 4a.

(3)

The coordinates of the centroid of each black square in the target image were obtained using computer processing. In addition, a circular area is created with the centre of the image as the centre and 100 pixel points as the radius, and the coordinates of the centroids of all black squares within the circular area are obtained, as shown in Figure 4b.

(4)

The computer is used to find the combination of three points that satisfy in the same line, and to determine all other points on the same line that are far from the center of the image. Using Equation (12) to find the furthest ideal image coordinate points from the center without distortion, the image points found are shown in Figure 4b as $A_{14}^{\prime }$, $A_{24}^{\prime }$, $A_{34}^{\prime }$ and $A_{44}^{\prime }$.

(5)

Combine the image points $A_{14}^{\prime }$, $A_{24}^{\prime }$, $A_{34}^{\prime }$ and $A_{44}^{\prime }$ in two according to Equation (15) to find the image coordinates (u₀, v₀) and the distortion coefficients k₁ and k₂. The values of ${\overline{u}}_0,{\overline{v}}_0,{\overline{k}}_1$ and ${\overline{k}}_2$ are then obtained according to the obtained parameters using Equation (16).

(6)

Using the average parameter values obtained, the image points on the line involved in the calculation are corrected according to Equation (17).

(7)

Repeat the steps 4, 5, and 6 until the desired parameters converge.

(8)

The camera is calibrated by using Equations (32)–(34) to find the other parameters of the camera based on the image coordinates of the optical axis center point and the distortion coefficient obtained.

(9)

The distorted image is corrected using the camera parameters acquired in step 8.

(10)

The camera calibration parameters were verified using the Zhang Zhengyou flat calibration method [26].

5.3. Results of Experiments

According to Equation (12) we can obtain the distortion-free ideal point image coordinates, and the relevant calculation results are shown in

Table 1. The relationship between coordinate parameters and iterations.

Number of Iterations	${\mathit{A}}_{\mathbf{14}}^{\mathbf{\prime }}$		${\mathit{A}}_{\mathbf{24}}^{\mathbf{\prime }}$		${\mathit{A}}_{\mathbf{34}}^{\mathbf{\prime }}$		${\mathit{A}}_{\mathbf{44}}^{\mathbf{\prime }}$
	${\mathit{u}}_{\mathbf{14}}^{\mathbf{\prime }}$	${\mathit{v}}_{\mathbf{14}}^{\mathbf{\prime }}$	${\mathit{u}}_{\mathbf{24}}^{\mathbf{\prime }}$	${\mathit{v}}_{\mathbf{24}}^{\mathbf{\prime }}$	${\mathit{u}}_{\mathbf{34}}^{\mathbf{\prime }}$	${\mathit{v}}_{\mathbf{34}}^{\mathbf{\prime }}$	${\mathit{u}}_{\mathbf{44}}^{\mathbf{\prime }}$	${\mathit{v}}_{\mathbf{44}}^{\mathbf{\prime }}$
1	53.91	−92.53	1145.24	−35.25	1130.92	1048.54	142.65	1061.38
2	24.83	−123.48	1143.43	−31.78	1116.75	1033.76	135.73	1070.73
3	31.52	−116.36	1144.05	−32.74	1120.17	1037.26	136.99	1068.96
4	30.25	−117.72	1143.98	−32.59	1119.48	1036.58	136.78	1069.24
5	30.48	−117.50	1143.97	−32.60	1119.58	1036.62	136.80	1069.21
6	30.45	−117.51	1143.93	−32.61	1119.58	1036.67	136.80	1069.26
7	30.45	−117.51	1143.93	−32.61	1119.58	1036.67	136.80	1069.26
8	30.45	−117.51	1143.93	−32.61	1119.58	1036.67	136.80	1069.26

.We can easily find that the distortion-free ideal image coordinates of $A_{14}^{\prime }$, $A_{24}^{\prime }$, $A_{34}^{\prime }$ and $A_{44}^{\prime }$ gradually smooth out with the increase of the number of iterations. Using Equations (15) and (16), the ${\overline{u}}_0,{\overline{v}}_0,{\overline{k}}_1$ and ${\overline{k}}_2$ are found, as shown in Figure 5. According to the convergence relationship between the image coordinates of the distortion-free ideal point at the edge of the image, the image coordinates of the center point of the optical axis, the distortion coefficient and the number of iterations, it can be seen that ${\overline{u}}_0,{\overline{v}}_0,{\overline{k}}_1$ and ${\overline{k}}_2$ can be obtained exactly after only five iterations of calculation, so the whole processing process is very fast.

The final camera internal reference matrix ${\mathit{M}}_{in}$ and external reference matrix ${}^c\mathit{M}_w$ were obtained from the ultra-wide-angle camera calibration experiments as follows.

$${\mathit{M}}_{in}=\left[\begin{array}{ccc}{k}_x& 0& u_0\\ 0& k_y& v_0\\ 0& 0& 1\end{array}\right]=\left[\begin{array}{ccc}2279.10& 0& 647.30\\ 0& 2759.30& 542.05\\ 0& 0& 1\end{array}\right]$$

(35)

$${}^c\mathit{M}_w=\left[\begin{array}{ccc}{n}_x& o_x& p_x\\ n_y& o_y& p_y\\ n_z& o_z& p_z\end{array}\right]=\left[\begin{array}{ccc}-0.9938& -0.0970& -56.0231\\ 0.0953& -0.9949& -54.2775\\ -0.0577& 0.0277& 239.7322\end{array}\right]$$

(36)

In this paper, the image correction is carried out according to the requested aberration coefficient and the image coordinates of the optical axis center point, and a good correction effect is obtained, as shown in Figure 6. This shows that the calibration method described in this paper has certain engineering application value.

A comparison of the camera parameters obtained in this paper with those obtained using Zhang’s calibration method is shown in

Table 2. Comparison of the camera parameters with different algorithms.

Parameters	Algorithms in This Paper	Zhang’s Algorithm	Relative Error
u0	647.30	646.81	0.08%
v0	542.05	540.23	0.34%
k1	−5.9035 × 10−7	−5.9325 × 10−7	0.44%
k2	2.3674 × 10−13	2.3891 × 10−13	0.91%
kx	2279.10	2283.04	0.17%
ky	2759.30	2770.13	0.39%

. The comparison shows that the values obtained are consistent, indicating that the algorithm described in this paper is able to obtain more accurate camera parameters. Therefore, the calibration method for ultra-wide-angle cameras proposed in this paper has strong application value and can be further applied in industry and agriculture.

6. Conclusions

In this paper, we have proposed a new calibration method, which is obviously different from others. In the experiment, this method has been tested from different observation angles and distances, and the results were satisfactory. Beginning with a few points in the center of the image, whose distortion is weaker, more accurate experimental data can be ensured. Therefore, the position of the planar plane is very important. In the experiment and simulation, the ratios of length are both D₁B₁:C₁B₁:B₁A₁ = 5.5:1:1 and are considered reasonable in the experiment. To obtain accurate data, more points are added in the planar plane, and repeated experiments have been carried out.

This method is suitable for a camera with obvious distortion, and only requires the camera to take one picture in a certain position. In addition, the algorithm described in this paper requires only one image acquisition for the entire calibration process, and after the image is acquired, no manual identification or guidance is required, and the ultra-wide-angle camera calibration can be automated by a computer program. In addition, only the aberration parameters and the image coordinates of the optical axis center point are iteratively calculated during the calculation process, and the number of iterations is low, making the whole calibration process faster than Zhang’s calibration method. As can be seen, the specific advantages of the algorithm proposed in this paper include the following.

(1)

Only one image acquisition of the target is required.

(2)

No expensive ancillary equipment is required and it is highly adaptable.

(3)

High calibration progress within 1% relative error.

(4)

Rapid calibration.

Therefore, the algorithm proposed in this paper has strong application value and is suitable for promotion in the field of robot vision in agriculture and industry.