A Three-Dimensional Reconstruction Method Based on Telecentric Epipolar Constraints

Abstract

When calibrating a microscopic fringe projection profile system with a telecentric camera, the orthogonality of the camera causes an ambiguity in the positive and negative signs of its external parameters. A common solution is to introduce additional constraints, which often increase the level of complexity and the calibration cost. Another solution is to abandon the internal/external parameter models derived from the physical imaging process and obtain a numerically optimal projection matrix through the least squares solution. This paper proposes a novel calibration method, which derives a telecentric epipolar constraint model from the conventional epipolar constraint relationship and uses this constraint relationship to complete the stereo calibration of the system. On the one hand, since only the camera’s intrinsic parameters are needed, there is no need to introduce additional constraints. On the other hand, the solution is optimized based on the full consideration of the imaging model to make the parameters confirm to the physical model. Our experiments proved the feasibility and accuracy of the method.

1. Introduction

With the rapid upgrading of the microelectronics industry, an increasing number of devices have reached the micrometer scale, demanding a higher level of precision for surface mount technology (SMT). Traditional contact detection methods, such as micro-CMMs (coordinate measurement machines) [1,2], can no longer meet the requirements for fast, high-precision, and non-destructive detection in the microelectronics manufacturing industry. Among the non-contact optical measurement methods, con-focal microscopy [3] and white-light interference microscopy [4,5] have the advantages of a high measurement accuracy, but their complex structure and high price make them unsuitable for harsh industrial environments [6]. Fringe projection profilometry (FPP) uses phase information to obtain high-precision, high-density three-dimensional topography [7]. Due to its insensitivity to changes in background, contrast, and noise, as well as its ability to acquire full-field data, its flexibility, and its low price [8,9], it has been widely used in industrial inspection, reverse engineering, plastic surgery, cultural relic evaluation, entertainment, and other fields [10,11]. Microscale FPP, also known as fringe projection three-dimensional microscopy (FP-3DM), plays an increasingly important role in micromanufacturing and roughness measurement [12]. Traditional stereo microscopes based on FP-3DM are often modified to achieve micron-level measurements [13,14].

However, the depth of field (DOF) of conventional microscopic profilometry systems is limited to the sub-millimeter level, which is insufficient for measuring three-dimensional objects with height variations of several millimeters. In addition, conventional lenses suffer from perspective effects and lens distortion, which can cause objects to appear distorted at short distances [15]. In contrast, telecentric lenses have many unparalleled advantages, such as a high resolution, almost no distortion, constant magnification, and an increased depth of field [16,17,18,19,20]. Since telecentric lenses can eliminate parallax, determine the exact size of objects, and produce a high measurement accuracy [21], they are very suitable for demanding machine vision applications [22,23,24] and the high-precision three-dimensional (3D) reconstruction of small objects [25,26,27].

The existing FPP system calibration methods can be roughly divided into triangular stereo models and phase–height models [28]. The phase–height model fits the relationship between the absolute phase and the three-dimensional coordinates through a calibration. The calibration process is relatively complicated and requires a precise translation platform and reference plane [29]. The triangular stereo model calibrates cameras of different perspectives and finally completes the reconstruction through the matching of points with the same name. Due to the orthogonal imaging model of telecentric lenses, the pinhole model calibration method is no longer applicable. Therefore, many methods for telecentric lens calibration have been proposed. They can be roughly divided into three categories: self-calibration [30,31,32], three-dimensional-target-based calibration, and planar-target-based calibration. The self-calibration method does not require a calibration target and can calibrate a moving camera directly from the scene image. This method is flexible but sensitive to noise and is not suitable for a fixed camera. The three-dimensional-target-based calibration method is simple and direct, but the micro-scale three-dimensional calibration cube suitable for telecentric lenses is very expensive. The planar-target-based method is most suitable for telecentric cameras because it is easy to apply and has good accuracy. Li Dong et al. proposed a two-step calibration method that takes into account the calibration accuracy under different distortion models [33]. It is widely used due to its flexibility and accuracy [34,35,36]. For example, Li Rao et al. regard the telecentric camera as an entocentric system with a large focal length and assume that the external parameters except $t_Z$ are unchanged in terms of both perspective and the affine projection transformation. Haskamp et al. directly use nonlinear optimization methods to estimate camera parameters [37], but this method is likely to fail to reach the global minimum. Yin et al. introduced a general imaging model and moved the calibration plate in a certain way during the calibration process to complete the calibration of a telecentric camera [12]. In refs. [38,39], a telecentric camera was calibrated by a simple and easy-to-implement linear fitting, and the relationship in the Z direction was determined by the movement of the micro-displacement platform. Based on the factorization of ref. [40], Lanman and Liu et al. successfully obtained an initial estimate of a camera’s internal and external parameters [41,42]. There is a problem in the telecentric calibration method based on planar objects; that is, there is sign ambiguity in the external parameters. In Lanman’s method, a checkerboard plane with a known height is introduced to restore the sign. Inspired by Zhang [43], Chen et al. [44] also proposed a closed-form solution method for parameters based on orthogonality, which uses a micro-displacement platform to move along the $Z$-axis to eliminate ambiguity. Many authors [45,46,47,48,49] have also proposed their own calibration methods on this basis. Interestingly, Chao Chen et al. [46] used a virtual calibration plate to avoid the use of a micro-displacement platform. Most methods introduce a micro-displacement platform and calibrate the internal and external parameters separately, which makes the calibration complicated and expensive. In refs. [6,50], the authors believe that the internal and external parameters of the telecentric lens are naturally coupled together and difficult to separate directly. Separate calibration leads to uncertainties in the accuracy of the imaging model; that is, the model itself is not robust enough. They convert the calibration corner points to the projector coordinate system (PCS), directly calculate the projection matrix formed by the coupling of the internal and external parameters, and thus complete the stereo calibration, avoiding the problem of sign ambiguity.

In this paper, we propose a flexible calibration method based on telecentric epipolar constraints and develop a three-dimensional reconstruction approach using it. In our method, we derive the telecentric epipolar constraint, using it to describe the poses between a camera and a projector. The issue of symbol ambiguity can be avoided since the extrinsic parameters of the telecentric camera are not needed when solving for the telecentric essential matrix; thus, no additional devices are required. Finally, the telecentric essential matrix does not conform to the form of an antisymmetric matrix multiplied with an orthogonal matrix. Therefore, a new method is introduced to ensure orthogonality between the rotation elements. The experimental results demonstrate the feasibility and accuracy of our approach.

A brief introduction to the calibration principles for cameras and projectors is in Section 2. The principle of telecentric epipolar constraints is described in detail in Section 3. We also briefly introduce the reconstruction method based on the projection matrix proposed by Zhang et al. [45]. as we will compare this method to our method in subsequent experiments. Finally, Section 4 and Section 5 cover the experiments and conclusion, respectively.

2. System Calibration

The calibration of pinhole models and telecentric models has already been well developed. This section will briefly introduce the methods used in this paper for the calibration of the camera and projector, respectively. However, we do not calibrate the extrinsic parameters for the telecentric camera.

2.1. Calibration for Telecentric Lens

The imaging model for telecentric lenses can be described as follows:

$$\begin{array}{c}\left[\begin{array}{c}{u}^c\\ v^c\\ 1\end{array}\right]=\left[\begin{array}{cccc}m& 0& 0& 0\\ 0& m& 0& 0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}{r}_{11}^c& r_{12}^c& r_{13}^c& t_x^c\\ r_{21}^c& r_{22}^c& r_{23}^c& t_y^c\\ r_{31}^c& r_{32}^c& r_{33}^c& t_z^c\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]\end{array}$$

(1)

where $m$ represents the magnification of the telecentric lens, ${\left[\begin{array}{cccc}{X}_w& Y_w& Z_w& 1\end{array}\right]}^T$ represents the homogeneous coordinates of a point in the world coordinate system (WCS), ${\left[\begin{array}{ccc}{u}^c& v^c& 1\end{array}\right]}^T$ represents the corresponding camera pixel coordinates, $r_{11}~r_{33}$ represent the rotation components, and $t_x,t_y,t_z$ represent the translation components. Additionally, the superscript $c$ represents the external parameters from the WCS to the telecentric camera coordinate system (CCS). Assuming $Z_w=0$, the equation above can be expressed in a more simplified form as follows:

$$\begin{array}{c}\left[\begin{array}{c}{u}^c\\ v^c\\ 1\end{array}\right]=\left[\begin{array}{ccc}m& 0& 0\\ 0& m& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}{r}_{11}^c& r_{12}^c& t_x^c\\ r_{21}^c& r_{22}^c& t_y^c\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ 1\end{array}\right]=M_s{K}_s{P}_s=H_s{P}_s\end{array}$$

(2)

where $M_s$ and $K_s$ correspond to the two matrices before the equal signs, respectively. $H_s=M_s{K}_s=\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ 0& 0& 1\end{array}\right]$, which can be easily solved through a direct linear transformation (DLT). In light of the rotation matrix in Equation (1) and leveraging its orthogonality, we obtain the following:

$$\begin{array}{c}\left\{\begin{array}{c}{r}_{11}^c{r}_{21}^c+r_{12}^c{r}_{22}^c+r_{13}^c{r}_{23}^c=0\\ r_{13}^c=\pm \sqrt{1-{r_{11}^c}^2-{r_{12}^c}^2}\\ r_{23}^c=\pm \sqrt{1-{r_{21}^c}^2-{r_{22}^c}^2}\end{array}\right.\end{array}$$

(3)

Combining Equations (2) and (3), a relation about the square of the magnification $m^2$ can be expressed as follows:

$$\begin{array}{c}{m}^4-\left(h_{21}^2+h_{22}^2+h_{11}^2+h_{12}^2\right)m^2+{\left(h_{11}{h}_{22}-h_{12}{h}_{21}\right)}^2=0\end{array}$$

(4)

By solving Equation (4), we obtain four solutions, among which the two non-negative solutions are meaningful. The final magnification can be determined through ${r_{ij}^c}^2=h_{ij}^2/m^2\le 1$, where $i=\mathrm{1,2},j=\mathrm{1,2}$.

2.2. Projector Calibration

The imaging model for the projectors follows the pinhole model, which is commonly calibrated using Zhang’s [43] calibration method. The key to projector calibration is enabling the projector to “see” corners like a camera. To achieve this, the projector projects a pattern and the camera captures it simultaneously. This indirect process allows the projector to effectively acquire image data.

In this paper, we employ an eight-step phase-shifting method combined with complementary Gray codes to enhance levels of robustness in stripe boundary decoding. The imaging model for the projector can be described as follows:

$$\begin{array}{c}\left[\begin{array}{c}{u}^p\\ v^p\\ 1\end{array}\right]=s\left[\begin{array}{cccc}{f}_x& 0& u_0& 0\\ 0& f_y& v_0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{cccc}{r}_{11}^p& r_{12}^p& r_{13}^p& t_x^p\\ r_{21}^p& r_{22}^p& r_{23}^p& t_y^p\\ r_{31}^p& r_{32}^p& r_{33}^p& t_z^p\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]\end{array}$$

(5)

where $s$ is a projection coefficient, $f_x,u_0$ and $f_y,v_0$ represent the focal length and principal point of the projector in the x and y directions, ${\left[\begin{array}{ccc}{u}^p& v^p& 1\end{array}\right]}^T$ represents the pixel coordinates in the projector, and the superscript $p$ in the external parameter indicates that it is transformed from the WCS to the PCS.

Assuming that the calibration pattern lies on $Z_w=0$, the imaging model can be presented as follows:

$$\begin{array}{c}\left[\begin{array}{c}{u}^p\\ v^p\\ 1\end{array}\right]=sM\left[\begin{array}{ccc}{\mathit{r}}_{\mathbf{1}}& {\mathit{r}}_{\mathbf{2}}& \mathit{t}\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ 1\end{array}\right]=\left[\begin{array}{ccc}{\mathit{h}}_{\mathbf{1}}& {\mathit{h}}_{\mathbf{2}}& {\mathit{h}}_{\mathbf{3}}\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ 1\end{array}\right]=H\left[\begin{array}{c}{X}_w\\ Y_w\\ 1\end{array}\right]\end{array}$$

(6)

where $M$ represents the internal matrix, ${\mathit{r}}_{\mathbf{1}},{\mathit{r}}_{\mathbf{2}}$ denote the vectors of the first and second columns in the rotation matrix, and $\mathit{t}$ is the translational vector. Furthermore, $\left[\begin{array}{ccc}{\mathit{h}}_{\mathbf{1}}& {\mathit{h}}_{\mathbf{2}}& {\mathit{h}}_{\mathbf{3}}\end{array}\right]$ represents each column of the homology matrix $H$, which can be determined using DLT.

Using the orthogonality of the rotation matrix in Equation (5), the following equations can be obtained:

$$\begin{array}{c}\left\{\begin{array}{c}{\mathit{r}}_{\mathbf{1}}^{\mathit{T}}{\mathit{r}}_{\mathbf{2}}=0={\mathit{h}}_{\mathbf{1}}^{\mathit{T}}{M}^{-T}{M}^{-1}{\mathit{h}}_{\mathbf{2}}\\ {\mathit{r}}_{\mathbf{1}}^{\mathit{T}}{\mathit{r}}_{\mathbf{1}}=1={\mathit{h}}_{\mathbf{1}}^{\mathit{T}}{M}^{-T}{M}^{-\mathbf{1}}{\mathit{h}}_{\mathbf{1}}\end{array}\right.\end{array}$$

(7)

Let $B=M^{-T}{M}^{-1}=\left[\begin{array}{ccc}{B}_{11}& B_{12}& B_{13}\\ B_{12}& B_{22}& B_{23}\\ B_{13}& B_{23}& B_{33}\end{array}\right]$. By inserting this into $h_i^{T}B{h}_j$ and expanding it, we can obtain the following:

$$\begin{array}{c}{h}_i^{T}B{h}_j={\left[\begin{array}{c}{h}_{i1}{h}_{j1}\\ h_{i1}{h}_{j2}+h_{i2}{h}_{j1}\\ \begin{array}{c}{h}_{i2}{h}_{j2}\\ h_{i3}{h}_{j1}+h_{i1}{h}_{j3}\\ \begin{array}{c}{h}_{i3}{h}_{j2}+h_{i2}{h}_{j3}\\ h_{i3}{h}_{j3}\end{array}\end{array}\end{array}\right]}^T\left[\begin{array}{c}{B}_{11}\\ B_{12}\\ B_{22}\\ B_{13}\\ B_{23}\\ B_{33}\end{array}\right]={\mathit{v}}_{\mathit{i}\mathit{j}}^{\mathit{T}}\mathit{b}\end{array}$$

(8)

where ${\mathit{v}}_{\mathit{i}\mathit{j}}^{\mathit{T}}$ and $\mathit{b}$ correspond to the two vectors before them.

According to ${\mathit{r}}_{\mathbf{1}}^{\mathit{T}}{\mathit{r}}_{\mathbf{2}}=0,‖{\mathit{r}}_{\mathbf{1}}‖=‖{\mathit{r}}_{\mathbf{2}}‖$, Equation (8) becomes the following:

$$\begin{array}{c}\left[\begin{array}{c}{\mathit{v}}_{\mathbf{12}}^{\mathit{T}}\\ {{\mathit{v}}_{\mathbf{11}}^{\mathit{T}}-\mathit{v}}_{\mathbf{22}}^{\mathit{T}}\end{array}\right]\mathit{b}=0\end{array}$$

(9)

After solving Equation (9) and obtaining $\mathit{b}$, the elements of the intrinsic and extrinsic matrices can be determined by $B=M^{-T}{M}^{-1}$ and Equation (6) easily.

3. Telecentric Epipolar Constraints

According to conventional epipolar constraints, there does not seem to have the same relationship between telecentric and pinhole models. However, we can still observe a certain constraint between them and derive it using equations.

3.1. Principle

The conventional epipolar constraint is discussed first. As shown in Figure 1, $O_1,O_2$ represent the optical centers of the camera and projector, respectively. $P$ is an object point in space, $p_1,p_2$ are the projections of point $P$, and $e_1,e_2$ are the points coming from the intersection of the line $O_1{O}_2$ with the image planes. Connect $e_1{,p}_1$ and $e_2{,p}_2$ to form $l_1$ and $l_2$. If $p_1$ is known, its corresponding point must lie on the intersection of the plane $O_1{O}_{2}P$ with the right image plane, which is referred to as the epipolar constraint.

As shown in Figure 2, for an object point $P_W$ in space, there is a telecentric CCS $O_c$-$X_c{Y}_c{Z}_c$ and a PCS $O_p$-$X_p{Y}_p{Z}_p$. In practice, the optical center of camera is located at infinity due to the telecentric lens, preventing the formation of a baseline. Meanwhile, $P_W$ is imaged at $p_l$ in the camera image and at $p_r$ in the projector image. Assuming that the optical center of the projector $O_p$ is imaged at the point $O_c^{\prime }$ in the telecentric camera, we can still observe the geometric relations: a point on the camera image plane corresponds to a line on the projector image plane; that is, point $p_l$’s corresponding point $p_r$ lies on the line $L_r$.

Following the form of the conventional epipolar constraint, the telecentric epipolar constraint is defined in the same way as follows:

$$\begin{array}{c}\begin{array}{c}{P}_l^{T}E{P}_r=0\\ p_l^{T}F{p}_r=0\end{array}\end{array}$$

(10)

Among them, $E$ and $F$ represent the essential matrix and fundamental matrix. $P_l$ and $P_r$ are the points of $P_W$ in the CCS and PCS, respectively, while $p_l$ and $p_r$ are the image points in the camera and the projector.

After the PCS is considered as the reference coordinate system (RCS), the projection equations for the camera and projector can be presented as follows:

$$\begin{array}{c}\begin{array}{c}{p}_l=M_l{K}_l{P}_w\\ {s_{r}p}_r=M_r{K}_r{P}_w\end{array}\end{array}$$

(11)

Among them, $M_l,K_l$ and $M_r{,K}_r$ are the internal and external matrix of camera and projector mentioned in Section 2, and $s_r$ is the coefficient in the projector imaging model.

Note that the rotation matrix $R_l$ and the translation vector $T_l$ in camera extrinsic matrix $K_l$ are expressed as follows:

$$\begin{array}{c}{R}_l=\left[\begin{array}{ccc}{r}_{11}^l& r_{12}^l& r_{13}^l\\ r_{21}^l& r_{22}^l& r_{23}^l\\ 0& 0& 0\end{array}\right],T_l=\left[\begin{array}{c}{t}_x^l\\ t_y^l\\ 1\end{array}\right]\end{array}$$

(12)

By substituting $P_w=R_r^{-1}\left(P_r-T_r\right)$ into $P_l=R_l{P}_w+T_l$, we can obtain the following:

$$\begin{array}{c}{P}_l=R_l{R}_r^{-1}{P}_r+\left(T_l-R_l{R}_r^{-1}{T}_r\right)\end{array}$$

(13)

Comparing this with $P_l=R{P}_r+T$, the rotation matrix and translation vector from the projector to the camera are given by $R=R_l{R}_r^{-1},T=T_l-R_l{R}_r^{-1}{T}_r$, respectively. It should be noted that the third row of $R$ and $T$ consists of $\mathbf{0}$ and $1$, respectively, due to Equation (12).

Perform a left multiplication on both sides of Equation (13) with matrix $\widehat{T}$, which is the antisymmetric matrix of $T$:

$$\begin{array}{c}\widehat{T}=\left[\begin{array}{ccc}0& -1& T_y\\ 1& 0& {-T}_x\\ {-T}_y& T_x& 0\end{array}\right]\end{array}$$

(14)

We can obtain $\widehat{T}{P}_l=\widehat{T}R{P}_r$ because $\widehat{T}T=\mathbf{0}$ and perform a left multiplication transformation with $P_l^T$ again. The final equation is presented as follows:

$$\begin{array}{c}{P}_l^T\widehat{T}R{P}_r=0\end{array}$$

(15)

Here, the telecentric essential matrix and the telecentric fundamental matrix are defined as $E=\widehat{T}R$ and $F={s_{r}M}_l^{-T}E{M}_r^{-1}$, respectively, and we obtain the following:

$$\begin{array}{c}\begin{array}{c}{P}_l^{T}E{P}_r=0\\ p_l^{T}F{p}_r=0\end{array}\end{array}$$

(16)

Expand matrix $F$ into vector $\mathit{F}={\left[\begin{array}{ccccccccc}{f}_{11}& f_{12}& f_{13}& f_{21}& f_{22}& f_{23}& f_{31}& f_{32}& f_{33}\end{array}\right]}^T$. Ideally, all corner points in the camera image $\left(u_l^i,v_l^i\right)$ and projector image $\left(u_r^i,v_r^i\right)$ should satisfy Equation (16), so the following overdetermined linear equations can be listed:

$$\left[\begin{array}{ccccccccc}{u}_l^1{u}_r^1& u_l^1{v}_r^1& u_l^1& v_l^1{u}_r^1& v_l^1{v}_r^1& v_l^1& u_r^1& v_r^1& 1\\ u_l^2{u}_r^2& u_l^2{v}_r^2& u_l^2& v_l^2{u}_r^2& v_l^2{v}_r^2& v_l^2& u_r^2& v_r^2& 1\\ & & & ⋮& & & & & \\ u_l^N{u}_r^N& u_l^N{v}_r^N& u_l^N& v_l^N{u}_r^N& v_l^N{v}_r^N& v_l^N& u_r^N& v_r^N& 1\end{array}\right]\left[\begin{array}{c}{f}_{11}\\ f_{12}\\ f_{13}\\ f_{21}\\ f_{22}\\ f_{23}\\ f_{31}\\ f_{32}\\ f_{33}\end{array}\right]=0$$

(17)

where $N$ represents the total number of corner points. Finding the least squares solution to Equation (17), we can obtain matrices $F$ and $E$.

3.2. Three-Dimensional Reconstruction Based on Telecentric Epipolar Constraint

To obtain the pose between the camera and projector, the telecentric essential matrix is extended according to $E=\widehat{T}R$:

$$\begin{array}{c}E=\left[\begin{array}{ccc}-r_{21}& -r_{22}& -r_{23}\\ r_{11}& r_{12}& r_{13}\\ r_{21}{t}_x-r_{11}{t}_y& r_{22}{t}_x-r_{12}{t}_y& r_{23}{t}_x-r_{13}{t}_y\end{array}\right]\end{array}$$

(18)

The components $r_{11}~r_{23},t_x,t_y$ in Equation (18) represent the external parameters from the projector to the camera. However, $r_{11}~r_{23}$ do not necessarily satisfy the orthogonality of the rotation matrix $R$ mentioned above. In the traditional epipolar constraint, a singular value decomposition (SVD) of the essential matrix is performed, and its singular value matrix is adjusted to $\Sigma =diag\left(\mathrm{1,1},0\right)$ to ensure orthogonality. However, the singular value matrix of the telecentric essential matrix we proposed does not satisfy the equation $\Sigma =diag\left(\mathrm{1,1},0\right)$ since it does not conform to the form of multiplying an antisymmetric matrix and an orthogonal matrix.

The singular values of the telecentric essential matrix are deduced below, assuming $\mathrm{E}=\mathrm{U}\Sigma {\mathrm{V}}^T$:

$$\begin{array}{c}{EE}^T=\widehat{T}R{R}^T{\widehat{T}}^T=U\Sigma V^{T}V{\Sigma }^T{U}^T=U{\Sigma }^2{U}^T\end{array}$$

(19)

which can be written in detail as follows:

$$\begin{array}{c}\widehat{T}R{R}^T{\widehat{T}}^T=\left[\begin{array}{ccc}1& 0& -t_x\\ 0& 1& -t_y\\ -t_x& -t_y& t_x^2+t_y^2\end{array}\right]\end{array}$$

(20)

The eigenvalues ${\lambda }_1=t_x^2+t_y^2+1,{\lambda }_2=1,{\lambda }_3=0$ can be obtained after solving Equation (20). So, its singular value matrix should be the following:

$$\begin{array}{c}\Sigma =diag\left(\sqrt{{\lambda }_1},\mathrm{1,0}\right)\end{array}$$

(21)

By performing an SVD on matrix $E$ and adjusting its singular value matrix into the form of $\Sigma =diag\left({\sigma }_1,\mathrm{1,0}\right)$, where ${\sigma }_1=\sqrt{{\lambda }_1}$, we can finally obtain the rotation vector ${\mathit{r}}_{\mathbf{1}}$, ${\mathit{r}}_{\mathbf{2}}$ and the translation vector $\mathit{t}$.

Based on the transformation of the RCS to the PCS, we combine the camera and projector projection equations for the three-dimensional reconstruction as follows:

$$\begin{array}{c}\left\{\begin{array}{c}\left[\begin{array}{c}\begin{array}{c}{u}_c\\ v_c\end{array}\\ 1\end{array}\right]=M_l{K}_l{P}_w=H_{3\times 4}{P}_w=\left[\begin{array}{ccc}{h}_{11}& h_{12}& \begin{array}{cc}{h}_{13}& h_{14}\end{array}\\ h_{21}& h_{22}& \begin{array}{cc}{h}_{23}& h_{24}\end{array}\\ 0& 0& \begin{array}{cc}0& 1\end{array}\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]\\ s_r\left[\begin{array}{c}\begin{array}{c}{u}_p\\ v_p\end{array}\\ 1\end{array}\right]=M_r{K}_r{P}_w=\left[\begin{array}{ccc}{f}_x& 0& \begin{array}{cc}{u}_0& 0\end{array}\\ 0& f_y& \begin{array}{cc}{v}_0& 0\end{array}\\ 0& 0& \begin{array}{cc}1& 0\end{array}\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]\end{array}\right.\end{array}$$

(22)

where the internal parameter matrices $M_l,M_r$ are obtained through the calibration steps described above. According to Equation (22), the equations can be established and written in matrix form as follows:

$$\begin{array}{c}\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ f_x& 0& u_0-u_p\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\end{array}\right]=\left[\begin{array}{c}{u}_c-h_{14}\\ v_c-h_{24}\\ 0\end{array}\right]\end{array}$$

(23)

By solving Equation (23), the three-dimensional coordinates can be obtained.

3.3. Three-Dimensional Reconstruction Based on Projection Matrix

Song Zhang’s method, which we mentioned above, couples the internal parameters and external parameters of the telecentric camera and calculates its projection matrix directly.

Because of this, we call it the three-dimensional reconstruction method based on the projection matrix.

Assuming that the projector has been calibrated through the method above, the corner points of the calibration board in all poses can be converted to the PCS through the external parameters of the projector and can be written as follows:

$$\begin{array}{c}{X}_n^i={\left[\begin{array}{cccc}{X}_p& Y_p& Z_p& 1\end{array}\right]}_{ni}^T\end{array}$$

(24)

where $n$ is the corner point number of each calibration board, and $i$ denotes the serial number of the calibration board in different poses. According to the imaging model of the telecentric camera, it can be expressed by the following equation:

$$\begin{array}{c}\left[\begin{array}{c}{u}_c\\ v_c\\ 1\end{array}\right]=\left[\begin{array}{ccc}{h}_{11}& h_{12}& \begin{array}{cc}{h}_{13}& h_{14}\end{array}\\ h_{21}& h_{22}& \begin{array}{cc}{h}_{23}& h_{24}\end{array}\\ 0& 0& \begin{array}{cc}0& 1\end{array}\end{array}\right]X_n^i\end{array}$$

(25)

which can also be rewritten as follows:

$$\begin{array}{c}\left\{\begin{array}{c}\left[\begin{array}{c}{X}_1^{1T}\\ X_2^{1T}\\ \begin{array}{c}⋮\\ X_n^{iT}\end{array}\end{array}\right]\left[\begin{array}{c}{h}_{11}\\ h_{12}\\ \begin{array}{c}{h}_{13}\\ h_{14}\end{array}\end{array}\right]=\left[\begin{array}{c}{u}_c^{\left(1\right)}\\ u_c^{\left(2\right)}\\ \begin{array}{c}⋮\\ u_c^{\left(ni\right)}\end{array}\end{array}\right]\\ \left[\begin{array}{c}{X}_1^{1T}\\ X_2^{1T}\\ \begin{array}{c}⋮\\ X_n^{iT}\end{array}\end{array}\right]\left[\begin{array}{c}{h}_{21}\\ h_{22}\\ \begin{array}{c}{h}_{23}\\ h_{24}\end{array}\end{array}\right]=\left[\begin{array}{c}{v}_c^{\left(1\right)}\\ v_c^{\left(2\right)}\\ \begin{array}{c}⋮\\ v_c^{\left(ni\right)}\end{array}\end{array}\right]\end{array}\right.\end{array}$$

(26)

All parameters in the projection matrix can be computed from the least squares solution of Equation (26). Similarly, we can complete the three-dimensional reconstruction by combining Equation (26) with projection equations of the projector, which has the same form as Equation (23). It can be seen that this three-dimensional reconstruction method is simple and convenient.

4. Experiment and Discussion

To verify the method we proposed, we establish an experiment system to evaluate the accuracy of the three-dimensional reconstruction and compare this method with the projection matrix method. We used a Hikvision camera, model MV-CH210-90YM-M58S-NN, with a resolution of 5120 pixel × 4096 pixel and a pixel size of 4.5 μm × 4.5 μm, and a CR Canrui XF-PTL05535-F-VI telecentric lens, whose object field-of-view target surface size is 35.0 mm × 28.0 mm, depth of field is 0.9~12.0 mm, and magnification is 0.658. We used an Anhua Optoelectronics model H9H projector with a resolution 1280 pixel × 720 pixel. The calibration board and the diameter of the standard ball used in the experiment are, respectively, GP050-9 × 9 and 10 mm. In addition, the manufacturing accuracy of the standard ball is ∅10.0041 mm. The camera and projector are installed by the brackets, which can also fix their poses. The platform built for the experiment is as shown in the Figure 3:

The experiment is carried out within the 3 mm depth of field of the telecentric lens, at which clear pictures can be captured. Place the calibration board at different positions at each depth to ensure that the camera’s field of view is covered. Specifically, the board is moved at 0.75 mm intervals over the depth range, and at each depth, the board is moved to the four corners and the center to ensure that all the positions in the camera’s field of view are covered. Project the Gray code and phase-shift stripes after photographing the calibration board. Theoretically, the higher the number of steps, the higher the accuracy. However, when the number of steps reaches a specific threshold, further increments do not markedly enhance the precision, but rather augment the computational burden [51]. So, we use an eight-step phase shift in this experiment. In order to cover the projector image plane, we chose a 10-bit Gray code because the projector’s resolution is 1280 pixels × 720 pixels and the stripe is 4 pixels wide. We could have chosen a 9-bit Gray code, but we chose the 10-bit code because of the complementary Gray Code. The stripes are shown in Figure 4:

After calibrating the system, the pose between the camera and the projector is calculated through the telecentric epipolar constraint algorithm. Based on the theory above, the code is written to reconstruct the point clouds of the planes and spheres. We utilize Geomagic Studio 2013 for point cloud data processing, which includes modeling, noise reduction, the elimination of non-connected points, fitting, etc. In this study, we use it to fit and calculate the deviation for the plane and sphere data. These results serve as the basis for comparing different methods. However, because Geomagic Studio 2013 cannot draw a diagram with an axis line, we use MATLAB R2020a to plot a precise diagram.

Based on the telecentric epipolar constraints, a reconstruction is performed for the planes and spheres. In the pseudo-colored map, the color represents the distance deviation between the point cloud data and the fitting feature. The correspondence between the degree of deviation and the color is given by the axis next to it.

In Figure 5, the fitting standard deviation of the plane is 11.0 μm, and the color map for the plane fitting deviation is relatively uniform, indicating that the fitting plane has an overall flatness. The fitting standard deviation of the sphere is 14.9 μm, and the color map for the deviation is also relatively uniform, with a diameter of 10.0064 mm.

Meanwhile, the three-dimensional reconstruction based on the projection matrix method is performed on the same plane and sphere, following the same process. The final reconstructed results are shown below.

In Figure 6, the fitting standard deviation of the plane is 23.81 μm. The pseudo-colored map reveals distortion at the edges of the plane, indicating that the fitting plane has some curvature. The diameter of the sphere is 10.0304 mm, with a fitting standard deviation of 13.4 μm.

To compare the reconstruction results across the entire field of view, the sphere is placed at different positions within the camera’s view as shown in Figure 7. Two methods are employed to reconstruct the sphere under varying fields of view. The point cloud data are then fitted to find the best-fitting sphere. The standard deviation of the fitting data and the absolute deviation of diameter (the difference between the fitting diameter and the diameter of a standard ball (10 mm)) are calculated. The specific comparative data for the reconstructions are presented in

Table 1. Standard deviation and diameter absolute deviation of sphere reconstruction at different positions.

Sphere	Position	Standard Deviation (M)/μm	Standard Deviation (E)/μm	Diameter Absolute Deviation (M)/μm	Diameter Absolute Deviation (E)/μm
S1	upper left	13.4	14.9	30.4	6.4
S2	upper right	14.0	14.7	16.3	8.4
S3	middle	13.1	14.6	25.4	1.0
S4	lower left	14.1	15.4	26.3	9.4
S5	lower right	13.9	14.8	14.1	6.3

. Please note that in Table 1 and

Table 2. Comparison of standard deviations for plane reconstruction at different depths.

Plane	Z/mm	Standard Deviation (M)/μm	Standard Deviation (E)/μm
Plane 1	0.00	23.8	11.0
Plane 2	0.75	21.7	11.1
Plane 3	1.50	21.9	10.9
Plane 4	2.25	21.8	9.9
Plane 5	3.00	23.1	9.1

, we use ‘M’ to represent the data produced from the projection matrix method and ‘E’ to represent the data produced from the method we proposed.

It can be observed that the standard deviations of the two methods are generally at the same level and relatively stable. The absolute deviation of the sphere’s diameter reconstructed by our proposed method is within 10 μm, which is better than that achieved by the projection matrix method.

Similarly, the plane is reconstructed at different depths. The baseline plane is taken from the farthest depth, and reconstructions are performed by moving along the plane’s Z-axis with a step size of 0.75 mm. The specific comparative data are presented in the following table.

As shown in Table 2, the fitting standard deviations of the plane reconstructed by our method are consistently better than those obtained with the projection matrix method. Taking Plane 3 as an example, the fitting standard deviation achieved by our method is 10.9 μm, which outperforms the projection matrix reconstruction’s fitting standard deviation of 21.9 μm, resulting in an improvement of approximately 50.2%.

To further investigate the cause of the fitting standard deviation difference between the two methods, we uniformly sample a set of point clouds from the four corners and center of each plane. The average distance from these point clouds to the fitted plane is calculated. The absolute difference between the corner-to-plane distance and the center-to-plane distance is defined as the curvature of that plane. The specific experimental data for the curvature of planes reconstructed using the projection matrix method and the epipolar constraint method are shown in

Table 3. Corner curvature of plane reconstruction at different depths using the projection matrix method.

	Distance to Fitting Plane/μm					Curvature/μm
	Upper Left	Upper Right	Lower Left	Lower Right	Center	Upper Left	Upper Right	Lower Left	Lower Right	AVG
Plane 1	41.5	62.2	27.2	60.6	47.9	6.4	14.3	20.7	12.7	13.5
Plane 2	39.0	65.2	28.7	55.5	47.1	8.1	18.1	18.4	8.4	13.3
Plane 3	24.1	52.0	31.3	66.9	43.5	19.4	8.5	12.2	23.4	15.9
Plane 4	29.4	63.4	33.9	61.2	47.0	17.6	16.4	13.1	14.2	15.3
Plane 5	33.7	77.7	26.5	87.1	56.2	22.5	21.5	29.7	30.9	26.2

and

Table 4. Corner curvature of plane reconstruction at different depths using the epipolar constraint method.

	Distance to Fitting Plane/μm					Curvature/μm
	Upper Left	Upper Right	Lower Left	Lower Right	Center	Upper Left	Upper Right	Lower Left	Lower Right	AVG
Plane 1	10.9	15.2	9.9	9.5	8.6	2.3	6.6	1.3	0.9	2.8
Plane 2	15.0	16.9	13.1	12.4	8.2	6.8	8.7	4.9	4.2	6.2
Plane 3	17.2	16.7	9.0	12.2	8.2	9.0	8.5	0.8	4.0	5.6
Plane 4	8.6	17.2	10.4	13.0	8.8	0.2	8.4	1.6	4.2	3.6
Plane 5	5.4	27.4	8.3	18.8	7.5	2.1	19.9	0.8	11.3	8.5

, respectively.

From Table 3 and Table 4, it can be observed that the corner curvature of the planes reconstructed using the projection matrix method is consistently greater than that achieved by our method. Taking Plane 3 as an example, the average curvature of the plane reconstructed using the epipolar constraint method is 5.6 μm, which is significantly lower than the average curvature of 15.9 μm obtained by the projection matrix method. This reduction in curvature by approximately 64.8% further confirms that the larger standard deviation in the projection matrix method primarily stems from distortion at the edges of the reconstructed plane, resulting in an overall curved shape. In contrast, our proposed method yields planes with better planar characteristics. This result aligns with the pseudo-colored deviation maps of the planes shown in Figure 5 and Figure 6.

5. Conclusions

In this paper, we propose a calibration method based on the constraint of telecentric epipolar geometry. It utilizes the epipolar constraint relationship between the telecentric and pinhole models to solve for the pose between the camera and the projector, based on which we realize a three-dimensional reconstruction.

Compared to the projection matrix method, the approach we proposed achieves diametric absolute deviations within 10 μm, with both methods yielding similar standard deviations for the sphere reconstruction. Moreover, in the case of the plane reconstruction, our method results in generally flat planes, while the projection matrix method exhibits curvature at the edges. This discrepancy arises because the projection matrix method does not consider the inherent properties of the rotation matrix; instead, it directly couples the intrinsic and extrinsic parameters. In contrast, our approach derives the essential matrix from the geometry relation, fully considering its properties and optimizing the rotation parameters to ensure orthogonality.

In conclusion, this paper has made improvements in the following three aspects:

• When recovering the extrinsic parameters between the camera and the projector using the telecentric essential matrix, we ensure the intrinsic properties of the essential matrix under telecentric conditions by performing an SVD and adjusting the singular value matrix. This guarantees that the decomposed rotation matrix satisfies orthogonality constraints. Compared to the projection matrix method, the approach proposed in this paper results in smaller reconstruction standard deviations.
• It is more flexible. The equation calculating the essential matrix does not involve the extrinsic parameters of the camera. Therefore, during the camera calibration process, it is only necessary to determine the intrinsic parameters, which avoids the symbolic ambiguity of the extrinsic parameters. Additionally, in the experimental procedure, there is no need to introduce a micro-displacement platform, simplifying the process and minimizing potential errors.
• During the essential matrix estimation process, incorporating the calibration board corner data from all poses for the optimization results in a certain level of average error improvement.

The method we proposed abandons extrinsic parameters when calibrating the telecentric camera but requires the intrinsic parameters of the camera and projector when solving the telecentric essential matrix. In the code of the paper, the nonlinear optimization is adopted for the optimal projector internal parameters while the internal parameters of the camera depend on the quality of the calibration images. A better accuracy can be achieved with additional pose optimization between the camera and the projector.