Stereo Bi-Telecentric Phase-Measuring Deflectometry

Abstract

Replacing the endocentric lenses in traditional Phase-Measuring Deflectometry (PMD) with bi-telecentric lenses can reduce the number of parameters to be optimized during the calibration process, which can effectively increase both measurement precision and efficiency. Consequently, the low distortion characteristics of bi-telecentric PMD contribute to improved measurement accuracy. However, the calibration of the extrinsic parameters of bi-telecentric lenses requires the help of a micro-positioning stage. Using a micro-positioning stage for the calibration of external parameters can result in an excessively cumbersome and time-consuming calibration process. Thus, this study proposes a holistic and flexible calibration solution for which only one flat mirror in three poses is needed. In order to obtain accurate measurement results, the calibration residuals are utilized to construct the inverse distortion map through bicubic Hermite interpolation in order to obtain accurate anchor positioning result. The calibrated stereo bi-telecentric PMD can achieve 3.5 μm (Peak-to-Valley value) accuracy within 100 mm (Width) × 100 mm (Height) × 200 mm (Depth) domain for various surfaces. This allows the obtaining of reliable measurement results without restricting the placement of the surface under test.

1. Introduction

1.1. Typical Measurement Techniques for Specular Surfaces

Optical freeform surfaces, characterized by their complex three-dimensional shapes and high design flexibility, play a pivotal role in modern optical systems [1]. These surfaces break away from the constraints of traditional spherical and aspherical designs, enabling more flexible optical mapping relationships [2]. This contributes to enhanced optical system performance and greater integration capabilities.

In recent years, freeform specular mirrors have been widely used in various fields, such as ultra short throw projectors [3,4], HUD (head-up displays) [5,6], and eyeglass lens molds [7]. Freeform specular mirrors, due to their unique design and manufacturing processes, are capable of maintaining high-precision optical performance even in complex environments. However, their intricate shapes pose challenges for both fabrication and measurement. To ensure surface accuracy, advanced measurement techniques are essential.

In contact measurement techniques, coordinate measuring machines (CMMs) or contact profilometers can perform 3D measurements of freeform surfaces for specific apertures through raster or radial scanning [8,9]. However, the preload force of the probe may scratch the surface of the mirror being measured. The UA3P series equipment from Panasonic, based on atomic force probes, can achieve non-destructive measurement of freeform surfaces with a contact force of no more than 0.3 mN, but its measurement speed is slow, and the spacing between measurement points is usually large, making it difficult to characterize mid-to-high-frequency information such as tool marks or scratches on specular surfaces [10].

In non-contact measurement techniques, the LUPHOScan series of non-contact surface measurement devices from Taylor Hobson (Leicester, UK) use multi-wavelength interference (MWLI) technology, which employs a non-contact point probe to precisely measure the sag (surface height) and XYZ position of the probe, thereby achieving nanometer-level precision in surface reconstruction. However, when the tangential slope of the mirror exceeds ± 8° along the scanning path, the probe cannot obtain valid sag data, limiting its application on surfaces with steep angles [11]. Phase-measuring profilometry (PMP) is a non-contact measurement method that uses a projector light source to project sinusoidal or Gray code patterns onto the measured surface, allowing for 3D surface measurement without physical contact and with a high dynamic range. However, when measuring high-reflectivity mirrors, specular reflections may cause distortion in the measurement results in bright regions. The traditional solution is to coat the mirror with a layer of powder to alter its reflective properties and reduce reflection distortion. However, the thickness and uniformity of the powder layer may affect measurement accuracy, and removing the powder after measurement is time-consuming and tedious. In certain sensitive or specific measurement scenarios, using powder may not be a viable option [12]. Modern Ronchi measurement methods use Ronchi gratings as the light source, capturing the Ronchi grating fringe images and, with advanced digital image processing techniques, the optical wavefront aberrations can be reconstructed, achieving accurate non-contact measurement of mirror surface shapes. However, Ronchi gratings need to be custom-made and replaced according to the dynamic range of the measured mirror surface, limiting its versatility [13]. Häusler et al. [14] were among the first to replace the projector light source in PMP with an LCD screen, thus proposing a technique called PMD for measuring large-aperture freeform mirrors. The core advantage of this technology lies in its ability to achieve micron-level measurement accuracy for apertures in the range of hundreds of millimeters [15], while offering nanometer-level resolution in the sag direction [16]. Since the LCD screen can project custom-coded information, it offers greater versatility compared to Ronchi measurement methods. The projector light source has high directionality, which can cause overexposure when measuring reflective surfaces, leading to reconstruction failures. The LCD screen used in PMD is a non-directional light source, making it more suitable for measuring reflective surfaces.

Figure 1 shows the characteristics of above typical contact and non-contact measurement techniques. PMD plays a crucial role in measuring freeform mirrors.

1.2. Typical PMD Configurations with Endocentric Lenses

There are three typical PMD setups with endocentric lenses, as shown in Figure 2. P represents the point corresponding to the Camera, while Q represents the point in the Screen. n is the normal calculated from the PMD.

The first typical configuration of PMD is the single-sensor-single-screen setup, as shown in Figure 2a. Su et al. [17] proposed an innovative idea that treats PMD as an inverse Hartmann measurement principle to measure the freeform mirror. In this method, the theoretical model is treated as the predicted shape of the surface being measured, and surface reconstruction is performed by iterative gradient calculation and integration, leading to the final surface measurement result. Huang et al. [18] used Fourier transform-based phase extraction algorithms on the single-sensor-single-screen setup PMD to achieve single-frame measurements of dynamic reflective surfaces. Meanwhile, Wang et al. [19] greatly improved the measurement speed of phase deflection techniques through the development of high-speed displays based on LED arrays, enabling dynamic surface measurements while retaining phase detail This method allows for the rapid capture of surface information in complex and dynamic environments, significantly enhancing measurement efficiency, particularly for precise detection of rapidly changing surfaces. Xu et al. [20] during the ultra-precision machining of freeform reflective mirrors, proposed an in-situ measurement method based on single-sensor-single-screen setup PMD. A common drawback of single-sensor-single-screen setup PMD is that it requires prior knowledge of the measured surface shape.

The second typical configuration of PMD is the single-sensor-multiple-screen setup, as shown in Figure 2b. Guo et al. [21] move the screen to different positions using a displacement platform for imaging, while Li et al. [22] introduce a transparent LCD, forming a multiple-screen structure. In addition, Tang et al. [23], Liu et al. [24] and Zhang et al. [25] incorporate beam splitter allowing the LCD to image at two different positions. The single-sensor-multiple-screen setup PMD eliminates the need for the estimated surface shape in single-sensor-single-screen PMD setup. However, the introduction of additional mechanical movement structures, beam splitter, and a second display increased the system’s size and cost. According to the performance analysis and evaluation of monocular bi-telecentric PMD, the distance between two LCD screen [26] and the refraction error of the beam splitter [27] should be concerned.

The third typical configuration of PMD is the multiple-sensors-single-screen setup, as shown in Figure 2c. Li et al. [28] and Han et al. [29] have built upon Häusler’s method by replacing the flat display with a curved display. This modification significantly enhanced the ability of stereophotogrammetry to measure the maximum slope of complex surfaces, making the technology more widely applicable and accurate for measuring irregular surfaces. Additionally, Zhang et al. [30] and Wang et al. [31] have further expanded the scope of stereophotogrammetry by using camera field stitching and fusion techniques, successfully enlarging the measurable surface aperture. Han et al. [32] proposed a reconstruction procedure for multiple-sensors-single-screen setup PMD, in which the specular surface under test (SUT) can be achieved through anchor point positioning followed by iterative surface reconstruction. Thus, the multiple-sensors-single-screen setup PMD does not require the predicted shape of the SUT.

Table 1. Comparison of three typical PMD configurations.

PMD Configuration	Advantages	Disadvantages
Single-sensor-single-screen	Simple setup	Predicted SUT shape needed
Single-sensor-multiple-screens	Easy to reconstruct the shape of SUT	Extra equipment introduces more error sources.
Multiple-sensors-single-screen	Don’t need the predicted shape of SUT	Difficult to calibrate

summarizes the characteristics of three typical PMD configurations.

1.3. PMD Configurations with Bi-Telecentric Lenses

Compared to endocentric lenses in high accuracy machine vision [33], bi-telecentric lenses not only exhibit lower image distortion but also simplify the calibration process [34]. The imaging models of endocentric lenses and bi-telecentric lenses are shown in Figure 3.

For a given pixel ^up = [u, v]^T, its corresponding 3D point in the camera’s local coordinate can be denoted as ^cp = [x_c, y_c, z_c]^T. For the imaging model of endocentric lenses shown in Figure 3a, its principal point can be noted as ^up₀ = [u₀, v₀]^T and focal length can be noted as f_x and f_y in x and y direction. The projection of an arbitrary point ^cp to the sensor’s image coordinate in pixels is expressed as

$$z_{\mathrm{c}}\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_x& 0& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_{\mathrm{c}}\\ y_{\mathrm{c}}\\ z_{\mathrm{c}}\end{array}\right].$$

(1)

As shown in Equation (1), the position of point ^cp in endocentric lenses’ local coordinate varies with z_c.

For the imaging model of the bi-telecentric lenses shown in Figure 3b, magnification can be noted as m_x and m_y in x and y direction. The projection of an arbitrary point ^cp to the sensor’s image coordinate in pixels is expressed as

$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{m}_x& 0& 0\\ 0& m_y& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_{\mathrm{c}}\\ y_{\mathrm{c}}\\ 1\end{array}\right].$$

(2)

As shown in Equation (2), the bi-telecentric lens cannot identify the depth of z_c.

Except for the distortion, the intrinsic parameters of bi-telecentric lenses only include magnification. In contrast, endocentric lenses adhere to the pinhole model, which requires calibration of the optical center u₀, v₀ and focal lengths f_x and f_y [35]. Moreover, a pinhole camera model is far from a true camera model [36,37]. Due to the lower distortion and fewer intrinsic parameters of bi-telecentric lenses, incorporating them into PMD can easily yield high-precision calibration results, thereby enhancing measurement accuracy. The three typical PMD setups with bi-telecentric lenses are shown in Figure 4.

In the single-sensor-single-screen configuration shown in Figure 4a, endocentric lenses have been substituted by bi-telecentric lenses. To further validate the nanometer-level height resolution of PMD, Häusler et al. [38] proposed a simplified method using a single-screen setup in a monocular microscopic system to measure surfaces with reflective properties. Liu et al. [39] replaced the traditional TFT screen light source with a combination of photolithography film, diffusion film, and LED lamps. This improvement effectively avoided the impact of screen refresh rates and grayscale discretization on measurement results, significantly enhancing both the accuracy and reliability of measurements.

As shown in Figure 4b, Niu et al. [40] and Huang et al. [41] position the LCD screen perpendicularly to the optical axis of the bi-telecentric imaging system. This approach eliminates ambiguity in the LCD screen’s rotation matrix, enabling the implementation of a single-sensor-multiple-screens phase-measuring deflectometry (PMD) using bi-telecentric lenses.

According to the literature review summarized in

Table 2. The literature review of three PMD setups with endocentric and bi-telecentric lenses.

PMD Configurations	Single Sensor and Single Screen	Single Sensor and Multiple Screens	Multiple Sensors and Single Screen
With endocentric lenses	Häusler et al. [14]	Guo et al. [21]	Li et al. [28]
	Su et al. [17]	Li et al. [22]	Han et al. [29]
	Huang et al. [18]	Tang et al. [23]	Zhang et al. [30]
	Wang et al. [19]	Liu et al. [24]	Wang et al. [31]
	Xu et al. [20]	Zhang et al. [25]	Han et al. [32]
With bi-telecentric lenses	Häusler et al. [38]	Niu et al. [40]	Proposed in this paper
	Liu et al. [39]	Huang et al. [41]

, there is currently no PMD technique that introduces bi-telecentric lenses into the multiple-sensors-single-screen PMD configuration. Based on Table 1, to improve measurement accuracy, avoid surface shape estimation, and reduce error sources in PMD, it is necessary to propose a stereo bi-telecentric PMD technique.

1.4. Proposed Stereo Bi-Telecentric PMD

This paper presents a stereo bi-telecentric PMD technique, while a holistic and flexible calibration method for stereo bi-telecentric PMD is also presented. In monocular bi-telecentric PMD, the LCD screen can be adjusted to be perpendicular to the optical axis of the bi-telecentric imaging system to avoid ambiguity in the LCD screen rotation matrix [40]. However, in the stereo bi-telecentric PMD, the LCD screen cannot be perpendicular to the optical axes of both two bi-telecentric imaging systems. The self-calibration method without the assistance of third-party devices is more popular in PMD calibration due to its low operational complexity and high measurement efficiency [42]. Under the same calibration conditions as stereo PMD based on endocentric lenses [43,44], only a markerless flat mirror is needed, which should be placed in three different poses to obtain the phase information of the flat LCD screen, thereby completing the calibration. The proposed calibration method does not require the help of a micro-positioning stage [45,46,47] in the telecentric lens calibration process to resolve the ambiguity of the rotation matrix in the extrinsic parameters. Therefore, this stereo bi-telecentric PMD technique and its calibration method are easy to set up in practical implementation.

Stereo PMD requires two steps to obtain the measurement result: the anchor point positioning procedure and iterative surface reconstruction procedure. In order to obtain accurate measurement results, the calibration residuals are utilized to construct the inverse distortion map through bicubic Hermite interpolation to obtain accurate anchor positioning result. The repeatability and reproducibility experiments have been conducted to investigate the accuracy of the proposed stereo bi-telecentric PMD system.

2. Flat-Mirror-Only Calibration Procedure

2.1. Measurement Principle of the Stereo Bi-Telecentric PMD

As shown in Figure 5, similar to stereo PMD with endocentric lenses [32], the reconstruction of the SUT in the stereo bi-telecentric PMD system can be accomplished through anchor point positioning, followed by iterative surface S reconstruction.

(1)

The detailed positioning procedure for anchor point Z will be discussed in Section 3.1, which deals with the triangulation in stereo-PMD setup. Although bi-telecentric lenses exhibit significantly lower distortion compared to endocentric lenses, it is still necessary to accurately handle the distortion mapping relationship during the anchor point positioning process to improve the accuracy of the measurement system. The distortion mapping for the pixel coordinate ^up_A and 3D point p_A in camera A will be discussed in Section 3.2, while the inverse distortion mapping for the 3D point p_B and pixel coordinate ^up_B in camera B will be discussed in detail in Section 3.4.

(2)

Once the anchor point has been established, the iteration starts with initial plane S₀ in mono-PMD setup. Then the slope data of the guessed surface can be calculated. The updated iteration surface Sn can be calculated by the integral reconstruction of the slope data [48,49], until the iteration result converges [50].

The developed hardware (as shown in Figure 6) includes two bi-telecentric lenses with cameras, a flat LCD screen, and a 535 nm, 50 mW laser diode serving as the laser pointer. Since the mirror surface lacks distinct features, it is necessary to utilize its diffuse reflection characteristics during the anchor point positioning process. This can be achieved by using a laser to mark a spot on the surface that exhibits diffuse reflection properties. The model of the bi-telecentric lenses is TC23144, with a nominal magnification of 0.061. The cameras use the Sony IMX264 CMOS sensor with a resolution of 2448 × 2048. The model of the flat LCD screen is a ViewSonic VP2768 with a resolution 3840 × 2160. The LCD coordinate system {L} takes the first row and first column of the LCD screen as the origin, with the x-axis along the horizontal direction of the LCD screen, the y-axis along the vertical direction of the LCD screen, and the z-axis perpendicular to the screen inward.

From the reconstruction procedure, it can be seen that the geometric relationships between the cameras and LCD and the cameras’ intrinsic parameters with distortion must be precisely calibrated. As the extrinsic parameters, the rotation matrix ${}_{\mathrm{L}}{}^{\mathrm{A}}R$ and the translation vector ${}_{\mathrm{L}}{}^{\mathrm{A}}T$ represent the relative positions between {L} and camera A’s local coordinate system {A}. The magnification $m_x^{\mathrm{A}}$ (along camera horizontal direction) and $m_y^{\mathrm{A}}$ (along camera vertical direction) are the intrinsic parameters of camera A. The rotation matrix ${}_{\mathrm{L}}{}^{\mathrm{B}}R$ and the translation vector ${}_{\mathrm{L}}{}^{\mathrm{B}}T$ represent the relative positions between {L} and camera B’s local coordinate system {B}. The magnification $m_x^{\mathrm{B}}$ (along camera horizontal direction) and $m_y^{\mathrm{B}}$ (along camera vertical direction) are the intrinsic parameters of camera B.

2.2. Formula Derivation in the Calibration

The calibration process starts by placing a flat mirror in 3 different positions, allowing camera A and camera B to capture the phase-shifting fringes on the flat LCD screen (as shown in Figure 7). The red dots in Figure 7 are indicated by the laser pointer for anchor point positioning. And the blue frames are the bounding boxes of the red dots. To obtain reliable calibration results, nonlinear errors need to be suppressed [51]. By the optimal phase-shifting technique [52], the corresponding physical position p_L = [x_L, y_L,0]^T in the flat LCD coordinate system {L} of camera pixel ^up = [u, v]^T can be acquired. Because the LCD is flat, the z-value of p_L is always zero. So the 3D point p_L = [x_L, y_L,0]^T can be noted as a 2D point p_L = [x_L, y_L]^T. Its coordinate in camera B’s local coordinate system {B} is also a 2D point p_B = [x_B, y_B]^T. For camera B, the geometric relationship is drawn in Figure 8.

The rotation matrix ${}_{\mathrm{L}}{}^{\mathrm{B}}A$ (whose determinant is −1) and the translation vector ${}_{\mathrm{L}}{}^{\mathrm{B}}b$ represent the position relationship between the flat LCD screen virtual image coordinate system {L’} and camera B’s local coordinate system {B} when the flat mirror is at pose 1. The normal of flat mirror in {B} can be noted as ^Bn = [n_x, n_y, n_z]^T. The distance between the origin of {B} and the flat mirror is noted as L.

According to the geometric relationship [53], the relationship between the 2D coordinates of p in {L} and {B} can be derived as

$$p_{\mathrm{B}}={}_{\mathrm{L}}{}^{\mathrm{B}}R_{2\times 2}{p}_{\mathrm{L}}+{}_{\mathrm{L}}{}^{\mathrm{B}}T_{2\times 1}.$$

(3)

where the number in the bottom right corner of the matrix symbol, for example, 2 × 2, represents the upper 2 × 2 submatrix set of this matrix. The geometric relationship between the p_B and its virtual image p’_B with respect to the plane mirror is

$$p_{\mathrm{B}}^{\prime }=p_{\mathrm{B}}+2d^{\mathrm{B}}{n}_{2\times 1}.$$

(4)

where the number 2 × 1 in the bottom right corner of ^Bn represents the upper 2 × 1 submatrix set of this ^Bn_2×1 = [n_x, n_y]^T, and the distance d between the given point p_B of {B} and the flat mirror is noted as

$$d=L-{}^{\mathrm{B}}n_{2\times 1}^{\mathrm{L}}{p}_{\mathrm{B}}.$$

(5)

Combining Equation (3) to Equation (5), the following equation can be obtained as

$$\left.\begin{array}{c}{p}_{\mathrm{B}}={}_{\mathrm{L}}{}^{\mathrm{B}}R_{2\times 1}{p}_{\mathrm{L}}+{}_{\mathrm{L}}{}^{\mathrm{B}}T_{2\times 1}\\ p_{\mathrm{B}}^{\prime }=p_{\mathrm{B}}+2d^{\mathrm{B}}{n}_{2\times 1}\\ d=L-{}^{\mathrm{B}}n_{2\times 1}^T{p}_{\mathrm{B}}\end{array}\right\}\Rightarrow \left[\begin{array}{c}{p}_{\mathrm{B}}^{\prime }\\ 1\end{array}\right]=\left[\begin{array}{cc}{}_{\mathrm{L}}{}^{\mathrm{B}}A_{2\times 2}& {}_{\mathrm{L}}{}^{\mathrm{B}}b_{2\times 1}\\ 0_{1\times 2}& 1\end{array}\right]\left[\begin{array}{c}{p}_{\mathrm{L}}\\ 1\end{array}\right],$$

(6)

where

$$\left\{\begin{array}{c}{}_{\mathrm{L}}{}^{\mathrm{B}}A_{2\times 1}=(I_2-2{}^{\mathrm{B}}n_{2\times 1}{}^{\mathrm{B}}n_{2\times 1}^{\mathrm{T}}){}_{\mathrm{L}}{}^{\mathrm{B}}R_{2\times 1}\\ {}_{\mathrm{L}}{}^{\mathrm{B}}b_{2\times 1}=(I_2-2{}^{\mathrm{B}}n_{2\times 1}{}^{\mathrm{B}}n_{2\times 1}^{\mathrm{T}}){}_{\mathrm{L}}{}^{\mathrm{B}}T_{2\times 1}+2L{}^{\mathrm{B}}n_{2\times 1}\end{array}\right..$$

(7)

where I₂ means a 2 × 2 identity matrix. As bi-telecentric lenses perform parallel projection, it cannot identify the “z value” (or depth) from the coordinates of p and the normals of n in camera B’s local coordinate system {B}. Thus, these 2D equations in the camera’s local coordinate system can be obtained from the 3D ones by simply taking the first two components.

For the Sony IMX264 CMOS sensor, its pixel size is equal in x and y directions. Thus, it can be noted that $m=m_x^{\mathrm{B}}=m_y^{\mathrm{B}}$, then ^up = [u, v]^T and p_B = [x_B, y_B]^T have the relationship as follows:

$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}m& 0& 0\\ 0& m& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_{\mathrm{B}}\\ y_{\mathrm{B}}\\ 1\end{array}\right].$$

(8)

Substituting Equation (6) into Equation (8), the following equation can be obtained as

$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}m& 0& 0\\ 0& m& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}{r}_{11}& r_{12}& t_x\\ r_{21}& r_{22}& t_y\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_{\mathrm{L}}\\ y_{\mathrm{L}}\\ 1\end{array}\right]=\left[\begin{array}{ccc}m{r}_{11}& m{r}_{12}& m{t}_x\\ m{r}_{21}& m{r}_{22}& m{t}_y\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_{\mathrm{L}}\\ y_{\mathrm{L}}\\ 1\end{array}\right]=\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_{\mathrm{L}}\\ y_{\mathrm{L}}\\ 1\end{array}\right]=H\left[\begin{array}{c}{x}_{\mathrm{L}}\\ y_{\mathrm{L}}\\ 1\end{array}\right],$$

(9)

where r_ij is the element of rotation matrix ${}_{\mathrm{L}}{}^{\mathrm{B}}A$, and $i,j\in \left\{1,2,3\right\}$. ${}_{\mathrm{L}}{}^{\mathrm{B}}b$ = [t_x, t_y]^T. Equation (9) can be rewritten as

$$\left[\begin{array}{cccccc}{x}_{\mathrm{L}}& y_{\mathrm{L}}& 1& 0& 0& 0\\ 0& 0& 0& x_{\mathrm{L}}& y_{\mathrm{L}}& 1\end{array}\right]\left[\begin{array}{c}{h}_{11}\\ h_{12}\\ h_{13}\\ h_{21}\\ h_{22}\\ h_{23}\end{array}\right]=\left[\begin{array}{c}u\\ v\end{array}\right],$$

(10)

where h_ij can be calculated directly from a least square solution from Equation (10). Thus, to calculate ${}_{\mathrm{L}}{}^{\mathrm{B}}A$ and ${}_{\mathrm{L}}{}^{\mathrm{B}}b$, m must be determined first.

With the orthogonality of the rotation matrix ${}_{\mathrm{L}}{}^{\mathrm{B}}A$ with reflection, r_ij has a relationship as

$$r_{11}{r}_{12}+r_{21}{r}_{22}-r_{31}{r}_{32}=0.$$

(11)

Since each column of the rotation matrix is a unit vector, r_ij has the second relationship as

$$\left\{\begin{array}{c}{r}_{31}=\pm \sqrt{1-r_{11}^2-r_{21}^2}\\ r_{32}=\mp \sqrt{1-r_{12}^2-r_{22}^2}\end{array}\right..$$

(12)

From Equations (11) and (12), the following equation can be obtained as

$$1-\left(r_{11}^2+r_{21}^2+r_{12}^2+r_{22}^2\right)+{\left(r_{11}{r}_{22}-r_{12}{r}_{21}\right)}^2=0.$$

(13)

From the relationship between r_ij and h_ij in Equation (10), Equation (13) can be rewritten as

$$m^4-\left(h_{11}^2+h_{21}^2+h_{12}^2+h_{22}^2\right)m^2+{\left(h_{11}{h}_{22}-h_{12}{h}_{21}\right)}^2=0.$$

(14)

where m is the maximum non-negative root satisfying

$$r_{ij}={\displaystyle \frac{h_{ij}}{m}}\le 1\hspace{1em}\hspace{1em}\hspace{1em}i,j\in \{1,2\}.$$

(15)

Now m has been determined. According to Equation (9), the translation vector ${}_{\mathrm{L}}{}^{\mathrm{B}}b$ can be retrieved by

$$\left\{\begin{array}{c}{t}_x={\displaystyle \frac{h_{13}}{m}}\\ t_y={\displaystyle \frac{h_{23}}{m}}\end{array}\right..$$

(16)

By Equations (12) and (15), r_ij where $i,j\in \left\{1,2\right\}$ can be determined. The signs of r₃₁ and r₃₂ are uncertain. Since the rotation matrix is unitary and orthogonal, the remains can be determined by

$$[r_{13},r_{23},r_{33}]=[r_{11},r_{21},r_{31}]\times [r_{12},r_{22},r_{32}].$$

(17)

Now, the following equation can be obtained as

$${}_{\mathrm{L}}{}^{\mathrm{B}}A^{+}=\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\\ r_{31}& r_{32}& r_{32}\end{array}\right]\hspace{1em}{}_{\mathrm{L}}{}^{\mathrm{B}}A^{-}=\left[\begin{array}{ccc}{r}_{11}& r_{12}& -r_{13}\\ r_{21}& r_{22}& -r_{23}\\ -r_{31}& -r_{32}& r_{32}\end{array}\right]\hspace{1em}{}_{\mathrm{L}}{}^{\mathrm{B}}b=\left[\begin{array}{c}{t}_x\\ t_y\end{array}\right].$$

(18)

It should be noticed that ${}_{\mathrm{L}}{}^{\mathrm{B}}A$ has an ambiguous solution. These two solutions are noted as ${}_{\mathrm{L}}{}^{\mathrm{B}}A^{+}$ and ${}_{\mathrm{L}}{}^{\mathrm{B}}A^{-}$. The reason why there are two solutions of ${}_{\mathrm{L}}{}^{\mathrm{B}}A$ is that the bi-telecentric lenses perform parallel projection, and cannot identify the depth. Figure 9 shows the geometric relationship of the two possible LCD virtual images, which the bi-telecentric lenses cannot identify.

2.3. Eliminating Ambiguities within Monocular Setup

According to Equation (7), the rotation matrix ${}_{\mathrm{L}}{}^{\mathrm{B}}A$ at pose i ($i\in \left\{1,2,3\right\}$) can be calculated as

$${}_{\mathrm{L}}{}^{\mathrm{B}}A_i=\left(I_3-2^{\mathrm{B}}{n_i}^{\mathrm{B}}{n}_i^T\right){}_{\mathrm{L}}{}^{\mathrm{B}}R_{\mathrm{B}}=\left[\begin{array}{ccc}1-2n_{ix}{n}_{ix}& -2n_{ix}{n}_{iy}& -2n_{ix}{n}_{iz}\\ -2n_{ix}{n}_{iy}& 1-2n_{iy}{n}_{iy}& -2n_{\mathrm{iy}}{n}_{iz}\\ -2n_{ix}{n}_{iz}& -2n_{\mathrm{iy}}{n}_{iz}& 1-2n_{iz}{n}_{iz}\end{array}\right]{}_{\mathrm{L}}{}^{\mathrm{B}}R.$$

(19)

Let

$${}_{\mathrm{L}}{}^{\mathrm{B}}A_i{}_{\mathrm{L}}{}^{\mathrm{B}}A_j^T=\left(I_3-2^{\mathrm{B}}{n_i}^{\mathrm{B}}{n}_i^T\right){\left(I_3-2^{\mathrm{B}}{n_j}^{\mathrm{B}}{n}_j^T\right)}^T\hspace{1em}i,j\in \left\{1,2,3\right\}andi\ne j.$$

(20)

It can be noticed that

$$\det ({}_{\mathrm{L}}{}^{\mathrm{B}}A_i{}_{\mathrm{L}}{}^{\mathrm{B}}A_j^T)=1,$$

(21)

and ${}_{\mathrm{L}}{}^{\mathrm{B}}A_i{}_{\mathrm{L}}{}^{\mathrm{B}}A_j^T$ is orthogonal. Therefore, ${}_{\mathrm{L}}{}^{\mathrm{B}}A_i{}_{\mathrm{L}}{}^{\mathrm{B}}A_j^T$ is a rotation matrix. The eigenvector ${\psi }_{ij}$ of ${}_{\mathrm{L}}{}^{\mathrm{B}}A_i{}_{\mathrm{L}}{}^{\mathrm{B}}A_j^T$ corresponding to the unit eigenvalue can be calculated from

$$\begin{array}{l}\left({}_{\mathrm{L}}{}^{\mathrm{B}}A_i{}_{\mathrm{L}}{}^{\mathrm{B}}A_j^T-I_3\right){\psi }_{ij}=0\\ \Rightarrow {\psi }_{ij}=\left[\begin{array}{c}{\displaystyle \frac{n_{iy}{n}_{jz}-n_{jy}{n}_{iz}}{n_{ix}{n}_{jy}-n_{jx}{n}_{iy}}}\\ {\displaystyle \frac{n_{jx}{n}_{iz}-n_{ix}{n}_{jz}}{n_{ix}{n}_{jy}-n_{jx}{n}_{iy}}}\\ 1\end{array}\right]={\displaystyle \frac{1}{n_{ix}{n}_{jy}-n_{jx}{n}_{iy}}}\left[\begin{array}{c}{n}_{iy}{n}_{jz}-n_{jy}{n}_{iz}\\ n_{jx}{n}_{iz}-n_{ix}{n}_{jz}\\ n_{ix}{n}_{jy}-n_{jx}{n}_{iy}\end{array}\right]\end{array}.$$

(22)

It also can be noted that

$${}^{\mathrm{B}}n_i\times {}^{\mathrm{B}}n_j=\left[\begin{array}{c}{n}_{ix}\\ n_{iy}\\ n_{iz}\end{array}\right]\times \left[\begin{array}{c}{n}_{jx}\\ n_{jy}\\ n_{jz}\end{array}\right]=\left[\begin{array}{c}{n}_{iy}{n}_{jz}-n_{jy}{n}_{iz}\\ n_{jx}{n}_{iz}-n_{ix}{n}_{jz}\\ n_{ix}{n}_{jy}-n_{jx}{n}_{iy}\end{array}\right].$$

(23)

Compare Equations (22) and (23), which have the relationship as

$${}^{\mathrm{B}}n_i\times {}^{\mathrm{B}}n_j=\left(n_{ix}{n}_{jy}-n_{jx}{n}_{iy}\right){\psi }_{ij}.$$

(24)

Here, $\otimes$ is defined as the operator that performs normalization after the cross product. Thus, all the normal vectors of the flat mirror at three different poses can be obtained by

$$\left.\begin{array}{c}{}^{\mathrm{B}}n_1\otimes {}^{\mathrm{B}}n_2={\psi }_{12}\\ {}^{\mathrm{B}}n_1\otimes {}_{\mathrm{B}}n_3={\psi }_{13}\\ {}^{\mathrm{B}}n_2\otimes {}_{\mathrm{B}}n_3={\psi }_{23}\end{array}\right\}\Rightarrow \left\{\begin{array}{c}{}^{\mathrm{B}}n_1={\psi }_{12}\otimes {\psi }_{13}\\ {}^{\mathrm{B}}n_2={\psi }_{12}\otimes {\psi }_{23}\\ {}_{\mathrm{B}}n_3={\psi }_{13}\otimes {\psi }_{23}\end{array}\right..$$

(25)

Derived from Equation (6),

$$\left.\begin{array}{c}{}_{\mathrm{L}}{}^{\mathrm{B}}R_1=\left(I_3-2^{\mathrm{B}}{n_1}^{\mathrm{B}}{n}_1^T\right){}_{\mathrm{L}}{}^{\mathrm{B}}A_1\\ {}_{\mathrm{L}}{}^{\mathrm{B}}R_2=\left(I_3-2^{\mathrm{B}}{n_2}^{\mathrm{B}}{n}_2^T\right){}_{\mathrm{L}}{}^{\mathrm{B}}A_2\\ {}_{\mathrm{L}}{}^{\mathrm{B}}R_3=\left(I_3-2^{\mathrm{B}}{n_3}^{\mathrm{B}}{n}_3^T\right){}_{\mathrm{L}}{}^{\mathrm{B}}A_3\end{array}\right\}\Rightarrow {}_{\mathrm{L}}{}^{\mathrm{B}}\overline{R}={\displaystyle \frac{\left({}_{\mathrm{L}}{}^{\mathrm{B}}R_1+{}_{\mathrm{L}}{}^{\mathrm{B}}R_2+{}_{\mathrm{L}}{}^{\mathrm{B}}R_3\right)}{3}}.$$

(26)

Thus, according to [43] the rotation matrix between camera B and flat LCD screen can be calculated as

$${}_{\mathrm{L}}{}^{\mathrm{B}}R={\left[{\left({}_{\mathrm{L}}{}^{\mathrm{B}}\overline{R}^T{}_{\mathrm{L}}{}^{\mathrm{B}}\overline{R}\right)}^{0.5}\right]}^{-1}{}_{\mathrm{L}}{}^{\mathrm{B}}\overline{R},$$

(27)

where ${( )}^{0.5}$ indicates the square root of the elements within the matrix. The translation vector between {B} and {L} can be determined by a least square solution of

$$\left[\begin{array}{cccc}\left(I_2-2^{\mathrm{B}}{n_{1_{2\times 1}}}^{\mathrm{B}}{n}_{1_{2\times 1}}^T\right)& 2^{\mathrm{B}}{n}_{1_{2\times 1}}& {\mathbf{0}}_{2\times 1}& {\mathbf{0}}_{2\times 1}\\ \left(I_2-2^{\mathrm{B}}{n_{2_{2\times 1}}}^{\mathrm{B}}{n}_{2_{2\times 1}}^T\right)& {\mathbf{0}}_{2\times 1}& 2^{\mathrm{B}}{n}_{2_{2\times 1}}& {\mathbf{0}}_{2\times 1}\\ \left(I_2-2^{\mathrm{B}}{n_{3_{2\times 1}}}^{\mathrm{B}}{n}_{3_{2\times 1}}^T\right)& {\mathbf{0}}_{2\times 1}& {\mathbf{0}}_{2\times 1}& 2^{\mathrm{B}}{n}_{3_{2\times 1}}\end{array}\right]\left[\begin{array}{c}{}_{\mathrm{L}}{}^{\mathrm{B}}T\\ L_1\\ L_2\\ L_3\end{array}\right]=\left[\begin{array}{c}{}_{\mathrm{L}}{}^{\mathrm{B}}b_1\\ {}_{\mathrm{L}}{}^{\mathrm{B}}b_2\\ {}_{\mathrm{L}}{}^{\mathrm{B}}b_3\end{array}\right],$$

(28)

which is a general form Dx = c (where D is a 6 × 6 known matrix, c is a 6 × 1 know vector, and x is the 6 × 1 vector of unknown). The reprojection error of the least square process can be defined as

$$Error={‖Dx-c‖}_2.$$

(29)

It is very important to know that the ambiguous solution of ${}_{\mathrm{L}}{}^{\mathrm{B}}A_i\hspace{1em}i\in \left\{1,2,3\right\}$ may cause an increase in error. The rotation matrix at three different LCD virtual image positions, with each position having two solutions, results in a total of eight possible solutions.

Table 3. Reprojection error and solutions of Equation (28) for camera A.

$\mathbf{sgn}({}_{\mathbf{L}}{}^{\mathbf{A}}\mathbf{A}_{\mathit{i}})\mathit{i}\in \left\{1,2,3\right\}$	+++	++-	+-+	+--	-++	-+-	--+	---
L1	−21.531	−71.481	−66.002	−5.894	5.894	66.002	71.481	21.531
L2	66.461	29.054	−66.991	−6.886	6.886	66.991	29.054	66.461
L3	80.482	−72.310	41.866	−6.806	6.806	−41.866	72.310	−80.482
Error	0.0133	0.1481	0.1364	0.0002	0.0002	0.1364	0.1481	0.0133

An operator sgn() is a symbol that indicates two rotation matrices ${}_{\mathrm{L}}{}^{\mathrm{A}}A_i$ by + and -. Two possible solutions are marked in red and blue.

and

Table 4. Reprojection error and solutions of Equation (28) for camera B.

$\mathbf{sgn}({}_{\mathbf{L}}{}^{\mathbf{B}}\mathbf{A}_{\mathit{i}})\mathit{i}\in \left\{1,2,3\right\}$	+++	++-	+-+	+--	-++	-+-	--+	---
L1	121.001	−340.782	−291.935	−123.652	123.652	291.935	340.782	−121.001
L2	121.362	−344.473	−272.385	−191.708	191.708	272.385	344.473	−121.362
L3	121.478	−356.023	−295.936	−194.714	194.714	295.936	356.023	−121.478
Error	0.0005	4.1532	4.1539	4.1220	4.1220	4.1539	4.1532	0.0005

An operator sgn() is a symbol that indicates two rotation matrices ${}_{\mathrm{L}}{}^{\mathrm{B}}A_i$ by + and -. Two possible solutions are marked in red and blue.

show the reprojection error and solutions of Equation (28) at these eight possible solutions for camera A and B, respectively. For every camera, six incorrect solutions can be eliminated from the Error, leaving two solutions marked in red and blue in Table 3 and Table 4.

2.4. Eliminating Ambiguities within Stereo Setup

Now the stereo system has four solutions. Obviously, the distance d between the given point p and the flat mirror should be same in the two cameras’ geometric relationships. Here, the given point can be set as the origin p_L = [0, 0]^T of {L}. Combining Equations (3) and (5), the distance d can be calculated as

$$d=L-{}^{\mathrm{B}}n_{2\times 1}^T{}_{\mathrm{L}}{}^{\mathrm{B}}T_{2\times 1}.$$

(30)

Here, the calculated d at three poses can be collected as d = [d₁, d₂, d₃]^T. Thus, the remaining four solutions can be filtered using the d from Equation (30). As shown in

Table 5. The remaining two solutions filtered by d and their differences.

$\mathbf{sgn}({\mathbf{A}}_{\mathit{i}})\mathit{i}\in \left\{1,2,3\right\}$	Camera A	Camera B	Camera A	Camera B	Camera A	Camera B	Camera A	Camera B
	+--	+++	-++	---	+--	---	-++	+++
d1	−136.790	136.545	136.790	−136.545	−136.790	−136.545	136.790	136.545
d2	−131.450	131.159	131.450	−131.159	−131.450	−131.159	131.450	131.159
d3	−128.906	128.656	128.906	−128.656	−128.906	−128.656	128.906	128.656
${‖d_{\mathrm{A}}-d_{\mathrm{B}}‖}_2$	458.272		458.272		0.455		0.455

An operator sgn() is a symbol that indicates two rotation matrices $A_i$ by + and -. Two possible solutions are marked in red and blue.

, ${‖d_{\mathrm{A}}-d_{\mathrm{B}}‖}_2$ can be utilized to eliminate two incorrect solutions with higher differences of distance d. The remaining two solutions are marked in red (as solution A) and blue (as solution B) in Table 5.

One way to find the last ambiguity is to plot the geometric relationships for each of these two solutions and observe which scenario matches the real physical situation depicted in Figure 6. Figure 10 illustrates the geometric relationships of solution A (Figure 10a) and solution B (Figure 10b). It can be easily noticed that solution B is the solution that matches the real physical situation depicted in Figure 6.

Another way to resolve the ambiguity is to use stereo deflectometry technology. For the anchor point highlighted with the laser shown in Figure 7, the ambiguity solution can be filtered by the residual of the angle θ minimization between the normal, calculated from camera A and B.

$$\underset{t}{\widehat{t}=\arg \min }〈{\mathbf{normal}}_{\mathrm{A}}\left(t\right),{\mathbf{normal}}_{\mathrm{B}}\left(t\right)〉.$$

(31)

As shown in Figure 11, the anchor point Z must hit in the surface under test, when the searching depth t makes the angle between the calculated normal_A and normal_B equal to zero. The ambiguity solution will make the minimization residual very large.

2.5. Refine Calibration Results by the Bundle Adjustment

Finally, a bundle adjustment with distortion compensation should be performed by Equation (32) to refine all parameters in the above calibration process for accuracy [54]

$$‖{∆}^{u}p‖={\displaystyle \sum _{j=1}^N{\displaystyle \sum _{i=1}^3‖{}^{u}p_{\mathrm{d}}\left(m_x^{\mathrm{A}},m_y^{\mathrm{A}},{}_{\mathrm{L}}{}^{\mathrm{A}}R,{}_{\mathrm{L}}{}^{\mathrm{A}}T,m_x^{\mathrm{B}},m_y^{\mathrm{B}},{}_{\mathrm{L}}{}^{\mathrm{B}}R,{}_{\mathrm{L}}{}^{\mathrm{B}}T,{}^{\mathrm{L}}n_1,{}^{\mathrm{L}}n_2,{}^{\mathrm{L}}n_3,d_1,d_2,d_3,{p_{\mathrm{L}}}_{ij}\right){-}^{u}p\left({p_{\mathrm{L}}}_{ij}\right)‖}},$$

(32)

where N is the point sampling number at pose i. ^Ln is the normal in coordinate system {L}. p_Lij represents the LCD sampling coordinates at pose i and sampling order j. The integer sampling real image coordinates can be represented as ^up(p_Lij) according to the phase-shifting method. The optimized image decimal coordinates ^up_d can be calculated from the optimization parameters. Because the phase-shifting method is needed to retrieve the LCD sampling coordinates p_Lij, the u and v coordinates of ^up are all integer camera pixel indices (as in the blue dots in Figure 12). There will always be some deviation Δ^up between the optimized image coordinates ^up_d and the real image integer coordinates ^up. Here, ^up_d is noted as the distorted image coordinates ^up_d = [u_d, v_d]^T (as in the red crosses in Figure 12). So u_d and v_d are decimal numbers. The relationship between ^up_d and ^up_d is

$$\begin{array}{c}{∆}^{u}p={}^{u}p_d-{}^{u}p\\ \left[\begin{array}{c}∆u\\ ∆v\end{array}\right]=\left[\begin{array}{c}{u}_{\mathrm{d}}\\ v_{\mathrm{d}}\end{array}\right]-\left[\begin{array}{c}u\\ v\end{array}\right]\end{array}.$$

(33)

After the bundle adjustment, the calibration results are shown in

Table 6. The calibration results after the bundle adjustment.

Parameters	Camera A	Camera B
magnification	$m_x^{\mathrm{A}}=0.05790$	$m_x^{\mathrm{B}}=0.05796$
	$m_y^{\mathrm{A}}=0.05628$	$m_y^{\mathrm{B}}=0.05609$
Rotation matrix between the flat LCD and Camera	${}_{\mathrm{L}}{}^{\mathrm{A}}R=\left[\begin{array}{ccc}-0.98765& 0.02942& 0.15388\\ 0.08725& 0.71251& -0.69621\\ -0.13012& -0.70104& -0.70115\end{array}\right]$	${}_{\mathrm{L}}{}^{\mathrm{B}}R=\left[\begin{array}{ccc}-0.98798& -0.12017& -0.09725\\ -0.01597& 0.70510& -0.70892\\ 0.15376& -0.69885& -0.69855\end{array}\right]$
Translation vector between the flat LCD and Camera	${}_{\mathrm{L}}{}^{\mathrm{A}}T=\left[\begin{array}{c}461.28868\\ -254.18019\end{array}\right]$	${}_{\mathrm{L}}{}^{\mathrm{B}}T=\left[\begin{array}{c}409.74525\\ -282.33540\end{array}\right]$
The normals of the flat mirror	${}^{\mathrm{L}}n_1=\left[\begin{array}{c}-0.05334\\ -0.45445\\ -0.88914\end{array}\right],{}^{\mathrm{L}}n_2=\left[\begin{array}{c}-0.06838\\ -0.45371\\ -0.88849\end{array}\right],{}^{\mathrm{L}}n_3=\left[\begin{array}{c}-0.06999\\ -0.45954\\ -0.88536\end{array}\right]$
The distances between the origin of {L} and the flat mirror	$d_1=135.633123,d_2=133.37208,d_3=130.78636$

. The calibration procedure for the proposed stereo bi-telecentric phase-measuring deflectometry, which requires only a flat mirror, is illustrated in Figure 13.

The translation vector ${}_{\mathrm{L}}{}^{\mathrm{A}}T$ and ${}_{\mathrm{L}}{}^{\mathrm{B}}T$ in Table 6 are the 2D translation vectors. The physical implication is that the bi-telecentric imaging system cannot perceive depth information, meaning its spatial position can be at any location along its optical axis. To facilitate subsequent surface reconstruction in 3D space, the depth information component T_z is added to the 2D translation vector. When T_z is 0, 50, or 100, the schematic diagram of the position of the bi-telecentric imaging system in the coordinate system is shown in Figure 14. The value of T_z does not affect the distribution of the incident light rays for imaging. Therefore, to facilitate the subsequent coordinate transformation calculations, the third dimension T_z = 0 is added to the translation vector ${}_{\mathrm{L}}{}^{\mathrm{A}}T$ and ${}_{\mathrm{L}}{}^{\mathrm{B}}T$ to clearly define the relative spatial position of the bi-telecentric imaging system. Let the previous 2D translation vectors ${}_{\mathrm{L}}{}^{\mathrm{A}}T$ and ${}_{\mathrm{L}}{}^{\mathrm{B}}T$ be denoted as ${}_{\mathrm{L}}{}^{\mathrm{A}}T_{2x1}$ and ${}_{\mathrm{L}}{}^{\mathrm{B}}T_{2x1}$, respectively. After adding the depth component T_z = 0, the resulting 3D translation vectors are

$$\begin{array}{l}{}_{\mathrm{L}}{}^{\mathrm{A}}T=\left[\begin{array}{c}461.28868\\ -254.18019\\ 0\end{array}\right]\\ {}_{\mathrm{L}}{}^{\mathrm{B}}T=\left[\begin{array}{c}409.74525\\ -282.33540\\ 0\end{array}\right]\end{array}.$$

(34)

3. Accurate Inverse Distortion Mapping

3.1. Anchor Point Positioning Procedure

To achieve accurate surface measurements, it is essential to obtain precise anchor points Z [55,56] from the calibrated parameters. The calculation procedure of anchor point Z is shown in Figure 15. For a give pixel ^up of camera A,

(a1)

Use the phase-shifting method to retrieve its corresponding LCD point Q_A.

(a2)

According to the calibration residuals, find ^up’s corresponding distorted image coordinate ^up_d = [u_d, v_d]^T from the distortion map D(u, v).

(a3)

According Equation (35), retrieve the x and y coordinates p_A = [x_A, y_A]^T of 2D anchor point Z in camera A’s local coordinate {A}.

$$\left[\begin{array}{c}{u}_d\\ v_d\\ 1\end{array}\right]=\left[\begin{array}{ccc}{m}_x^{\mathrm{A}}& 0& 0\\ 0& m_y^{\mathrm{A}}& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_{\mathrm{A}}\\ y_{\mathrm{A}}\\ 1\end{array}\right].$$

(35)

(a4)

The bi-telecentric lenses cannot identify the depth. By setting the searching depth as t, Z_A can be noted as Z_A = [x_A, y_A, t]^T.

(a5)

According Equation (36), retrieve the 3D coordinates of anchor point Z from camera local coordinate {A} to LCD coordinate {L} as

$$Z_{\mathrm{L}}={}_{\mathrm{L}}{}^{\mathrm{A}}R^{-1}(Z_{\mathrm{A}}-{}_{\mathrm{L}}{}^{\mathrm{A}}T).$$

(36)

(a6)

According to the law of reflection, the calculated normal_A is the bisector of line $\overrightarrow{Z_{\mathrm{L}}{p}_{\mathrm{A}}}$ and $\overrightarrow{Z_{\mathrm{L}}{\mathrm{Q}}_{\mathrm{A}}}$.

(b1)

According Equation (37), retrieve the 2D coordinates of anchor point Z in camera local coordinate {B} as

$$Z_{\mathrm{B}}={({}_{\mathrm{L}}{}^{\mathrm{B}}R{Z}_{\mathrm{L}})}_{2x1}+{}_{\mathrm{L}}{}^{\mathrm{B}}T_{2x1}.$$

(37)

(b2)

The corresponding distorted image coordinate ^up_d = [u_d, v_d]^T of Z_B can be calculated from

$$\left[\begin{array}{c}{u}_d\\ v_d\\ 1\end{array}\right]=\left[\begin{array}{ccc}{m}_x^{\mathrm{B}}& 0& 0\\ 0& m_y^{\mathrm{B}}& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_{\mathrm{B}}\\ y_{\mathrm{B}}\\ 1\end{array}\right].$$

(38)

(b3)

Find distorted image coordinate ^up_d’s corresponding pixel coordinate ^up = [u, v]^T of camera B from the inverse distortion map D⁻¹(u_d, v_d).

(b4)

Use the phase-shifting method to retrieve its corresponding LCD point Q_B.

(b5)

The calculated normal_B is the bisector of line $\overrightarrow{Z_{\mathrm{L}}{p}_{\mathrm{B}}}$ and $\overrightarrow{Z_{\mathrm{L}}{\mathrm{Q}}_{\mathrm{B}}}$.

(c1)

By optimizing the searching depth t, the anchor point Z can be precisely positioned, when the angle θ between normal_A and normal_B is normal_B zero.

The schematic positioning procedure of anchor point Z is shown in Figure 15. From the positioning process of anchor point Z, it can be found how to accurately achieve the distortion map D(u, v) in Procedure (a2), and the inverse distortion map D⁻¹(u_d, v_d) in Procedure (b3) is the key to determining the accuracy of the anchor point Z calculation.

3.2. Build Distortion Maps

As the pixel coordinates of ^up, the u and v are all integer values. There will always be some deviation between the optimized image coordinates ^up_d and the real image integer coordinates ^up. So u_d and v_d are decimal values. The relationship between ^up_d and ^up_d is

$$\left[\begin{array}{c}{u}_{\mathrm{d}}\\ v_{\mathrm{d}}\end{array}\right]=\left[\begin{array}{c}u\\ v\end{array}\right]+\left[\begin{array}{c}∆u\\ ∆v\end{array}\right].$$

(39)

Note Equation (39) as the distortion map D() between the integer pixel ^up and the distorted decimal pixel ^up_d. Therefore, Equation (33) can be formed as

$$\left\{\begin{array}{c}{u}_{\mathrm{d}}=D_u(u,v)\\ v_{\mathrm{d}}=D_v(u,v)\end{array}\right..$$

(40)

Because ^up is the 2D integer tabulated coordinate, the bicubic Hermite interpolation can be utilized as the distortion map, whose interpolation control grid is composed by the 2D integer tabulated coordinate [u, v]^T [57].

3.3. Use Polynomial Fitting Method to Build Inverse Distortion Maps

To build the inverse distortion map D⁻¹() between the distorted decimal pixel [u_d, v_d]^T and the integer pixel [u, v]^T, a natural idea is to perform polynomial fitting on the scatter pointset (u_d, v_d, Δu) shown in Figure 16a. The inverse distortion map Δu = D⁻¹(u_d, v_d) constructed by a polynomial fitting is shown in Figure 16b. It can be seen that there will be some fitting residuals, as shown in Figure 16c. Thus, the inverse distortion mapping constructed by polynomial fitting has low mapping accuracy due to the presence of fitting residuals.

3.4. Use Bicubic Hermite Interpolation to Build Inverse Distortion Maps

The inverse distortion map D⁻¹ can also be accurately achieved by a smooth triangulation-based interpolation for the 2D look-up table (u_d, v_d, Δu). (It is not recommended to use linear interpolation because the interpolating function is not differentiable at the control points). However, applying the analytical derivatives framework or the automatic differentiation framework into this triangulation-based interpolation is almost impossible [58]. The characteristics of the above two common inverse distortion mapping methods are shown in

Table 7. The characteristics of various inverse distortion mapping methods.

The Inverse Distortion Mapping Method	The Method Preserves the Detailed Information for Accuracy	The Automatic Differentiation Framework is Applied Easily
Polynomial fitting	No	Yes
Triangulation-based interpolation	Yes	No
Bicubic Hermite interpolation (proposed)	Yes	Yes

. To obtain the reconstruction results accurately and efficiently in deflectometry, the inverse distortion mapping should be constructed as follows: detailed preserved and automatic differentiable. Thus, the bicubic Hermite interpolation is utilized to build the inverse distortion mapping [59].

To further illustrate the construction mechanisms of the inverse distortion map, the cubic Hermite interpolation for dimension u is taken as an example. The control points (u, u_d) are the red pots in Figure 17a. Thus, the original distortion map D(u) = u_d can be built by a cubic Hermite interpolation of (u, u_d) (the yellow curve in Figure 17a). The resampled control points (u’, u_d’) (the blue dots in Figure 17a) can be retrieved from

$$u^{\prime }=\underset{u^{\prime }}{\arg \min }〈{‖D(u)-\mathrm{round}[D(u)]‖}_2〉,$$

(41)

where resampled u_d’ = D(u’) is the integer value. Thanks to the fact that u_d’ has a monotonically increasing nature, the inverse distortion map constructed by cubic Hermite interpolation D⁻¹(u_d’) = u’ (shown in Figure 17b) can be constructed directly by swapping the dependent and independent variables of D(u’) = u_d’ (the blue curve in Figure 17a). It should be noted that the round [D(u’)] may not cover all integers in the interval, so a further interpolation [60] should be employed to fill the absence control points (the magenta dots in Figure 17a).

Similarly, the distortion maps u_d = D_u(u, v) and v_d = D_v(u, v) for camera A in u and v direction acquired by the calibration are shown in Figure 18a,c (only Δu and Δv components are shown). The inverse distortion mapping u = D_u⁻¹(u_d, v_d) and v = D_v⁻¹(u_d, v_d) (shown in Figure 18b,e) can be constructed by the bicubic Hermite interpolation in the same manner as

$$\left[u^{\prime },v^{\prime }\right]=\underset{u^{\prime },v^{\prime }}{\arg \min }〈{‖D_u\left(u,v\right)-\mathrm{round}\left[D_u\left(u,v\right)\right]‖}_2+{‖D_v\left(u,v\right)-\mathrm{round}\left[D_v\left(u,v\right)\right]‖}_2〉.$$

(42)

The accuracy of the constructed inverse distortion maps can be indicated by the reprojection errors Error_u and Error_v according to Equation (43). The reprojection errors are shown in Figure 18c,f. The construction results of the inverse distortion maps for camera B are shown in Figure 19. The majority reprojection errors in both u and v directions for both cameras are both below 0.1 pixel. This is lower than the reprojection errors of the polynomial fitting method.

$$\left\{\begin{array}{c}Erro{r}_u=D_u^{-1}\left(D_u\left(u,v\right),D_v\left(u,v\right)\right)-u\\ Erro{r}_v=D_v^{-1}\left(D_u\left(u,v\right),D_v\left(u,v\right)\right)-v\end{array}\right.$$

(43)

4. Accuracy Investigation

4.1. Repeatability

After the holistic calibration of the stereo bi-telecentric PMD system and the construction of the mutual distortion maps, the surface under test can be retrieved from the shape reconstruction algorithms [48,49,50]. In order to quantitatively evaluate the repeatability of the proposed stereo bi-telecentric PMD, a concave mirror (stock #43-553) from Edmund with λ/4 surface accuracy was measured four times at the same position. The experimental setup is shown in Figure 20. The red dots in Figure 20 are indicated by the laser pointer for anchor point positioning. And the blue frames are the bounding boxes of the red dots. The results of the last three surface measurements differ from the first one and are shown in Figure 21. For the measurement of this mirror, the repeatability error of proposed stereo bi-telecentric PMD is less than 0.1 μm.

4.2. Reproducibility

In order to quantitatively evaluate the reproducibility of the proposed stereo dual telecentric PMD, five different aperture and curvature mirrors were used to measure at different positions in the measurement space. The reference radii of all mirrors have been calibrated by the 3D optical profilometer LUPHOScan 260 HD from Taylor Hobson. The corresponding measurement errors are summarized in Figure 22.

The geometric relationships between the various measured mirrors and the calibrated stereo bi-telecentric PMD in {L} are shown in Figure 23. The study shows that the proposed approach can achieve a measurement accuracy of less than 3.5 μm (Peak-to-Valley value) within 100 mm (Width) × 100 mm (Height) × 200 mm (Depth) domain for various surfaces. The proposed method does not require the help of a micro-positioning stage in the bi-telecentric lens calibration process to resolve the ambiguity of the rotation matrix in the extrinsic parameters. The accurate calibration results and inverse distortion maps allow for obtaining reliable measurement results without restricting the placement of the SUT.

5. Conclusions

This work combines the Hesch’s stereo PMD calibration method (suitable for endocentric lenses) and Li’s telecentric lens calibration method (a translate stage is needed), and develops the 3D geometric relationship of flat screen, endocentric cameras and flat mirrors to the 2D geometric relationship of flat screen, bi-telecentric cameras and flat mirrors through the formula derivation. The detailed derivation process is used to demonstrate the rigor of the method and to derive how to avoid using the translate stage and even chessboard during the calibration process.

The method proposed in this work successfully avoids the use of the translate stage in any calibration procedure for the stereo bi-telecentric PMD system by filtering the reprojection errors, which significantly enhances the efficiency and reliability of the measurement. In order to obtain accurate measurement results, the calibration residuals of camera pixels are utilized to construct the inverse distortion map through bicubic Hermite interpolation to obtain an accurate anchor positioning result.