A Single-Shot 3D Measuring Method Based on Quadrature Phase-Shifting Color Composite Grating Projection

Abstract

A single-shot three-dimensional measuring method based on quadrature phase-shifting color composite grating projection is proposed. Firstly, three quadrature phase-shifting sinusoidal gratings are encoded in red (R), green (G), and blue (B) channels respectively, composed single- frame color composite grating. This color composite grating is projecting obliquely on the object by DLP. After that, the color camera which is placed in a specific location is used to capture the corresponding color deformed pattern and send it to the PC. Then, by color separation, the color deformed pattern is demodulated as the corresponding three-frame monochromatic deformed patterns with a shifted quadrature phase. Due to the existences of sensitivity differences and color crosstalk among the tricolor channels, we propose a gray imbalance correction method based on the DC component’s consistency approximation. By the established 3D reconstruction physical model, the measurement of 3D shape can be achieved. Many experimental results for static and moving objects prove the proposed method’s feasibility and practicability. Owing to the single-shot feature of the proposed method, it has a good application prospect in real-time and high-speed 3D measurement.

1. Introduction

In the past few decades, the optical three-dimensional (3D) shape measuring method with the structured light projection has an enormous application in face recognition, industrial inspection, and so on [1,2,3,4,5,6]. With the improvement of photoelectric device technology, it has an increased performance in some fields, such as modern industry [7] and medical treatment [8]. For measuring static objects, multiple-shot methods [9,10] have higher measurement accuracy than single-shot methods. However, the image quality will be degraded due to the vibration, environmental noise, and other interferences in the image shooting gap. The single-shot method [11] only needs one image and has less sensitivity to vibration noise. Many single-shot methods are proposed. Takeda et al. proposed the method of Fourier transform profilometry (FTP) [12,13] which can reconstruct the 3D shape by only one image. However, due to the filtering process, its measuring accuracy has some disadvantages in the actual measurement process. He et al. proposed a 3D measurement method with orthogonal composite grating aided by fringe contrast and background calibration [14]. The contrast and background of the demodulated deformed fringes can be calibrated by these ratio coefficients to achieve the measurement of 3D shape. Li et al. proposed a computer-generated Moiré Profilometry to get the 3D shape of the objects [15,16]. Wan et al. proposed a single-shot 3D measurement based on hue-height mapping [17]. This method calculated the phase with the hue information of the color fringe pattern pixels to obtain the height distribution. Huang P et al. proposed the method of single-shot color phase measuring profilometry (PMP) [18], a single-shot color grating whose major color components are designed as three sinusoidal fringes having a shifted phase of $2\pi /3$. When this color composite grating is used to measure objects, the image sensor captures a corresponding color deformed pattern and sends it to the PC to achieve 3D measurement. It may be introduced some errors by the existences of sensitivity differences and color crosstalk [17,18,19,20,21,22,23] among the tricolor channels in the measuring process, which are manifested in the inconsistency of the intensity of the tricolor components of the color deformed pattern. Many scholars have proposed corresponding correction algorithms to solve these problems of the existences of sensitivity differences and color crosstalk. For example, Cao Yiping et al. proposed a function of chroma transfer function (CTF) to improve tricolor measuring accuracy. Because the CTF can well calibrate the color crosstalk and the grayscale imbalance, this method successfully reduces the errors caused by the above issues in the fast phase measuring profilometry [24]. Pan Jiahui et al. proposed the method based on tricolor camera combined with a tricolor filter for the calibration of color crosstalk coefficients and the compensation of gray scale imbalance problems [25,26]. However, this method is too complex due to additional cameras and filters. Guangkai Fu et al. proposed a 3D measurement with the color binary grating [27]. This method may avoid the effect of color crosstalk by projecting the binary grating. However, because its binary grating projection requires filtering, its measuring accuracy is still insufficient compared to the sinusoidal grating projection.

A 3D measuring method with single-shot quadrature phase-shifting color composite grating projection is proposed. Three quadrature phase-shifting sinusoidal gratings are encoded into tricolor channels, respectively, to compose a frame of color composite grating. This color composite grating is projected obliquely on the object by DLP to get the corresponding color deformed pattern with a camera. Then, by color separating, this color deformed pattern is demodulated as the corresponding three frames of monochrome deformed patterns with the shifted phase of $\pi /2$. Because this method is a non-full cycle equal-step phase-shifting algorithm, the sum of shifted phases within a cycle is $\pi$. The background image is well exacted by the sum of red and blue components. A gray imbalance correction method based on the DC component’s consistency approximation is proposed to correct the image caused by the existences of sensitivity differences and color crosstalk. By the established 3D reconstruction physical model, the measurement of 3D shape can be achieved.

2. Principle

The schematic diagram of the optical 3D measurement with a grating projection is shown in Figure 1. DLP obliquely projects a frame of coded grating onto the object to be measured. The camera which is placed in a specific location captures the corresponding deformed pattern and sends it to the PC. The PC can effectively restore the 3D shape of objects with the corresponding algorithms.

2.1. The Phase Measuring Principle of the Traditional Color PMP

In the PMP of the traditional single-shot color grating projection, tricolor channels of the color grating are encoded with three sinusoidal gratings which have a shifted phase of $2\pi /3$. The captured corresponding color deformed pattern is expressed as:

$$I_C(x,y)=I_R(x,y)\overrightarrow{R}+I_G(x,y)\overrightarrow{G}+I_B(x,y)\overrightarrow{B}$$

(1)

where $I_R(x,y)$, $I_G(x,y)$, and $I_B(x,y)$ can be expressed as:

$$\{\begin{array}{l}{I}_R(x,y)=A(x,y)+B(x,y)\cos [2\pi fx+\phi (x,y)]\\ I_G(x,y)=A(x,y)+B(x,y)\cos [2\pi fx+2\pi /3+\phi (\mathrm{x},\mathrm{y})]\\ I_B(x,y)=A(x,y)+B(x,y)\cos [2\pi fx+4\pi /3+\phi (\mathrm{x},\mathrm{y})]\end{array}$$

(2)

with $(x,y)$ the pixel coordinates of the captured image. The DC component $A(x,y)$, the contrast $B(x,y)$, the frequency $f$, and the modulated phase $\phi (x,y)$ are parameters of the deformed pattern. $\phi (x,y)$ can be calculated as:

$$\phi (x,y)=\arctan \left\{\frac{\sqrt{3}[I_R(x,y)-I_B(x,y)]}{2I_G(x,y)-I_R(x,y)-I_B(x,y)}\right\}$$

(3)

2.2. The Phase Measuring Principle of Single-Shot Quadrature Phase-Shifting Color Composite Grating

Figure 2 shows a single-shot quadrature phase-shifting color composite grating:

The three sinusoidal gratings designed with a shifted phase of $\pi /2$, as shown in Figure 2a, are, respectively, encoded into the tricolor channels and compose a frame of the quadrature phase-shifting color composite grating, as shown in Figure 2b. The captured corresponding color deformed pattern obtained is expressed as:

$$I_C(x,y)=I_R(x,y)\overrightarrow{R}+I_G(x,y)\overrightarrow{G}+I_B(x,y)\overrightarrow{B}$$

(4)

where $I_R(x,y)$, $I_G(x,y)$, and $I_B(x,y)$ are the intensity distribution of three components in the tricolor channels, respectively. Their mathematical model is expressed as:

$$\{\begin{array}{l}{I}_R(x,y)=A(x,y)+B(x,y)\cos [2\pi fy+\phi (x,y)]\\ I_G(x,y)=A(x,y)+B(x,y)\sin [2\pi fy+\phi (x,y)]\\ I_B(x,y)=A(x,y)-B(x,y)\cos [2\pi fy+\phi (x,y)]\end{array}$$

(5)

$\phi (x,y)$ is calculated as:

$$\varphi (x,y)=\arctan \left\{\frac{2I_G(x,y)-\left[I_R(x,y)+I_B(x,y)\right]}{I_R(x,y)-I_B(x,y)}\right\}$$

(6)

2.3. Gray Imbalance Correction Method

In the actual measurement process, these demodulated monochrome phase shifting deformed patterns have problems of gray imbalance due to the existences of sensitivity differences and color crosstalk among the three color channels, as mentioned in the Introduction, seriously affecting the measuring accuracy; thus, it is necessary to correct the above three-frame monochromatic deformed patterns.

The existences of sensitivity differences and color crosstalk are mainly generated by the inconsistency of white balance and spectral mismatch between the digital projector and CMOS camera. Therefore, the monochrome phase shifting deformed patterns demodulated by the captured single frame of color deformed pattern are used for coefficient compensation to make the three-frame monochrome phase shifting deformed patterns intensity distribution map closer to the same. The coefficient compensation method is proposed. First, the intensity of one component of the three-frame demodulated monochromatic deformed patterns is selected as the reference, and the other two components are multiplied by a compensation coefficient to make the intensity of the three components basically the same. Taking the green component as the benchmark here, the three-frame corrected monochrome deformed patterns are expressed as:

$$\{\begin{array}{l}{I}_{RM}(x,y)=K_{GR}*\left(I_R\right(x,y)-I_{\mathrm{G}0}+B_{GRP})+I_{\mathrm{G}0}\\ I_{GM}(x,y)=I_G(x,y)\\ I_{BM}(x,y)=K_{GB}*\left(I_B\right(x,y)-I_{\mathrm{G}0}+B_{GBP})+I_{\mathrm{G}0}\end{array}$$

(7)

where $I_{RM}(x,y)$, $I_{GM}(x,y)$, and $I_{BM}(x,y)$ represent the intensity distribution diagrams of the tricolor channels after correction. $K_{GR}$ and $K_{GB}$ are the compensation coefficients, $B_{GRP}$ and $B_{GBP}$ are the offsets of the zero phase point of the monochromatic phase shifting deformed pattern, and $I_{G0}$ is the intensity value of the zero phase point of the green component. They can be expressed as:

$$\begin{array}{l}{K}_{GR}=\frac{{\displaystyle \sum _{x=1}^M{I}_{RPEAKS}(y)}}{{\displaystyle \sum _{x=1}^M{I}_{GPEAKS}(y)}}\hfill \\ K_{GB}=\frac{{\displaystyle \sum _{x=1}^M{I}_{BPEAKS}(y)}}{{\displaystyle \sum _{x=1}^M{I}_{GPEAKS}(y)}}\hfill \\ B_{GRP}=\frac{{\displaystyle \sum _{x=1}^M{\displaystyle \sum _{y=1}^N(I_G(x,y)}-I_R(x,y))}}{M\times N}\hfill \\ B_{GBP}=\frac{{\displaystyle \sum _{x=1}^M{\displaystyle \sum _{y=1}^N(I_G(x,y)}-I_B(x,y))}}{M\times N}\hfill \\ I_{G0}=\frac{{\displaystyle \sum _{x=1}^M{\displaystyle \sum _{y=1}^N{I}_G(x,y)}}}{M\times N}\hfill \end{array}$$

(8)

where $I_{RPEAKS}\left(y\right)$, $I_{GPEAKS}\left(y\right)$, and $I_{BPEAKS}\left(y\right)$, respectively, represent the average of the peak values of each cycle of the row of monochromatic phase shifting deformed patterns of the tricolor channels. $M,N$ are the height and width of the image.

In the proposed measurement method, the numerator in the calculated truncated phase (Equation (6)) is the sum of the two components of the red and blue channels. According to Equation (5), it can be seen that the designed quadrature color composite grating has a phase shifting between the components in the red and blue channels, and the half sum of these components should be the background image, as shown in Equation (9):

$$I_{RB0}=(I_{RM}+I_{BM})/2=A(x,y)$$

(9)

Due to the existence of sensitivity differences and color crosstalk in the color deformed pattern, the half sum of the red and blue components may show volatility to some extent. According to the method of low-pass filtering, the Fourier spectrum of $I_{RB0}(x,y)$ is acquired by Fast Fourier Transform. Through Gaussian filtering window, the zero-frequency components can be extracted by filtering out higher-order harmonic components of that Fourier spectrum. Then, this zero-frequency component is transformed by inverse Fourier transform to get a “uniform light field” $I_{RB1}(x,y)$ which is closer to the actual value. Due to the influence of filtering, the high-frequency energy is removed and some errors are introduced. To reduce the error caused by filtering, a gray imbalance correction method based on the DC component’s consistency approximation is proposed. In this method, by the least square method, an evaluation function, is introduced to compare the similarity of two images, which is expressed as:

$$E={\displaystyle \sum _{x=1}^M{\displaystyle \sum _{y=1}^N{[I_{RB0}(x,y)-k*I_{RB1}(x,y)]}^2}}$$

(10)

where $E$ is the square of the gray difference value of each pixel of the image between $I_{RB0}(x,y)$ and $I_{RB1}(x,y)$. When $E$ takes the minimum value, $dE/dk$ is zero, which is expressed as:

$$\frac{dE}{dk}=\frac{{\displaystyle \sum _{x=1}^M{\displaystyle \sum _{y=1}^N{[I_{RB0}(x,y)-k*I_{RB1}(x,y)]}^2}}}{dk}=0$$

(11)

The value of $k$ is calculated to be:

$$k=\frac{{\displaystyle \sum _{x=1}^M{\displaystyle \sum _{y=1}^N{I}_{RB0}(x,y)I_{RB1}(x,y)}}}{{\displaystyle \sum _{x=1}^M{\displaystyle \sum _{y=1}^N{I}_{RB1}(x,y)I_{RB1}(x,y)}}}$$

(12)

The half sum of red and blue can be calibrated as:

$$I_{RBM}=k*I_{RB1}$$

(13)

Thus, Equation (6) can be rewritten as:

$${\varphi }_M(x,y)=\arctan (\frac{2*(I_G(x,y)-I_{RBM}(x,y))}{I_R(x,y)-I_B(x,y)})$$

(14)

Due to the arctan function, ${\phi }_M(x,y)$ is wrapped and should be unwrapped to be ${\mathsf{\Phi }}_M(x,y)$ [28]. In the same way, the corresponding unwrapped phase of the reference plane is acquired as ${\mathsf{\Phi }}_r(x,y)$ and the phase ${\mathsf{\Phi }}_h(x,y)$ that is caused by the height of measured object can be represented as:

$${\mathsf{\Phi }}_h(x,y)={\mathsf{\Phi }}_M(x,y)-{\mathsf{\Phi }}_r(x,y)$$

(15)

Thus, by the phase-to-height mapping [29,30] shown as

$$\frac{1}{h(x,y)}=a(x,y)+b(x,y)\frac{1}{{\mathsf{\Phi }}_h^{}(x,y)}+c(x,y)\frac{{\mathsf{\Phi }}_r(x,y)}{{\mathsf{\Phi }}_h^{}(x,y)}$$

(16)

the measurement of 3D shape is successfully achieved. $a(x,y)$, $b(x,y)$, and $c(x,y)$ are the experiment system parameters obtained through calibration.

3. Experiments and Analysis

3.1. Experimental System

Using the experimental platform shown in Figure 3, we measured the 3D shape of many measured objects with the proposed method to verify its feasibility and validity in the actual measurement process. In this experimental platform, the digital light projector is PLED-W20 produced by ViewSonic, and the color camera is DFM 72AUCO2 with the imaging resolution of 1024 × 768 pixels.

3.2. Experiments and Analysis

Many experiments measuring the 3D shape of many objects with the proposed method were carried out. Figure 4 shows one of the experimental results. Here, the face model shown in Figure 4a is used to be measured. Figure 4b shows the color fringe pattern that is caused by a reference plane, which is shown in Figure 3, captured by color camera, and the single-frame color deformed pattern of the measured object is shown in Figure 4c. The three-frame monochromatic fringe patterns demodulated in tricolor channels of the single-frame captured color fringe pattern are shown in Figure 4d–f and the three-frame monochromatic deformed patterns demodulated in tricolor channels of the single-frame captured color deformed pattern are shown as Figure 4g–i. Due to the overlapping among tricolor channels, the issues of the existence of sensitivity differences and color crosstalk are unavoidable. The experimental data show that the above issues exist and seriously introduce some errors in phase resolution. Thus, we need to correct the monochromatic phase-shifting deformed patterns before calculating the phase to reduce the errors introduced by the existences of sensitivity differences and color crosstalk.

The existences of sensitivity differences and color crosstalk among these monochromatic phase-shifting deformed patterns are corrected by the proposed method. The reference plane background image before gray imbalance correction is shown in Figure 5a; the reference plane background image after gray imbalance correction is shown in Figure 5b; the background image of the measured object before gray imbalance correction is shown in Figure 5c; and the background image of the object after gray imbalance correction is shown in Figure 5d. After comparison and analysis, we found that, by using the proposed method to correct the background, the errors caused by the existence of sensitivity differences and color crosstalk are further reduced. It must be pointed out that some blurriness in the corrected background image of measured object is caused because of the filtering process. Therefore, the proposed method has some disadvantages in measuring sharp objects.

Figure 6a shows the corresponding wrapped phase calculated by these monochromatic deformed patterns after correction. The reconstructed result of the measured face model is shown in Figure 6b. It is clearly shown that the 3D shape of measured face model is well restored, and this reconstructed result proves the feasibility of the proposed method in measurement of 3D shape. The restored result by the eight-step PMP was chosen as the quasi-truth value to analyze measurement accuracy of the proposed method. Then, a comparison experiment between the traditional color PMP and the proposed method was carried out.

Figure 7 shows the 547th column data of the reconstructed result by these methods. The reconstructed result of measured face model by the eight-step PMP is shown as a dotted line in Figure 7. The reconstructed result of measured object by the traditional single-shot color PMP is shown as a chain line in Figure 7. The reconstructed result of measured object by the proposed method is shown as a solid line in Figure 7. It is obvious that the reconstructed 3D shapes of the measured object with all three methods are well achieved. As is known, the large-step PMP has very high efficiency and measuring accuracy. Therefore, we take the data reconstructed by eight-step PMP as the reference standard. By comparing and analyzing the data of Column 548, Lines 378–408, 432–462, and 555–591 shown in Regions A–C in Figure 7a, which are the “bridge of nose”, “tip of nose”, and “jaw” parts of the face model, the reconstructed details of the above area after correction by the method are closer to the reconstructed results by eight-step PMP than those of the traditional color PMP. The comparison experimental results show that the detail accuracy of the proposed method is better than that of the traditional color PMP.

To further compare the measuring accuracy between the traditional single-shot color PMP and the proposed method, by the difference method, the restored result of the eight-step PMP was chosen as the quasi-truth value to calculate height error distribution of the above two methods. Figure 8a shows the height error distribution by traditional single-shot color PMP. Figure 8b shows the height error distribution by the proposed method. It is observed that the height error distribution of the proposed method is smaller than that of single-shot color PMP. To quantitatively compare and analyze the measurement errors of the traditional color PMP and the proposed method, the mean absolute error (MAE) and the root of mean square error (RMSE) of these methods are calculated. By traditional color PMP, the MAE is 0.2683 mm and the RMSE is 0.3453 mm. By the proposed method, the MAE is 0.1992 mm and the RMSE is 0.2546 mm. It is obvious that the measurement errors of the proposed method are smaller, and the measuring result is closer to that of the eight-step phase-shifting algorithm. The comparison of MAE and RMSE proves the validity of conclusions.

The experiments and analysis sufficiently proved that the proposed method has a good feasibility and validity in the measurement of 3D shape. In this work, we use a set of complementary deformations to extract the background. Because it has a $\pi$ shifted phase between red and blue channels and the interplay of them is lower, the background image is obtained with smaller errors. These complementary deformed patterns can effectively suppress random errors in the measuring process due to the feature of inverse phase between the red and the blue channels. Therefore, the proposed method has better measuring accuracy than the method of single-shot color PMP after the gray imbalance correction. It must be pointed out that there are some shortcomings of the proposed method for measuring colored objects. Because the captured deformed pattern is also modulated by the color of the object, some errors are introduced to result in lower measurement accuracy. However, it is single-shot so that it has higher speed and less sensitivity to mechanical disturbances, and it also has a good application prospect in real-time measurement.

3.3. Real-Time Measurement

To show the real-time measuring practicality of the proposed method, many real-time measuring experiments were also carried out. Here, the real-time measurement of the moving gloved hand is taken as an example. Figure 9a–c shows the deformed patterns at States A–C from the real-time recorded video. Figure 9d–f shows the deformed patterns of G component after correction. The corresponding reconstruction results of the gloved hand are shown as Figure 9g–i, respectively. It is observed that the different states of the moving gloved hand can be well restored by the proposed method. The real-time experimental results prove the practicality of the proposed method in real-time measurement process. In the real-time measurement, the frame rate of the recorded video is only 30 fps with the video resolution of 640 × 480 pixels due to the performance limitation of the camera. Because of its single-shot feature, if a higher performance DLP and camera are adopted, the frame rate can be higher.

4. Conclusions

A single-shot 3D measuring method based on quadrature phase-shifting color composite grating projection is proposed. A corresponding physical model of 3D reconstruction is derived systematically. The existence of sensitivity differences and color crosstalk among the tricolor channels are effectively corrected. Many experimental results prove the practicality and validity of the proposed method. Compared with the traditional color PMP, the proposed method uses a set of complementary deformed patterns to extract a purer background. Therefore, the proposed method has the better measuring accuracy than the method of traditional single-shot color PMP. Owing to its single-shot feature, it also has a good application prospect in high speed and real-time measurement.