Modeling and Calibration of Active Thermal-Infrared Visual System for Industrial HMI

Abstract

In the industrial application of the human-machine interface (HMI), thermal-infrared cameras can detect objects that are limited by visible-spectrum cameras. It can receive the energy radiated by a target through an infrared detector and obtain the thermal image corresponding to the heat distribution field on the target surface. Because of its special imaging principle, a thermal-infrared camera is not affected by the light source when imaging. Compared to visible-spectrum cameras, thermal imaging cameras can better detect defects with poor contrast but temperature differences or internal defects in products. Therefore, it can be used in many specific industrial inspection applications. However, thermal-infrared imaging has the phenomenon of thermal diffusion, which leads to noisy thermal infrared images and limits its applications in high precision industrial environments. In this paper, we proposed a high precision measurement system for industrial HMI based on thermal-infrared vision. The accurate measurement model of the system was established to deal with the problems caused by imaging noise. The experiments conducted suggest that the proposed model and calibration method is valid for the active thermal-infrared visual system and achieves high precision measurements.

1. Introduction

Computer vision has made good progress in the industry in recent years [1]. For visual measurement systems, cameras are widely used as measuring sensors [2,3,4]. Thermal-infrared cameras nowadays are getting more and more popular [5], but visible-spectrum cameras are still far more common. Thermal infrared imaging sensors have demonstrated excellent performance in unconventional environments and have historically found their use limited to military, security, and medical applications [6]. However, with increasing image quality and decreasing price, some of the thermal sensing devices are finding commercial deployment for home and office monitoring [7,8] as well as industrial HMI applications [9]. Thermal image sensors typically have a spectral sensitivity ranging from 7-micron to 14-micron bands of wavelength. The capacity of these imaging devices to appropriately capture images of objects depends on their emissivity and reflectivity [10]. In general, thermal-infrared cameras are superior or complementary to conventional visible-spectrum cameras in these aspects:

• More adaptive to poor or changing lighting;
• Effective for detecting an object with low contrast but differences in temperature;
• Can detect surface information of objects as well as internal information.

In some industry fields, the infrared camera plays an irreplaceable role in in-process inspection tasks [11]. For example, in an automated fiber placement inspection system, the defects inside composites are difficult to detect by visible-spectrum cameras because the black prepreg slit-tapes prevent a high visual contrast between the single tows and the plies on the tooling surface, but the thermal-infrared camera achieves good performance in detection of these defects inside composites [12].

However, research on infrared vision systems has been limited so far. Infrared cameras have not been widely used for industrial vision measurements because of blurred imaging and calibration difficulties to build reliable and efficient vision measurement systems with infrared-thermal image sensors. The objective of this paper is to address this concern.

A common visual measurement system used in industry is the eye-in-hand inspection system [13]. The camera is mounted at the end of a robot arm and scans the detected objects. The eye-in-hand inspection system should be calibrated before measurement. The calibration results are used to establish the corresponding relationship between the detected target and its image. For a visual measurement system based on a visible-spectrum camera, we can calibrate the system by the camera’s intrinsic parameters calibration and eye-in-hand calibration, which can acquire the relationship between the camera frame and the robot’s end frame [14]. However, for a visual measurement system based on a thermal-infrared camera, the method above is not feasible. On the one hand, the thermal infrared camera is unable to capture the calibration patterns of the visible-spectrum camera [15]. On the other hand, even if a calibration pattern with temperature difference is used, the thermal diffusion of the infrared image will cause blurring of the image and make it difficult to acquire calibration features. Figure 1a–c show the calibration board with temperature difference taken continuously by the thermal infrared camera, which can clearly show the phenomenon of thermal diffusion. The edges of the calibration pattern are not clear and become increasingly blurred.

In recent years, some researchers have been devoted to the geometric calibration of thermal-infrared cameras. In [16], the author conducted the calibration with the famous Zhang’s calibration method [17] by heating a chessboard calibration block. In many cases, hollow calibration targets based on burning lamps are very useful [18,19,20]. However, the procedure is too complex, and the size of the calibration object is very large, which is not suitable for the calibration of high-precision vision systems with small FOV (field of view). An alternative solution is creating calibration patterns by the combination of materials with different emissivity, which makes the calibration more convenient [21,22]. However, the imaging of calibration patterns in this way is not clear enough and still requires complex heating equipment to assist in heating the calibration target [20]. In addition to the lack of a good solution for the calibration target for geometric calibration, it is also difficult to process calibration images [23]. The corners in the thermal image are blurred and are thus extracted manually. As it is impossible to obtain the subpixel position of the corner accurately, the thermal-infrared camera cannot be calibrated accurately as well [24].

The main contribution of this paper is in proposing an active thermal infrared visual measurement system for high precision industrial HMI applications, which mainly consists of a thermal infrared camera, a heating lamp, and a robotic arm. Mathematical modeling and calibration methods for this system are also proposed, and the results are verified to achieve high accuracy measurements. The rest of this paper is organized as follows: In Section 2, system construction and working principles are provided. Section 3 proposes the mathematical model of the thermal-infrared visual measurement system. Section 4 introduces the calibration method. In Section 5, experiments are conducted to test the performance of the proposed method, finalizing it by presenting an application case for the industrial HMI.

2. System Construction and Working Principle

As shown in Figure 2, the thermal-infrared visual measurement system mainly consists of an industrial robotic arm, a thermal infrared camera, and a heating lamp. The robot arm uses an ABB IRB640 M2240 robot. The thermal infrared camera is an i3-thermal expert TE-EQ1, which has a resolution of 640 × 480 pixels and a frame rate of 60 fps. The wavelength range of the camera is 8–14 μm. Objects in the temperature range of −10 to 120 °C can be seen with this camera. The heating lamp is mounted on the left side of the end of the robotic arm, and the angle between them is about 0°. The thermal infrared camera is mounted on the right side of the end of the robotic arm, and the angle between them is about 45°. The distance between the camera and the detected target is about 15 cm. When the system starts to work, the heating lamp heats the surface of the inspected workpiece to create a temperature difference between defects and the surface of the workpiece. Therefore, the infrared camera can detect the defects of the workpiece. The thermal-infrared camera scans the workpiece with the movement of the robot arm to acquire infrared images of defects and measures the size and position of defects according to real-time infrared images. It has great potential applications for industrial HMI, especially for online inspection and real-time measurement of those defects that are difficult to be detected by visible-spectrum cameras. The accuracy of the system is mainly dependent on the mathematical modeling and calibration method, so an accurate calibration model and convenient calibration method are significant for the system.

3. Modeling of the Active Thermal-Infrared Visual Measurement System

To facilitate modeling of the thermal-infrared visual measurement system, coordinate frames were established, as shown in Figure 3. The workpiece coordinate system is denoted as {W}, the robot coordinate frame is {R}, the camera coordinate frame is {C}, and the end of the robot arm coordinate frame is {E}. A reference coordinate frame {S} is established, which is associated with {E}. Coordinate frames at time t are denoted as {C(t)}, {E(t)}, and {S(t)}.

The surface of the workpiece is modeled as a point set Ω and denoted as:

$$\mathsf{\Omega }:\left\{\left(x,y,z\right)|{\displaystyle \prod }\left(x,y,z\right)=0\right\}$$

(1)

where Ω describes the shape of the workpiece. It is an analytical formula or recorded as an engineering diagram. The position of point P in {W} is denoted as ^WP = [^wdx, ^way, ^wdz]. ^IP= [u, v, 1]^T denotes its corresponding point in the thermal infrared image. The aim is to achieve ^WP, which is the position of the point P in the workpiece coordinate frame. Using the pin-hole model of the camera, we can obtain:

$$M_{\mathrm{in}}^{}·{}_{ }{}^{\mathrm{C}\left(\mathrm{t}\right)}\mathbf{T}{ }_{\mathrm{W}}·{}_{ }{}^{\mathrm{W}}\mathbf{P}={ \mathrm{z}}_{\mathrm{c}}{}_{ }{}^{\mathrm{I}}\mathbf{P}$$

(2)

where M_in is the camera’s intrinsic parameters matrix, ^C(t)T_W represents the external parameters between {C} and {W}. M_in is a constant matrix. In the robotic vision system, the camera is fixed at the end of the robot and moves with the robot arm. Thus, ^C(t)T_W is associated with the robot’s posture. Existing methods transform Equation (2) to Equation (3):

$$M_{\mathrm{in}}^{ }·{}_{ }{}^{\mathrm{C}\left(\mathrm{t}\right)}\mathbf{T}{}_{\mathrm{E}\left(\mathrm{t}\right)}·{}_{ }{}^{\mathrm{E}\left(\mathrm{t}\right)}\mathbf{T}{ }_{\mathrm{W}}·{}_{ }{}^{\mathrm{W}}\mathbf{P}={\mathrm{z}}_{\mathrm{c}}{}_{ }{}^{\mathrm{I}}\mathbf{P}$$

(3)

M_in can be obtained by camera intrinsic parameters calibration. ^C(t)T_E(t) can be obtained by eye-in-hand calibration, and ^E(t)T_W can be obtained by the output of the robot arm. Then we can establish the relationship between ^IP and ^WP. However, as analyzed in the first part, in the thermal infrared visual system, existing calibration methods cannot obtain accurate M_in and ^C(t)T_E(t). Therefore, this paper obtains the relationship parameters of ^IP and ^WP through the following analysis and modeling. ^WP can be measured according to Equation (2):

$${}_{ }{}^{\mathrm{W}}\mathbf{P}{=\mathrm{z}}_{\mathrm{c}}{}_{ }{}^{\mathrm{W}}\mathbf{T}{ }_{\mathrm{C}\left(\mathrm{t}\right)}·{\mathbf{M}}_{\mathrm{in}}^{-1}·{}_{ }{}^{\mathrm{I}}\mathbf{P}$$

(4)

When t = 0, the coordinate original point of {S(0)}, which is denoted as O_s(0), is established in the camera’s FOV and O_s(0)∈Ω. Then we can transform ^WT_C(t) to ^WT_S(t) ^S(t)T_C(t) and get Equation (5):

$${}_{ }{}^{\mathrm{W}}\mathbf{P}{=\mathrm{z}}_{\mathrm{c}}{}_{ }{}^{\mathrm{W}}\mathbf{T}{ }_{S\left(t\right)}·{}_{ }{}^{\mathrm{S}\left(\mathrm{t}\right)}\mathbf{T}{ }_{C\left(t\right)}·{\mathbf{M}}_{\mathrm{in}}^{-1}·{}_{ }{}^{\mathrm{I}}\mathbf{P}$$

(5)

^WT_S(t) is divided as ^WT_S(0)$·$^S(0)T_S(t) and combined with Equation (5), we can get the following equation:

$${}_{ }{}^{\mathrm{W}}\mathbf{P}={\mathrm{z}}_c{}_{ }{}^{\mathrm{W}}\mathbf{T}{ }_{S\left(0\right)}·{}_{ }{}^{S\left(0\right)}\mathbf{T}{ }_{\mathrm{S}\left(\mathrm{t}\right)}·{}_{ }{}^{\mathrm{S}\left(\mathrm{t}\right)}\mathbf{T}{ }_{\mathrm{C}\left(\mathrm{t}\right)}·{\mathbf{M}}_{\mathrm{in}}^{-1}·{}_{ }{}^{\mathrm{I}}\mathbf{P}$$

(6)

The transformation between {S(0)} and {S(t)} is equal to the transformation between {E(0)} and {E(t)}:

$${}_{ }{}^{\mathrm{S}\left(0\right)}\mathbf{T}{ }_{\mathrm{S}\left(\mathrm{t}\right)}={}_{ }{}^{\mathrm{E}\left(\mathrm{t}\right)} {\mathbf{T}}_{\mathrm{E}\left(\mathrm{t}\right)}$$

(7)

Usually, ^E(0)T_E(t) can be easily obtained from the robot’s control system as the so-called TCP pose. It consists of two parts, the position ^WP_TCP = (^WX_TCP, ^WY_TCP, ^WZ_TCP) and the quaternion Q = (q₁, q₂, q₃, q₄). There are four parameters in Equation (6) that need to be solved. They are z_c, ^WT_S(0), M_in, and ^S(t)T_C(t). z_cM_in^C(t)T_S(t) is a constant value. M_in^C(t)T_S(t) is denoted as M. The imaging model between the coordinate frame {S(t)} and {W} can be established. The position of P in {S(t)} is denoted as ^S(t)P, so we can get the following equation:

$${\mathrm{z}}_{\mathrm{c}}{}_{ }{}^{\mathrm{I}}\mathbf{P}={\mathbf{M}}_{\mathrm{in}}·{}_{ }{}^{\mathrm{C}\left(\mathrm{t}\right)}\mathbf{T}{ }_{\mathrm{S}\left(\mathrm{t}\right)}·{}_{ }{}^{\mathrm{S}\left(\mathrm{t}\right)}\mathbf{P}=\mathbf{M}{·}^{\mathrm{S}\left(\mathrm{t}\right)}\mathbf{P}$$

(8)

The robot can be controlled to move along the workpiece’s surface at a certain distance to keep O_s(t) on the surface and is keeping O_s(t)∈Ω. When the camera’s FOV is small enough, the detected surface can be approximated as a plane. According to Equation (1), the function of the tangent plane at the original point can be calculated as:

$$A\left({}_{ }{}^{\mathrm{w}}x-{}_{ }{}^{\mathrm{wd}}x\right)+B\left({}_{ }{}^{\mathrm{w}}y-{}_{ }{}^{\mathrm{wd}}y\right)+C\left({}_{ }{}^{\mathrm{w}}z-{}_{ }{}^{\mathrm{wd}}z\right)=0$$

(9)

Additionally, the point in this plane is ^wdP = (^wdx, ^wdy, ^wdz). Therefore,

$$\left[\begin{array}{c}\begin{array}{c}{}_{ }{}^{\mathrm{w}}x\\ {}_{ }{}^{\mathrm{w}}y\end{array}\\ {}_{ }{}^{\mathrm{w}}z\\ 1\end{array}\right]={}_{ }{}^{\mathrm{W}}T{ }_{S\left(0\right)}·{}_{ }{}^{\mathrm{E}\left(0\right)}T{ }_{\mathrm{E}\left(\mathrm{t}\right)}\left[\begin{array}{c}\begin{array}{c}{}_{ }{}^{\mathrm{S}}x\\ {}_{ }{}^{\mathrm{S}}y\end{array}\\ {}_{ }{}^{\mathrm{S}}z\\ 1\end{array}\right]$$

(10)

we can solve ^WP by Equations (8)–(10). It can be seen that we can obtain ^WP from ^IP if M and ^WT_S(0) are calibrated. We refer to these two parameters as system parameters in the following.

4. Calibration Method

4.1. Calibration of System Parameters M

The system parameters M include the intrinsic parameters of the thermal-infrared camera and transform the matrix between {C(t)} and {S(t)}. As mentioned in Section 1, the calibration methods for a visible-spectrum camera can hardly be transplanted to thermal-infrared cameras. Calibration methods for thermal-infrared cameras are poorly studied, and existing methods cannot accurately calibrate thermal-infrared cameras. To overcome this problem, we proposed an active system parameter calibration method for the thermal-infrared visual measurement system. The robot is controlled to scan the detected target according to its mechanical models and make sure O_s(t)∈Ω and the z-axis of {E} coincides with the normal line of the surface at O_s(t). With these constraints, s(t)z = 0. Therefore, parameters m₁₃, m₂₃, and m₃₃ in M are unnecessary.

$$\mathbf{M}=\left[\begin{array}{cccc}{k}_x& 0& u_0& 0\\ 0& k_y& v_0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{cccc}{n}_x& o_x& a_x& p_x\\ n_y& o_y& a_y& p_y\\ n_z& o_z& a_z& p_z\\ 0& 0& 0& 1\end{array}\right]=\left[\begin{array}{cccc}{\mathrm{m}}_{11}& {\mathrm{m}}_{12}& {\mathrm{m}}_{13}& {\mathrm{m}}_{14}\\ {\mathrm{m}}_{21}& {\mathrm{m}}_{22}& {\mathrm{m}}_{23}& {\mathrm{m}}_{24}\\ {\mathrm{m}}_{31}& {\mathrm{m}}_{23}& {\mathrm{m}}_{33}& {\mathrm{m}}_{34}\end{array}\right]$$

(11)

According to the analysis above, a polygonal flat calibration target that is smaller than the camera’s FOV and is of negligible thickness can be used. We put it on the workpiece and make sure the calibration target is located on the tangent plane of the workpiece. The coordinate frame established on the calibration target is denoted as {S(0)} whose z-axis parallels the normal line of the tangent plane. In Equation (11), m₃₄ is the distance between the control points and the camera’s optical center, so m₃₄ ≠ 0 [23]. The following equation can be obtained,

$$\{\begin{array}{c}{\mathrm{m}}_{11}^{\prime }{}_{ }{}^{\mathrm{S}\left(\mathrm{t}\right)}x+{\mathrm{m}}_{12}^{\prime }{}_{ }{}^{\mathrm{S}\left(\mathrm{t}\right)}y+{\mathrm{m}}_{14}^{\prime }-{\mathrm{m}}_{31}^{\prime }{}_{ }{}^{\mathrm{S}\left(\mathrm{t}\right)}x\mathrm{u}-{\mathrm{m}}_{32}^{\prime }{}_{ }{}^{\mathrm{S}\left(\mathrm{t}\right)}y\mathrm{u}=\mathrm{u}\\ {\mathrm{m}}_{21}^{\prime }{}_{ }{}^{\mathrm{S}\left(\mathrm{t}\right)}x+{\mathrm{m}}_{22}^{\prime }{}_{ }{}^{\mathrm{S}\left(\mathrm{t}\right)}y+{\mathrm{m}}_{24}^{\prime }-{\mathrm{m}}_{31}^{\prime }{}_{ }{}^{\mathrm{S}\left(\mathrm{t}\right)}x\mathrm{v}-{\mathrm{m}}_{32}^{\prime }{}_{ }{}^{\mathrm{S}\left(\mathrm{t}\right)}y\mathrm{v}=\mathrm{v}\end{array}$$

(12)

where m′ = m/m₃₄. Now, there are eight parameters in Equation (12). Four control points are enough for solving all of them, as long as any three of them are not located on the same line. Using more control points can optimize the results. The calibration target can be designed like Figure 4a or Figure 4b. The edges of the calibration target in its thermal image can be detected, and the intersections of the detected edges are used as the control points.

4.2. Calibration of System Parameters ^WT_S(0)

The system parameter ^WT_S(0) is the transform matrix between {S(0)} and {W}.The coordinate frame {S(0)} is established on the calibration target, as shown in Figure 5. The origin of {S(0)} is a corner point of the rectangular calibration target. The x-axis and the y-axis are the sides of the calibration target. The vertical direction of the calibration target is the z-axis. The calibration target is placed on the x-y surface of the {W}. Since the calibration target is thin enough, it can be considered that the x-y surfaces of {W} and {S(0)} are on the same plane. There are only three independent variables in the transform matrix ^WT_S(0), including two translation variables and one rotation variable. It can be described as:

$${}_{ }{}^{\mathrm{S}\left(0\right)}\mathbf{T}{ }_{\mathrm{W}}=\left[ \begin{array}{c}\begin{array}{c}\cos \alpha \\ \sin \alpha \end{array}\\ 0\\ 0\end{array} \begin{array}{c}\begin{array}{c}-\sin \alpha \\ \cos \alpha \end{array}\\ 0\\ 0\end{array} \begin{array}{c}\begin{array}{c}0\\ 0\end{array}\\ 1\\ 0\end{array} \begin{array}{c}\begin{array}{c}{p}_x\\ p_y\end{array}\\ 0\\ 1\end{array}\right]$$

Since the size of the calibration target is too small compared to the size of the workpiece, it is difficult to directly measure the rotation angle α. Two calibration targets with edges on the same straight line are used to solve this problem. Thermal-infrared images of these two calibration targets are taken by the robot’s movement for measurement. In industrial applications, we can print two identical calibration patterns at both ends of the workpiece and ensure that the two corresponding edges of calibration targets are on the same straight line. We use different materials to print the calibration patterns and ensure they have a different thermal conductivity with the workpiece. Therefore, the calibration patterns can be imaged with the thermal infrared camera after heating.

The two calibration targets are denoted as cal.1 and cal.2. Firstly, the picture of cal.1 donated as Img1 is captured. Then, the robot is moved along the y-axis of the {W} until cal.2 appears in the camera’s FOV. Then, the picture of cal.2 is captured and donated as Img2. The distance of the robot’s movement is recorded as Δy. According to the corresponding points in Img1 and Img2 and the calibration results of M, the offset Δx can be calculated. Then, the angle α between {S(0)} and {W}can be denoted as:

$$\alpha =arctan\left(\frac{\Delta \mathrm{y}}{\Delta \mathrm{x}}\right)$$

(13)

When using the method, we should consider the robot’s accuracy and calibration accuracy. If the robot’s motion accuracy is high enough, the distance between the two calibration targets can be increased to optimize the calibration results. Otherwise, the distance should be shortened because the errors will make one of the calibration targets beyond the camera’s FOV.

4.3. Calibration Points Extraction Algorithm

For the imaging of a certain object, the thermal-infrared image is much blurrier than its visible-spectrum image. The continuous change in temperature at the edge of the object inevitably causes a blurring of the infrared image. Therefore, it is necessary to fit the sidelines of the calibration target and use their intersections as the calibration points. If using the edge extraction operator directly, a lot of interference points will be extracted and affect the correctness of sideline fitting.

To accurately extract calibration points, an approach is proposed in this article, as shown in Algorithm 1. First, an edge extraction operator is used to extract the approximate edge points. These points are inaccurate, but the region of the edge can be determined. Then, the Hough transform is used to fit these points into a straight line. In this case, several straight lines will be fitted near the true edge. Hough parameters are clustered in the Hough parameter space into c categories. The value of c is equal to the number of sides of the calibration target. After clustering, the mean value of each class is calculated and output as the representative parameter of those straight lines. The intersections of these c lines are calculated, and the noise points are removed. Then, c line segments can be obtained, and the detection region can be determined according to these line segments. The two endpoints of the segment are denoted as C1(x1, y1) and C2(x2, y2). The rectangular detection region, which needs to be determined, is denoted as a Region of Interest (ROI) (C1′, C2′), where C1′ = (x1′, y1′), C2′ = (x1′, y1′). The ROI can be obtained as follows:

x1′ = x1 + α(x2 − x1), y1′ = y1 + β(y1 − y2)

x2′ = x2 + γ(x1 − x2), y2′ = y2 + φ(y2 − y1)

where, α, β, γ, φ are constants (< 1). A row scan and a column scan are performed for each ROI separately. If the difference between the x-axis of the two intersections is bigger than the difference between the y-axis of the two intersections, this edge is the horizontal edge. The column scan will be used in the subsequent processing. Otherwise, it is a vertical edge, and a row scan will be adopted. Taking the column scan as an example, the scan mode is that each column of pixels is scanned row by row starting from the top left corner of the ROI, and then the grayscale value of the pixel is achieved. The gray value of the nth pixel is subtracted from the gray value of the (n + 4th) pixel, and the four sets of data with the largest difference in each column are calculated. The coordinates of the (n + 2nd) point are output as the searched edge points. The least-squares method is used to fit these edge points into a straight line as the extracted edge. The intersections of these edges are calibration points. The edge extraction algorithm is listed in Algorithm 1.

Algorithm 1: Sideline Extraction

Input: Image I Data container: edge point set Pe, line parameters set Lh, point set Pr Pe ← Canny (I); Lh ← Hough (IC); Lh ← Cluster (Lh); Lh ← SideDetection (Lh); For each line segment Lh:  Pe ← EdgeScan (Lh);  Lh ← Fitting (Pe); End Output: Lh

5. Experiments

The proposed calibration method was used to calibrate the thermal-infrared visual measurement system to evaluate the feasibility and accuracy. After calibration, standard workpieces were measured several times. By comparing the measurement value with the true value of the workpieces, the efficiency and accuracy of the proposed calibration method were verified.

5.1. Extraction of Calibration Points in the Thermal-Infrared Image

Figure 6 shows the improvement of the proposed sideline extraction method compared with the traditional one. The red lines mark the extraction results. We can see that the traditional method of directly using an edge detector extracts multiple small line segments at the edge of the calibration target, as the edges are too blurred. Meanwhile, the line segments extracted by our method are accurate. After the sidelines are extracted, the calibration points are obtained by their intersections.

5.2. Calibration of System Parameters M

According to the method proposed in Section 4.1, a small rectangular calibration target of size 20 mm × 40 mm was used to acquire the thermal-infrared images for system calibration. Using the calibration points extraction method proposed in Section 4.3, the four calibration points were extracted, and their image coordinates were A(260.58, 256.61), B(278.64, 144.45), C(627.01, 148.36), and D(642.30, 261.41). In {S(0)}, the coordinates of the calibration points were A(0, 0), B(0, 20), C(40, 20), and D(40, 0). Combined with their image positions, Equation (12) can be solved; thus, we have:

m′ = [9.47 2.234 260.58 0.09 − 4.92 256.61 − 1.142 0.005]

The precision of m′ can be verified by measuring some standard workpieces. In our experiments, two different sizes of standard blocks were used. Their size was 20 mm × 40 mm and 30 mm × 50 mm. They were translated and rotated into different poses. Using the calibrated m′, we calculated their size and compared it to their true size. The measured errors were calculated as shown in

Table 1. Comparison of measured data with real values.

	Real Value (mm)	Measured Value (mm)	Absolute Error (mm)	Relative Error (%)
A1B1	20.0	20.1	0.1	0.50
B1C1	40.0	39.9	0.1	0.25
C1D1	20.0	19.7	0.3	1.50
D1A1	40.0	39.8	0.2	0.50
A2B2	40.0	40.1	0.1	0.25
B2C2	20.0	19.8	0.2	1.00
C2D2	40.0	39.9	0.1	0.25
D2A2	20.0	19.8	0.2	1.00
A3B3	30.0	30.1	0.1	0.33
B3C3	50.0	50.0	0.0	0.00
C3D3	30.0	29.9	0.1	0.33
D3A3	50.0	50.0	0.0	0.00

. It can be seen from the measurement results that the absolute error was lower than 0.3 mm, and the relative error was around 0.5%. This indicates the correctness and accuracy of our calibration model of parameter M.

5.3. Calibration of System Parameters ^WT_S(0)

According to the method proposed in Section 4.2, two calibration patterns were printed on the workpiece along the same line denoted as Lc. The calibration procedure was shown in Figure 7. First, we took a picture of the first pattern and obtained the camera parameters matrix m′ with the method introduced above. Then, we moved the robot along the y-axis of {W}. When the second pattern appeared in the camera, the distance of the robot’s movement was recorded, and the result was Δy = 400 mm. There was an offset angle between line Lc and the y-axis. This offset angle was exactly what we needed to calibrate. Calibration points extracted from the second pattern and Equation (13) were used to calculate the transform matrix between {W} and {S(0)}. The result is:

$${}_{ }{}^{\mathrm{W}}\mathbf{T}{ }_{\mathrm{S}\left(0\right)}=\left[\begin{array}{c}\begin{array}{c}0.0091\\ -1\end{array}\\ 0\\ 0\end{array}\begin{array}{c}\begin{array}{c}1\\ 0.0091\end{array}\\ 0\\ 0\end{array}\begin{array}{c}\begin{array}{c}0\\ 0\end{array}\\ 1\\ 0\end{array}\begin{array}{c}\begin{array}{c}-36.96\\ 20.76\end{array}\\ 0\\ 1\end{array}\right]$$

Positions were measured by the proposed method several times. The measurement results were compared with those measured by the robot’s calibration needle. For all of the points to be measured, we denoted them as A1~A3. The robot’s calibration needle was set on the measured points, as shown in Figure 8. The robot system would obtain their coordinates in {W}. Then, these points were measured with the proposed infrared visual system. The measured results are all shown in

Table 2. Position measurement results of two methods.

Index		Calibration Needle Data (mm)			Infrared Vision System Data (mm)			Difference (mm)
		XW	YW	ZW	XW	YW	ZW
1	A1	511.8	403.4	256.9	511.7	404.3	256.9	0.14
	A2	532.2	349.9	248.4	532.0	349.9	248.5	0.22
	A3	419.1	298.5	327.3	419.3	298.4	327.2	0.24
2	A1	511.9	403.5	256.8	511.7	404.3	256.9	0.30
	A2	532.3	349.7	248.3	532.1	349.9	248.4	0.30
	A3	419.2	298.5	327.3	419.3	298.4	327.2	0.17
3	A1	511.8	403.5	256.9	511.7	404.3	256.9	0.22
	A2	532.3	349.8	248.3	532.1	349.9	248.4	0.24
	A3	419.2	298.6	327.4	419.3	298.4	327.2	0.30

.

The results show that the measurement difference between the results measured by the calibration needle and the thermal-infrared camera was small. This indicates the correctness of the calibration model of the thermal-infrared measurement system. Since the robot’s calibration needle measures the 3D position by relying on a human to read the data, the measurement results have a certain error. This experiment can only determine whether the infrared measurement system measures correctly, but cannot evaluate the measurement accuracy. Therefore, we designed another experiment. We measured the dimensions of a standard workpiece simultaneously using an infrared measurement system calibrated with the method proposed in this paper, an infrared measurement system calibrated with the calibration method proposed in [25], and a calibration needle. Figure 9 shows the calibration images based on the method proposed in [25].

Table 3. Comparison of measurement results.

Real Value (mm)	Measurement Value (mm)			Error (mm)
	Needle	Traditional Method	Proposed Method	Needle	Traditional Method	Proposed Method
25	25.9	25.6	25.0	0.9	0.6	0.0
30	30.3	30.5	30.1	0.3	0.5	0.1
35	35.2	35.4	35.9	0.2	0.4	0.1
40	40.6	40.6	40.2	0.6	0.6	0.2
45	45.6	45.9	45.0	0.6	0.1	0.0
50	50.4	50.3	49.9	0.4	0.3	0.1

shows the measurement results and measurement errors of these three methods.

The root mean square errors (RMSE) were calculated as 0.225 mm, 0.185 mm, and 0.044 mm. Table 3 shows the measurement results of these three methods. The accuracy of the proposed method in this paper was improved compared to the other two methods. Paper [26] uses the infrared visual measurement system calibrated by this method to measure the defects in the online inspection of automated fiber placement. Figure 10 shows the human-computer interaction interface. The conditions of the composite placement experiment are set as follows: environment temperature 19 °C, environment humidity 40.5%, lay-up surface temperature 26 °C, heating lamp power 73%, compaction pressure 0.3 Mpa, and laying speed 150 mm/s. Defects such as gaps, foreign bodies, and air pockets are detected. By extracting the edges and calculating the pixel coordinate of defects, the position of defects can be achieved in the workpiece coordinate frame according to calibration results. The left side of the interface shows the original image captured by the thermal-infrared camera and the defects detected in real-time. The right side of the interface shows the location and dimensional information of the defects marked onto the workpiece model, forming a 3D defect model for industrial HMI. Through the human-computer interaction, the defect model can be manipulated with the mouse and click to select the defect of interest. The selected defect displays its position information on the interface; here, the information was (319.3, 298.4, 327.2). The results illustrate that the proposed method achieved good performance in practical applications.

6. Conclusions

This paper presented a thermal-infrared visual measurement system for industrial HMI together with its modeling and calibration method. Experiments show a good performance of the proposed system. Our work fills the gap in the application of infrared cameras in industrial measurement. The thermal infrared vision sensor designed in this paper can be used in a wider range of industrial HMI applications such as digital twins, AR/VR [27,28], human modeling scans, and modeling in Industry 4.0 [29].