Calibrate Phase

## Alignment Phase

**Figure 6.** This flowchart indicates the steps to simultaneously calibrate the camera and calculate the center of rotation of the machine.

A 12-point calibration method is introduced to auto-calibrate the camera system and calculate the rotation center of the carrier table simultaneously, from which nine points are

used to calculate a homogeneous transformation matrix based on Zhang's method [27], and three points for finding the center of rotation. Such twelve points are derived from different images that are acquired in nine positions and three different angles. Figure 7 shows a sample of twelve images, in which the feature points of the mark are searched based on the described pattern search technique. The center of the mark region is chosen as the feature point. After recording the coordinates of feature points, the corresponding positions of these feature points in the world coordinate were then calculated through the homogeneous transformation matrix.

**Figure 7.** (**a**) Uses the 12-point misalignment correction method to determine the center of rotation and to calculate the offset of translation. These twelve sample images are taken by a pre-alignment camera installed above the carrier table; (**b**) calculates the physical rotation center on the occasion of a large radius and a small angular interval.

The homogeneous transformation matrix stands for a correspondence between different coordinate systems which can be shown in Equation (1). The image point and the transformed point in the real world are denoted as (*xi*, *yi*),(*Xi*,*Yi*) in homogeneous coordinates.

$$
\begin{pmatrix} X\_i \\ Y\_i \\ 1 \end{pmatrix} = HMat2d \cdot \begin{pmatrix} x\_i \\ y\_i \\ 1 \end{pmatrix} \tag{1}
$$

A simplified transformation matrix that only consists of translation and rotation is represented as the Equation (2). *Tx* and *Ty* are the translation in the X- and Y- axis directions and *ra*–*rd* denote the rotation parameters.

$$HMat2d = \begin{pmatrix} ra & rb & T\_x \\ rc & rd & T\_y \\ 0 & 0 & 1 \end{pmatrix} \tag{2}$$

For a test subject, nine translation images and three rotation images are captured by controlling the motion of the carrier table. In each position, the coordinate of the feature point in the image space is measured by searching reference marks in view of an image while the corresponding location in the real world is recorded. The rotation angle of the carrier table is also included during this procedure. Nine of these pairs of points are used

to approximate the transformation matrix which is expected to minimize the distance error between those pairs of points following the Equation (3).

$$\min \left( \sum\_{i} \left\| \begin{pmatrix} X\_i \\ Y\_i \\ 1 \end{pmatrix} - HMat2d \cdot \begin{pmatrix} x\_i \\ y\_i \\ 1 \end{pmatrix} \right\|^2 \right) \tag{3}$$

Once the calibration process is finished, one point in image space can be transformed into that in the real world, thus enabling the calculation of the physical rotation center.

The rest three points are then offered to find the center of rotation. As a rotating angle can have an effect on the position of points in both X- and Y-axis directions, it is necessary to correct the angle deviation before translating the tested object to align with the reference one. Due to a different design of the mechanic structure, the rotation center of the carrier table can vary a lot, which leads to different impacts on the deviation of points. The main idea of calculating the rotation center of the carrier table, known as fitting a circle, is to obtain multiple points by rotating the carrier table and calculating the point of the circle which is supposed to cover those points in the real world. This method is based on an assumption that a pure rotational motion should form a track of a circle for a point on the plane. Given that we can acquire the physical rotation angle of the carrier table, the center of the circle can be calculated by each two of the three points as well as their included angle.

As demonstrated in Figure 7b, *<sup>A</sup>*(*<sup>X</sup>*1,*Y*1) and *<sup>B</sup>*(*<sup>X</sup>*2,*Y*2) are two points transformed from image space to the real-world coordinates. The midpoint *<sup>D</sup>*(*<sup>X</sup>*0,*Y*0) of the line AB through (*<sup>X</sup>*1,*Y*1) and (*<sup>X</sup>*2,*Y*2) is ( *X*1+*X*2 2 , *Y*1+*Y*2 2 ), from which we can form the parametric equation of perpendicular bisector through *C* and *D*.

$$\begin{aligned} X &= X\_0 + t\Delta X\\ Y &= Y\_0 + t\Delta Y \end{aligned} \tag{4}$$

Because line CD is perpendicular to line AB, the slope of CD is reciprocal to that of AB, which can be expressed as

$$\frac{\Delta Y}{\Delta X} = -\frac{1}{\frac{Y\_1 - Y\_2}{X\_1 - X\_2}} = -\frac{X\_1 - X\_2}{Y\_1 - Y\_2} \tag{5}$$

Therefore, Equation (4) can be transformed into

$$\begin{aligned} X &= \frac{X\_1 + X\_2}{2} + t(Y\_1 - Y\_2) \\ Y &= \frac{Y\_1 + Y\_2}{2} + t(X\_1 - X\_2) \end{aligned} \tag{6}$$

Additionally, we can obtain the following equation from the lengths of AD and CD and their included ∠*ACD*

$$\begin{split} AD &= \left( X\_1 - \frac{X\_1 + X\_2}{2} \right)^2 + \left( Y\_1 - \frac{Y\_1 + Y\_2}{2} \right)^2 \\ &= \frac{(X\_1 - X\_2)^2 + (Y\_1 - Y\_2)^2}{4} \\ CD &= \left( t(Y\_1 - Y\_2)^2 \right) + \left( -t(X\_1 - X\_2)^2 \right) \\ &= t^2 \left[ (X\_1 - X\_2)^2 + (Y\_1 - Y\_2)^2 \right] \\ \angle ACD &= \frac{\theta}{2} = \arctan \frac{AD}{CD} = \arctan \frac{1}{2t} \end{split} \tag{7}$$

Therefore, we can derive the relationship between *t* and the included angle *θ* as

$$t = \frac{\cos\frac{\theta}{2}}{2\sin\frac{\theta}{2}} = \frac{\sin\theta}{2(1+\cos\theta)}\tag{8}$$

Therefore, we have

$$\begin{aligned} \mathbf{C}\_{\mathbf{X}} &= \frac{X\_1 + X\_2}{2} + \frac{\sin \theta (Y\_1 - Y\_2)}{2(1 - \cos \theta)} \\ \mathbf{C}\_{\mathbf{Y}} &= \frac{Y\_1 + Y\_2}{2} + \frac{-\sin \theta (X\_1 - X\_2)}{2(1 - \cos \theta)} \\ \mathbf{R} &= \sqrt{(X\_1 - \mathbf{C}\_{\mathbf{X}})^2 + (Y\_1 - \mathbf{C}\_{\mathbf{Y}})^2} \end{aligned} \tag{9}$$

where (*Cx*, *Cy*) represents the center of the fitting circle and *R* specifies the radius. Three centers are given through the formula above and averaged to improve the accuracy of the center of the fitting circle.

According to the flowchart in Figure 6, in the alignment phase, we first measure the angle of the reference image whereby we can obtain the deviation of the angle between the test image and the reference one. The camera can take images on the left and right side of the LCD module, respectively. Through searching for the feature marks printed on both sides, two feature points can be drawn to form a line that can act as a baseline. To compare the different angles of two lines, points have to be transformed to a uniform world coordinate. The transformed positions are then normalized by rotation center (*Cx*, *Cy*) to take the center as the origin point. By calculating *θt* and *θc* for reference angle and current angle, the difference of angle can be represented as Δ*θ* = *θt* − *θ<sup>c</sup>*. Secondly, the offset of translation is determined. Based on the design that the center of rotation is the origin of the coordinate, the radius and angle of a point in polar coordinate can be expressed as

$$r = \sqrt{X\_{\varepsilon}^2 + Y\_{\varepsilon'}^2} \quad \theta = \arctan \frac{Y\_{\varepsilon}}{X\_{\varepsilon}} \tag{10}$$

Then, the rotated points (*Xr*,*Yr*) can be obtained by

$$\begin{aligned} X\_r &= r \cos(\theta + \Delta\theta) \\ Y\_I &= r \sin(\theta + \Delta\theta) \end{aligned} \tag{11}$$

Either point of left or right side feature point can be used to calculate the offset of translation. We use (*Xr*,*Yr*) and (*Xt*,*Yt*) to denote rotated point by Δ*θ* and the reference point, respectively. Finally, the translation offset can be obtained by

$$
\Delta X = X\_r - X\_t
$$

$$
\Delta Y = Y\_r - Y\_t \tag{12}
$$

After determining the rotation center and the translation offset, the tested module can be aligned to the reference position as Figure 8 shows.

**Figure 8.** Expected result after the misalignment correction process.

## *2.3. Bump Positioning*

After aligning the test object with the reference one, the positional deviation can be reduced thus contributing to accurately positioning bump areas containing the particles to be detected. The aligned LCD module is scanned by a line scan camera installed under the carrier table. Figure 9 shows a scanned image in the COG process. The cross-shaped pattern on the top right of the image serves as the positioning mark, while the red and blue regions in the image stand for ITO bumps. Two major types of bumps are annotated in Figure 9, which are block-like regions for type I and bar-like regions for type II, respectively.

As a whole imaging region consists of numerous bumps and other irrelevant pixels that make up a large portion of the image, it is difficult and time-consuming to extract valid information. Bump positioning intends to segmen<sup>t</sup> essential regions from the whole image and allows the detection algorithm to operate on every smaller region, which can reduce the size of detection areas and improve efficiency. Considering that the bump positions of a type of component are relatively fixed, we can construct a standard file with position information of all bump areas in it. Such information can be stored in the form of values relative to the position of the mark, which can be quickly determined by the shape-based match method. The feature mark is first searched in the test image, then the bump areas can be annotated with information in the standard file. The standard file can also be finetuned to adapt to the bump distribution in different LCD modules.

**Figure 9.** The local image of LCD module containing bump and mark areas.

## *2.4. Particle Detection*

The number of particles captured is calculated in every segmented area of the bump. ACF conductive particles represent microspheres and deform with different levels of press strength. During the bonding process, these particles are trapped between the bump areas, causing those areas to show uneven pits. Figure 10a extracts a sample of welldeformed conductive particles, while (b) shows a local image of particles taken under the inappropriate pressure condition.

**Figure 10.** (**a**) Well deformed conductive particles; (**b**) over-deformed particles.

We can observe that the properly pressed particles appear as distinct light and dark parts in the image, while over-deformed particles have a weak contrast compared with neighbor areas. Based on such an observation, the particle areas are intended to be recognized by locating the extreme point of gray value of every particle. However, many false extreme points occur possibly caused by sharp changes in the gray-scale values of the edge of the bump or the intersection area between particles.

To reduce the effect of edge and intersection area on the detection accuracy, we first smooth the original image Figure 11a by the gaussian kernel, as shown in Figure 11b. We denote the original image as *<sup>I</sup>*(*<sup>x</sup>*, *y*) and the smoothed image as *Ig*(*<sup>x</sup>*, *y*). During the smoothing process, the original image is filtered by the 2D gaussian kernel *<sup>G</sup>*(*<sup>x</sup>*, *y*) which can be formulated as the Equation (13)

$$G(x,y) = \frac{1}{2\pi\sigma^2} \exp^{-\frac{x^2+y^2}{2\sigma^2}}\tag{13}$$

where *x* is the distance from the origin in the horizontal axis, *y* is the distance from the origin in the vertical axis. The smoothed image can be denoted as

$$I\_{\mathcal{K}}(\mathbf{x}, \mathbf{y}) = I(\mathbf{x}, \mathbf{y}) \* G(\mathbf{x}, \mathbf{y}). \tag{14}$$

Subsequently, we perform a gray value dilation on the smoothed image with a structuring element *b* in order to highlight the region of each particle. An example of output image after the gray value dilation is shown in Figure 11c. The gray-scale value of the dilated image is defined with the maximum value of the overlapped region between the structuring element and the target image. Such processing can be demonstrated as

$$I(I \oplus b)(\mathbf{x}, \mathbf{y}) = \max\_{(s, t) \in \mathcal{b}} I(\mathbf{x} - s, \mathbf{y} - t) \tag{15}$$

where *s* and *t* denote the distance from the origin in both directions. According to the size of structuring element *b*, the *s* and *t* should be in [−*size*,*size*]. We can regard the structuring element as another filtering kernel because it serves to further reduce the variation of local gray values. As a result, the gray values of a particle region are equivalent to the maximum value of this region, which is usually the gray value of the center of the particle. Following the process in Figure 11d, the dilated image is compared with the smoothed image to create a differential image where the extreme points of every particle are expected to be emphasized. To achieve this, we normalize the mean gray value of the smoothed image to 128 following the Equation (16)

$$I\_{\mathcal{S}}(\mathbf{x}, \mathbf{y}) = I\_{\mathcal{S}}(\mathbf{x}, \mathbf{y}) \times \frac{128}{Mean} \tag{16}$$

then the differential image *Id*(*<sup>x</sup>*, *y*) is created by the Equation (17).

$$I\_d(\mathbf{x}, \mathbf{y}) = I\_{\S}(\mathbf{x}, \mathbf{y}) - [I \oplus b](\mathbf{x}, \mathbf{y}) + 128 \tag{17}$$

In the differential image, the central parts of particles are brighter than neighbor pixels indicating the existence of particles.

**Figure 11.** This flowchart shows the procedure of particle detection. The original image (**a**) is smoothed to image (**b**), (**b**) is dilated by a structuring element to produce (**c**), (**d**) highlights the particle centers, (**e**) remove noisy objects by features of the particle, and the final result is presented in (**f**). The processing and visualization of images are supported by OpenCV.

To refine the detection result, we also adopt the blob analysis to exclude noisy objects. During the blob analysis, detected objects are checked case by case, such as a sample shown in Figure 11e. Some gray features of particles can be used to enhance the reliability of the detection result. For example, the gray strength of the particle is assumed to exceed a certain threshold while the result with a lower strength should be removed. In addition, since the standard deviation of the brightness in the particle region should be within a certain range, we can extract the particle objects based on this criterion. After excluding noisy objects, the detection results in Figure 11f can be visualized by the detected centers of particles and a specified particle radius.

The number and density of particles in a given region can be determined from the results depicted in Figure 11f. Normal particles tend to be numerous and have a relatively uniform and dense distribution, while over-deformed particles can be identified by a small number of particles and a sparse distribution in the image analysis. In practical production, the threshold for distinguishing between normal and over-deformed particles should be determined based on the specific criteria of the plant.

#### **3. Results and Discussion**

## *3.1. Experiment Setup*

As depicted in Figure 4, the majority of the system consists of mechanical components and sensor sets. The mechanical components primarily consist of robotic grippers responsible for loading and unloading LCD modules, a carrier platform equipped with translation and rotation motors, and a large horizontal track providing X-axis movement for pre-alignment cameras. The sensor sets, including the pre-alignment camera, line scan camera, and laser displacement sensor, are essential tools in the detection system. A detailed configuration can be found in Table 1. In terms of the parameters for the detection algorithm, a Gaussian kernel with a size of 3 × 3 and a *σ* of 0.6 was chosen, and the structuring element employed a circle with a radius of 8 pixels. In the blob analysis, the strength threshold of particles is set to 8, and the range of brightness within a particle region is 20–220.

## *3.2. Calibration Result*

In our experiment, we first calibrated the homogeneous transformation matrix between the image space and the world coordinate and calculated the center of rotation of the carrier table. In Figure 12, we show the relationship between the captured images in terms of physical position, where subfigures 1–9 are captured by translating the carrier by 1 mm, while subfigures 10–12 are captured by rotating the carrier by 0.25◦.

**Figure 12.** The relationship between the captured images in terms of physical position.

Table 2 shows the pixel location and the physical location of twelve images. The calibration result is shown in the last column. The location and rotation in the world coordinate were determined by the relative distance to the origin of the carrier table. We derived the homogeneous transformation matrix *HMat*2*d* through images 1–9 while calculating the center of rotation through images 10–12. As the center of rotation of the carrier platform was far from the location of captured images, three centers were calculated by two images and the included angle following the Equation (9) and then averaged to the final center.


**Table 2.** The pixel location and the physical location of twelve images and the calibration result.

## *3.3. Alignment Result*

We selected some test samples to check the precision of our proposed method. Samples were acquired by controlling the machine to move different offsets in *Y*-axis or around the rotating axis while related information is recorded. This motion was conducted to simulate the occasion when the robotic gripper fails to make sure that the LCD module keeps the same position as the standard module.

In our experiments, the line scan camera was installed at the *X*-axis. We moved the line scan camera and took images of the left panel and right panel containing the feature marks. The mark locations in both panels were recorded and compared with those in the reference panel. Table 3 presents the positions of the left and right markers in various settings, with the positions being unified to a coordinate system with the center of rotation of the carrier table serving as the origin. For example, Y0.3 moves the carrier platform in the Y-axis direction by 0.3 mm, and Y0.6-R0.3 moves 0.6 mm alongside the Y-axis and rotates 0.3◦ around the center of rotation of the carrier table.


**Table 3.** The positions of the left and the right mark (unit:mm).

Table 4 demonstrates the alignment precision as below. The rightmost column shows the difference between the alignment result and previously recorded information, which represents Δ*X*, Δ*Y*, and Δ*θ*, respectively. Such results show that our method achieves high precision in restoring the offset of translation and rotation of tested modules.


**Table 4.** The result of misalignment correction in different settings (unit: <sup>Δ</sup>*X*(*mm*), <sup>Δ</sup>*<sup>Y</sup>*(*mm*), Δ*θ*(◦)).


**Table 4.** *Cont.*

Although the setting of the *X*-axis was not verified separately, the offset of the *X*-axis was also calculated simultaneously during the verification of *Y* and *θ*. Considering the similarity of the *X*-axis and *Y*-axis, the experimental results can also illustrate the alignment accuracy of the *X*-axis.

Furthermore, we collected all data in three groups listed in Table 5 and analyzed how different offsets can affect the accuracy of our proposed method. We can observe from this table that our alignment method can keep a stable result in pure translation or rotation.

**Table 5.** The impact of different offset configurations on the accuracy of the alignment algorithm (unit: *<sup>X</sup>*(*mm*),*<sup>Y</sup>*(*mm*), *θ*(◦)).


1 Different colors are used to denote the relevance of the results. For instance, the red text "Y0.3" signifies an offset of 0.3 mm along the Y-axis, with a corresponding difference in the result of "0.006".

## *3.4. Detection Results*

We evaluated the effectiveness of the proposed method in bump areas acquired from both COG and FOG samples. Figure 13 shows the results of particle detection in COG and FOG, respectively. Bump areas were segmented according to the saved file about the relative position of bumps to the feature mark in different types of samples such as COG and FOG. Two major types of bumps were categorized the bar-shape and the block-shape. The bar-shape bump is long and narrow, while the block-shape bump is short and wide that contains relatively dense particles. The target area of the block-shaped bump is smaller in size and the visibility of particles is weaker, thus making it more difficult to accurately inspect the number of particles.

**Figure 13.** The first row is a part of the bump area of the COG sample (**left**) and the corresponding result (**right**), while the second row displays the input and output of the FOG test in the same order.

Figure 14 looks closer at bar-like bump areas. Bumps with an insufficient number of particles would be sent to NG output based on the manufacturer's requirement.

**Figure 14.** A closer view of the result of particle counts in bar-shape bump areas. The first row is the raw image while the second row is the detection result.

We also compared our method with Lin's [12] and Chen's [26] method and verified the effectiveness of our method as shown in Figure 15. Lin's method was sensitive to slight changes in gray value, thus causing the algorithm to detect more particles in regions where particles overlap densely. Such a method treated the NG module as the OK module and may produce many false positive samples, which is inappropriate in the practical industry. Contrarily, Chen's method recognized the number of overlapped particles more accurately. However, the performance of Chen's method depended on the initialization of the cluster center and thus may lose a number of particles on some occasions.

The detection result in 433 particle samples of bar-shape bumps and 629 samples of block-shape bumps are shown in Table 6. Lin's method demonstrates a slightly improved recall rate for bar bumps due to its tendency to generate more particle candidates than are actually present, thereby covering a larger portion of the region on samples with smaller dimensions. Conversely, this method exhibits a significant decline in recall performance on samples with larger dimensions, such as the block-shape bumps. Combined with the low precision, Lin's method shows a high over-checking rate which will allow a number of false products to be passed and affect negatively quality control. Chen's precision rate is good but the number of detected particles is lower than that in other methods when the gray-scale value of the image is weak. In contrast, our method balances the precision and recall rate of the detection result and performs better than the aforementioned methods.

**Figure 15.** Examples of different detection methods on bar-shape bumps and block-shape bumps.

**Table 6.** The comparison of detection results of different methods.


## **4. Conclusions**

In this paper, we establish an automatic inspection system to serve multiple bonding processes such as COG and FOG in LCD module manufacturing. A 12-point calibration method is developed to adaptively control the position and rotation of the carrier table and to alleviate the high requirement of assembly accuracy. This method can complete the alignment by taking images once, which calculates a homogeneous transformation matrix with nine pairs of points and determines the center of rotation with extra three pairs of points. In our experiments, the alignment error can be less than 0.05 mm.

This study also proposes an automatic particle detection method based on gray morphology which achieves a fast and robust inspection of the number of conductive particles trapped in bump areas of the anisotropic conductive film. Based on the observation that the central part of conductive particles is brighter than the neighbor region in grayscale under our DIC imaging model, we apply the gray dilation method to the whole image and subtract dilated image from the original input, from which the center regions of particles are stressed out. Our experiments have examined the effectiveness of this method, in which the precision rate is 93%, while the recall rate is 92.4%. As our proposed detection method depends on finding the difference between the central and peripheral regions of particles, the overlap of multiple particles may intervene in this difference, thus creating an obstacle to accurate segmentation. In the future, further research will be conducted to improve the representation of particles and segmen<sup>t</sup> those overlapped particles more accurately.

**Author Contributions:** Conceptualization, T.L.; methodology, F.Z.; software, T.L. and F.Z.; validation, T.L.; formal analysis, F.Z.; resources, Z.Z.; writing—original draft preparation, T.L.; writing—review and editing, F.Z., H.Y. and H.L.; visualization, T.L.; supervision, F.Z. and Z.Z.; funding acquisition, Z.Z. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work is supported by the National Key Research and Development Program of China (2022YFB3303800), the National Natural Science Foundation of China (U21A20482, U1909218), the Science Technology Project funding from Binzhou Weiqiao Guoke Advanced Technology Research Institute(E2D21709, E2D21710, E2D21711).

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

## **Abbreviations**

The following abbreviations are used in this manuscript:

