A Vision-Based Bolt Looseness Detection Method for a Multi-Bolt Connection

Abstract

Many vision-based bolt looseness detection methods that directly observe the bolts have been developed. However, these methods have many limitations in terms of the conditions and processes of their implementation. To address these problems, this paper proposed a fully automated vision-based bolt looseness detection method for a rigid multi-bolt connection. The proposed method combines digital shearing speckle pattern interferometry (DSSPI) and recurrent neural network (RNN) and involves capturing speckle fringe patterns under various looseness cases using the DSSPI system and classifying these patterns with an RNN model to detect the loose bolts. The proposed method can detect all the bolts within the measured surface at one time, which is efficient. On the other hand, it eliminates the need for prior information such as the initial angle and position of each bolt. It can even detect unseen bolts in multi-bolt connections, making it applicable for connections in complex structures in which occlusion often occurs. Additionally, the method eliminates the complex process of distortion rectification. These features make the method achieve a single-judgment time (four bolts at one detection) of only 4.70 millisecond with a detection accuracy over 99%, which has potential for the real-time detection of loose bolts in multi-bolt connections.

1. Introduction

Bolt connections play a crucial role in uniting various components within structures and are indispensable in diverse industries like civil engineering, aerospace engineering, and mechanical engineering [1,2]. Ensuring the quality and stability of these bolt connections is paramount to maintaining the overall security of the structure. However, under adverse environmental conditions, intense shocks, or other external factors, removable bolt connections can become vulnerable points in a structure. This vulnerability poses a significant threat to the safety and reliability of the entire structure. Startlingly, statistical data reveal that over 20% of the accidents in mechanical systems, resulting in substantial losses, are attributed to bolt looseness [3].

In contemporary settings, torque wrenches are widely employed for detecting loosened bolts. Nevertheless, this method is prone to measurement errors of up to 50%, and it involves labor-intensive and hazardous processes [4,5]. Moreover, existing studies on bolt loosening detection predominantly focus on single-bolt structures, with limited research dedicated to multi-bolt connections. Many of these methods either lack accuracy or fail to recognize all potential cases of loosening [6,7]. Consequently, there is an urgent need to develop an automated method with a high detection accuracy to ensure the safety and reliability of multi-bolt connections.

For this subject, researchers have proposed various sensor-based methods, including piezoelectric-based methods, impedance-based methods, and acoustic-based methods [1,2]. The frequency selection is very important in ultrasound methods, which decides the accuracy and robustness of the method. In 2018, Fierro and Meo [8] detected the looseness of a three-bolt connection with a piezoelectric transducer array. They used a cross-correlation method to find higher harmonics or special frequency pairs that could identify loose bolts. In 2019, they [6] evaluated several nonlinear-ultrasound methods to identify the looseness of a four-bolt connection. They tried some nonlinear algorithms to recognize the individual frequency corresponding to each bolt. When one of the bolts is loose, it will produce more nonlinearities in the corresponding frequencies, so they can identify the loose one. But the method can only recognize several loosening cases in which just one bolt is loosened. Zhang et al. [9] evaluated a nonlinear vibro-acoustic modulation (VM) approach and a linear acoustic method to detect loose bolts. They found that the VM method had higher sensitivity and accuracy, and their method could even deal with multi-type connections. Eraliev et al. [10] proposed a bolt looseness detection method based on machine learning (ML) and vibration frequency recognition. They compared various ML algorithms based on accuracy and training time. The random forest (RF) algorithm was finally chosen for further studies.

The piezoelectric active sensing method represents another avenue of research in contacted sensor-based approaches. However, it faces challenges as energy-based indicators are susceptible to saturation and fluctuation. Jiang et al. [11] extracted some nonlinear stress wave features and chose the recursive entropy of the signal as the indicator to detect loose bolts. Lintao Wang et al. [12] found that the bolt looseness caused a shift in the impedance of the multi-bolt connection, so they used a fusing multi-frequency-based electro-mechanical impedance (EMI) method to identify loose bolts. But this method cannot identify all the 16 cases of the 4-bolt connection.

Feature extraction is a problem in wave signal processing-based methods. In 2020, Furui Wang et al. [13] trained a support vector machine (SVM) network based on a genetic algorithm. That network could recognize all the looseness cases of a four-bolt connection. They developed another novel indicator from the multi-variate multi-scale fuzzy entropy (MMFE) as the input of the classifier. In 2021, they [14] used the same type of stress wave signal as the method in [11], but they developed two novel indicators: the multi-scale range entropy and the multi-scale bubble entropy of the signal as feature, to train an ensemble learning model for connection looseness detection. In the previous two works, the features still had to be selected by a human. In 2021, they [7] utilized newly developed convolutional long short-term memory networks for multi-bolt looseness detection. The convolution blocks in the networks can extract the proper features automatically, and the SoftMax layer can output the detection result. Although this method skipped the feature extraction by humans, it could not detect all the looseness cases compared to the method in Ref. [13].

Contacted sensor-based methods have undergone significant development over the years, yielding commendable results. Nonetheless, these methods demand direct physical contact with the measured objects, posing challenges in terms of feasibility and requiring the prior installation of specific sensors. This not only proves inconvenient but also entails additional costs. Furthermore, these approaches are vulnerable to environmental factors like temperature changes and humidity, potentially leading to inaccurate judgments.

Vision-based methods are another major area of multi-bolt looseness detection. The advantage is their noncontact nature, and no pre-installed sensors are required. One of the basic vision-based approaches is to pre-mark the bolts and subsequently detect the angle of their rotation using the images captured by a camera. For example, Sun et al. [15] marked the nut and bolt beforehand and used the fifth version of You Only Look Once (YOLOv5) to identify the position of the marks. The relative position change of the marks can be used to calculate the rotation angle of the nut and finally identify the loose one. But it will fail when the marker point is obscured. To solve this problem, Pan et al. [16] utilized a brand new barcode mark (polarization-adjusted convolutional code, PAC-code) that covered the whole bottom surface of the bolt, which can avoid the failure caused by partial occlusion.

In addition to the pre-marking method, researchers have explored many other vision-based methods to estimate their angles and detect the loose bolts. The Hough transform is a classic feature extraction algorithm, which can recognize the bolts’ edges to estimate the bolts’ angles [17,18]. In 2015, Park et al. [18] combined the circular Hough transform with the Canny edge detector to recognize and segment each nut from the images. The method detected the loose bolt by calculating the rotation angles of the outlines of the bolt between the original image and the current image. But their experiment only considered the case in which the camera was facing the bolts. In 2020, Wang et al. [19] built on Park’s work, they introduced the perspective transformation to solve the problems caused by a changed perspective angle. In 2019, Huynh et al. [17] developed a regional convolutional neural network (RCNN)-based algorithm to recognize and segment bolts in captured images. They also used the Hough transform to calculate the rotation angles of bolts just like the method in [19]. In 2021, they [20] designed a new faster RCNN algorithm combined with adaptive bolt-angle calculation for the multi-bolt connection on the Dragon Bridge in Vietnam. Compared to the quasi-automated method in [19], this system consisted of a complete set of automated inspection processes that greatly improved inspection efficiency. Experiments have proved that their methods can accurately detect loose bolts when the shooting angle is greater than 50°. However, there are instances in which the Canny edge detector may struggle to detect the bolts’ edges, especially when the contrast is low around the edges. The distortion rectification algorithm of the method is limited to rectangular-based bolt arrays. Furthermore, a crucial prerequisite for implementing these methods is knowledge of the initial angle of the bolts before loosening, significantly constraining the scenarios in which they can be effectively used.

In addition to estimating the angle of the bolt, researchers have also measured the vertical length ($h$) between the top of the bolt and the flat surface to detect loose bolts. Cha et al. [21] achieved bolt looseness detection using only a smartphone. They combined several adaptive image processing algorithms to obtain various looseness-sensitive features. These features were used as the dataset of a linear SVM trained to recognize the loose bolts. But this method can only work well at a shooting distance between 78 mm and 122 mm and a perspective angle between 31° and 51°; that is quite a small detection range. Similarly, Ramana et al. [22] located and recognized bolts using the Viola–Jones algorithm. They selected the exposed length of the bolt and the size of the bolt head as looseness features to train an SVM model for looseness detection. And Zhang et al. [23] built a classifier with a faster RCNN, similar to the classifier in [20]. They constructed an image dataset obtained from different perspective angles and distances to improve the robustness and utility of the model. But the detection accuracy of the method turned out to be affected by the perspective angle, which is a lack in the principle of machine vision. Compared to several previous methods that measured the $h$, Pan et al. [24] used readily available 2D images to reconstruct the three-dimensional (3D) map of multi-bolt connections from high-resolution images. They introduced a convolutional neural network (CNN) to recognize and segment the bolts in the 3D map. The vertical length between the bolt head and the reference plane was calculated to judge whether the bolt was loose. However, all these vision-based methods share common drawbacks: the need for prior information such as the initial angle and position of each bolt, requiring the direct visual observation of the bolts. The first issue is unavoidable in principle for those methods. Addressing the second issue could involve exploring the use of multiple cameras placed orthogonally within a plane though delving into this topic is a separate and complex matter, and it does not work well for some complex structures.

In our previous work [25], we proposed a method to detect the looseness value of a single fastener (an elastic structure, which contains a deformable clip, just like a spring) in the track system. For elastic structure such as the fasteners in the track system, the previous method detects the loose fastener by detecting its looseness value, and it can only detect a single fastener at one time. In this paper, an automated vision-based method using digital shearing speckle pattern interferometry (DSSPI) and recurrent neural network (RNN) is proposed to detect multiple loose bolts in the multi-bolt connections (rigid structures) at one time. The main idea of this approach is to establish the correspondence between different kinds of fringe patterns and different bolt loosening states and then classify the speckle fringe patterns of the multi-bolt connection under different states to detect the loose bolts. The approach involves three key steps: (1) capturing the shearing speckle patterns of the multi-bolt connection under different states using the DSSPI system, (2) employing the DSSPI algorithm to obtain the speckle fringe patterns, and (3) inputting these fringe patterns to train the proposed RNN classifier. In actual detection, the pre-trained RNN classifier directly identifies the input fringe patterns to detect the loose bolts in multi-bolt connections.

2. Methodology

2.1. DSSPI

DSSPI, also known as shearography, is used for high-precision, nondestructive measurement of surface deformation. It has gained widespread industrial acceptance for nondestructive testing [26,27]. Figure 1 illustrates the schematic diagram of the DSSPI system, which captures speckle fringe patterns.

The shearing device (the Rochon prism in Figure 1) produces two beams that interfere almost in a common path. As a result, the temporal coherence of the laser is greatly relaxed, and the experimental setup is relatively simple. In general, DSSPI systems are typically employed to quantify deformations, pressures, and other parameters. However, in this paper, we endeavor to establish a direct correlation between fringe patterns and the states of bolts with the aim of simplifying the detection of multi-bolt looseness into an image classification task. By bypassing the intricate calculation process, the method becomes more concise and accessible. In the following part, we will explain how the fringe pattern is formed.

The absolute value of the intensity of the shearing speckle fringe patterns $I_s$ can be expressed as follows [25]:

$$\left|I_S\right|=\left|4I_0\gamma \left[\mathit{sin}\left({\phi }_0+\frac{∆\phi }{2}\right)\mathit{sin}\left(\frac{∆\phi }{2}\right)\right]\right|$$

(1)

where $I_0$ is the sum of irradiances of both waves, and $∆\phi$ is the relative phase difference caused by deformation. The relationship between the gradient along x-direction of the out-of-plane deformation and the phase difference ∆φ can be expressed as follows:

$$\frac{\partial \omega }{\partial x}=\frac{\lambda ∆\phi }{4\pi ∆x}$$

(2)

where $\frac{\partial \omega }{\partial x}$ is the derivative of the out-of-plane displacement along x-direction, $\lambda$ is the wavelength of the laser, and $∆x$ is the shear amount. According to Equations (1) and (2), when $∆\phi =2n\pi$, $\frac{\partial \omega }{\partial x}=\frac{\lambda n}{2∆x}$, $\left|I_s\right|=0$, the black interference fringes appear ($n=0,\pm 1,\pm 2,\dots$, for the fringe order).

Figure 2 aids in visualizing the correlation among them: Figure 2a represents the surface deformation; Figure 2b illustrates $\frac{\partial \omega }{\partial x}$; and Figure 2c displays the corresponding speckle fringe pattern. We can note the similarity between Figure 2b and Figure 2c, which corresponds to the above analysis. As mentioned earlier, the proposed method seeks to establish a direct connection between the speckle fringe patterns and the bolt states, addressing looseness detection through image classification. However, it proves challenging for humans to succinctly summarize and generalize the differences in speckle fringe patterns across various states. Therefore, this study employs an RNN to generalize and classify these diverse speckle fringe patterns.

2.2. RNN

In our previous work of detecting the looseness value of a single track fastener [25], we employed a modified VGG-16 model to classify the speckle fringe patterns of one track fastener, given its great performance in image classification [28]. The modified VGG-16 model achieved a single-judgment time of 2.03 millisecond with an accuracy of 92.57% for a single fastener. The accuracy is not high enough, and worse, the training process takes lots of time. There are two reasons for this: Firstly, the model has too many parameters, a single iteration takes a long time. Secondly, the model is difficult to converge and requires too many iterations. These make the training process slow and difficult. On the other hand, compared to previous work, the captured fringe patterns in this work showed much stronger time series characteristics, as shown in Section 3.2. These motivated us to look for a new deep neural network (DNN) that is more suitable for this situation.

In 1982, John Hopfield proposed a single-layer feedback neural network [29], named the Hopfield network, to solve combinatorial optimization problems. This was the earliest incarnation of the RNN. In the decades since, RNNs have been continuously refined and have developed into a deep learning algorithm in the 21st century [30]. RNNs have demonstrated their remarkable performance in processing temporal sequences [7]. They are suitable for processing time sequence data; they recurse along the development direction of the data, and all the network’s nodes are connected with each other in a chain structure. The core structure of an RNN consists of a hidden state vector that is updated with each time step of the input sequence, thus conveying information at different time steps. Through this mechanism, RNNs are able to capture long and short temporal dependencies in sequence and show excellent performance in speech recognition, natural language processing, time series prediction [30], and image classification [31], etc. In this paper, we employed an RNN model to classify these fringe patterns, given its great ability to handle temporal sequences. Figure 3 is a simplified schematic of an RNN.

Given a data sequence $x=(x_1,x_2,\cdots ,x_t,\cdots ,x_{T-1},x_T)$, $x_t$ are the data at the $t$th time step (the $t$th speckle fringe pattern in this paper). The recurrent hidden states $h_t$ update iteratively by

$$h_t=\left\{\begin{array}{cc}\hfill 0,& ift=0\hfill \\ \hfill tanh(h_{t-1},x_t),& otherwise\hfill \end{array}\right.$$

(3)

and can be generally implemented in the RNN as

$$h_t=f\left(W_{in}{x}_t+W_S{h}_{t-1}+b\right)$$

(4)

where $W_{in}$ are the coefficient matrices for the input at the present step, and $W_S$ are the coefficient matrices for the activation of recurrent hidden units at the previous step. Generally, different layers share the same $W_S$ to reduce the number of parameters. The corresponding output can be implemented as

$$y_t=softmax\left(W_{hy}{h}_t+b_y\right)$$

(5)

$W_{hy}$ is the hidden layer to output layer weight matrix. Substituting Equation (5) into Equation (6), we obtain the following:

$$y_t=softmax\left[W_{hy}f\left(W_{in}{x}_t+W_S{h}_{t-1}+b\right)+b_y\right]$$

(6)

Equation (6) reveals that the output $y_t$ depends not only on $x_t$ but also on $h_{t-1}$ ($x_{t-1}$). This creates an iterative relationship between the hidden layer/output layer over time; in other words, giving the network a “memory”. That is why the RNN has remarkable performance with temporal sequences.

3. Experimental Section

3.1. Experiment Setup

The lab-test experiment was conducted to evidence the effectiveness and performance of the method. As shown in Figure 4, the laser emitted from the semiconductor lasers passed through the beam lifter and was expanded by the spatial filter; then, it irradiated the disk surface to form laser speckles. After being reflected on the surface of the disk, the laser speckle passed through polarizer 1, the Rochon prism, and polarizer 2; was captured by the CCD; and was finally transmitted to the computer.

The aluminum disk, with a diameter of 20 mm and a thickness of 2 mm, was assembled using four M6 bolts on the bracket. In Figure 5a, the front view of the disk and bolts is depicted, and these bolts are labeled as B1, B2, B3, and B4, respectively. It is worth noting that the shooting area of the CCD is outlined by the red square in Figure 5a. This illustrates that the method can even detect unseen bolts in the multi-bolt connections, making it applicable for connections in complex structures in which occlusion often occurs.

The reciprocating actuator applies a load (pressure) at the center of the back of the disk, causing it to wrap, as shown in Figure 5b. The varying amount of deformation results in a series of distinct speckle fringe patterns.

For the rigid multi-bolt connection in this experiment, we simplify each bolt to have one of two states: fully tightened and fully loosened. As a result, there are 16 ($2^4$) looseness cases in total, as illustrated in

Table 1. Arrangement of all looseness cases.

Case	B1	B2	B3	B4	Case	B1	B2	B3	B4
1	×	×	×	×	9	×	○	○	×
2	○	×	×	×	10	×	○	×	○
3	×	○	×	×	11	×	×	○	○
4	×	×	○	×	12	○	○	○	×
5	×	×	×	○	13	○	○	×	○
6	○	○	×	×	14	○	×	○	○
7	○	×	○	×	15	×	○	○	○
8	○	×	×	○	16	○	○	○	○

(where “○” represents “Loose” and “×” represents “Tight”). For example, case 9 indicates that bolts B2 and B3 are loose, but B1 and B4 are tight.

3.2. Image Acquisition

Illuminate the disk with the laser, activate the reciprocating actuator’s power, causing a continuous variation in deformation (abbreviated as D). Subsequently, initiate the process of capturing images and transferring them to the computer. It is important to note that D changes throughout the deformation process, giving rise to a series of distinct speckle fringe patterns. So, we had to capture sufficient images to build the dataset. We captured 4000 images (speckle fringe patterns) for each case, culminating in a total of 64,000 images. Taking case 16 (indicating all bolts are loose) as an example to demonstrate the changing images, as depicted in Figure 6: The initial image exhibits minimal deformation, with almost no discernible fringes. As the actuator continues its operation, the deformation progressively increases, resulting in denser fringes. The actuator then gradually reverts to its initial position, causing a reduction in deformation, and the fringes gradually become less dense. We note that the captured fringe patterns show strong time series characteristics. That is also why we chose the RNN model.

4. Results and Discussion

4.1. Bolt Looseness Detection

The representative speckle fringe patterns under all 16 cases of looseness are illustrated in Figure 7, corresponding to the 16 looseness cases in Table 1. Certain differences with noticeable features can be identified in these representative images.

The dataset (named as Dataset 1) utilized in this part comprised 64,000 images (with 4000 images for each of the 16 cases), where 60% served as the training dataset, 20% as the test dataset, and 20% as the validation dataset. The experimental platform configuration for algorithm execution included the following: CPU—AMD 5800X; RAM—16 GB @ 3600 MHz; GPU—NVIDIA GeForce RTX 3070; experimental environment—Python 3.9.

The RNN model was trained with the training and test datasets. Initially, the detection accuracy stood at only 6.25% (there are all 16 cases of looseness, i.e., $6.25\%=\frac{100\%}{16}$). After 30,000 iterations, the accuracy steadily increased to 99.07% on the test dataset, as illustrated in Figure 8.

And the accuracy of the trained model on the validation dataset was calculated. The confusion matrix of the model depicted in Figure 9 offers a nuanced understanding of the detection precision of the methodology under investigation.

In this matrix, each row represents the true label, and each column represents the predicted label. Each cell of the matrix then contains the number of samples under that true and predicted label. Each true label contains 800 images. The model, after undergoing rigorous training, exhibited an exemplary accuracy of 99.55% on the validation dataset of Dataset 1. Notably, the accuracy observed on the validation dataset marginally surpassed that of the test dataset. This phenomenon can be attributed to the presence of Dropout during the testing phase, which slightly influences the model’s accuracy. Conversely, during the validation process, the Dropout component was deactivated, enabling the model to harness the collective strength of all the weak classifiers, thereby resulting in a marginal enhancement in accuracy.

4.2. Detection Result under Different Perspective Angles

Moreover, to rigorously evaluate the robustness of the method to shooting perspective angles, supplementary experimental trials were executed under nonstandard perspective angles of 60° and 30°. As delineated in Figure 10, the imaging apparatus was strategically positioned at perspective angles of 90°, 60°, and 30°, respectively, to ensure a comprehensive analysis.

These additional experiments involved the systematic acquisition of visual data for 16 distinct cases, which were subsequently utilized for both the training and the testing and validation phases of the model. These additional experiments were instrumental in ascertaining the versatility and adaptability of the method.

Figure 11 displays a set of illustrative sample images captured from the specified angles, alongside counterparts taken at a 90° angle for reference. It becomes apparent upon examination that a reduction in the shooting angle leads to a gradual decline in image contrast, with the fringes becoming increasingly subtle and challenging to discern, which is especially noticeable at the 30° angle.

Our investigation included training both the proposed RNN model and the modified VGG-16 model (referred to as the MV model) from our previous work, using images captured at perspective angles of 90°, 60°, and 30°. We named the datasets collected at perspective angles of 90°, 60°, and 30° Dataset 1, Dataset 2, and Dataset 3, respectively. A comparative analysis of their detection accuracy on the corresponding validation dataset is presented in

Table 2. The detection accuracy on validation dataset of the MV model and RNN model at different perspective angles.

Perspective Angle	90°	60°	30°
Dataset	Dataset 1	Dataset 2	Dataset 3
Accuracy of the MV model	96.65%	94.34%	90.76%
Accuracy of the RNN model	99.55%	99.49%	99.38%

.

The MV model exhibited detection accuracies of 96.65% at 90°, 94.34% at 60°, and 90.76% at 30°, while the RNN model detailed in this study achieved accuracies of 99.55% at 90°, 99.49% at 60°, and 99.38% at 30°. It is noteworthy that the MV model’s performance was sensitive to the shooting angle. As the angle decreased, image contrast was reduced, leading to less distinct fringe patterns and a consequent dip in detection precision, with the accuracy settling at 90.76% for the 30° angle. However, the RNN model proposed herein demonstrated remarkable resilience, with detection accuracies consistently above 99%, indicating a minimal impact from variations in the perspective angle on the detection outcomes. These outcomes indicate that the proposed method effectively detects the loose bolts in multi-bolt connections with precision. Importantly, the method eliminates the need for distortion rectification, which is a complex process for traditional machine vision methods, especially for irregular shapes.

Moreover, we compared these two models in terms of the characteristics shown in

Table 3. Property comparisons of the MV model and RNN model.

		Time for One Epoch (Minute)	Iterations Required for Convergence	Time for a Single Judgment (Millisecond)	File Size of Single Model (Kilobyte)
MV model		11.91	195,000	7.17	540,540
RNN		4.23	30,000	4.70	221

. The MV model costed 11.91 min for a whole epoch and required 195,000 iterations for convergence. While the RNN model took only 4.23 min for one epoch, and only needed 30,000 iterations for convergence. Furthermore, the training process was faster than that for the MV model. The MV model took 7.17 millisecond for a single judgment, which is slower than RNN, which took 4.70 millisecond. In addition, one MV model’s size was 540,540 kilobytes, which is father bigger than that of the RNN model.

In Table 2, we have the validated effectiveness of the method at each separate angle, and we then tried to train a new RNN classifier that can detect loose bolts at any angle (90°, 60°, and 30°). Therefore, we combined Datasets 1, 2, and 3 into a comprehensive dataset (containing in total 192,000 images) and trained another RNN model to validate the detection accuracy of the new model. The confusion matrix of the detection result of the model on the validation dataset of the comprehensive dataset is shown as Figure 12 below. Each true label contains 2400 images. The accuracy of the model on the validation dataset of the comprehensive dataset was 99.40%, which is nearly identical to the performance of these models on individual datasets as shown in Table 2. This indicates that the new RNN classifier can detect loose bolts at any perspective angle (90°, 60°, and 30°).

4.3. Comparison with Traditional Methods

To gain a more intuitive understanding of the advantages of this method over traditional approaches, we conduct a performance comparison of these methods in

Table 4. Performance comparisons of the proposed method and traditional methods.

	Vision-Based or Contact Sensor-Based Method	Can Detect All Looseness Cases of Multi-Bolt Connections	Detection Accuracy	Average Time for a Single Judgment
The proposed method	Vision-based	Yes	Over 99%	4.70 milliseconds
Method 1 in Ref. [21]		Yes	92.13%	35 seconds
Method 2 in Ref. [23]		Yes	91.65%	-
Method 3 in Ref. [7]	Contact sensor-based	No	Over 99%	-
Method 4 in Ref. [14]		No	94.17%	-
Method 5 in Ref. [13]		Yes	96.69%	-

below.

Contacted sensor-based methods have been proved to be inconvenient in many cases and are vulnerable to environmental factors like temperature changes and humidity, potentially leading to inaccurate judgments. For example, method 5 in Ref. [13] attained a detection accuracy of 97.39% at 45 ± 1 °C but achieved 95.43% at 35 ± 1 °C for a four-bolt connection. Moreover, such methods are rarely able to identify all the looseness cases of multi-bolt connections.

Traditional vision-based methods, as we illustrated in Section 1, take too much time on prep work such as recognizing bolts, correcting perspective distortion, or 3D reconstruction before they can detect bolt looseness. The distortion rectification procedure in Ref. [20] is shown in Figure 13, which is complex and is limited to rectangular-based bolt arrays. Another limitation for these methods is the prerequisite of the initial angle of the bolts before loosening, significantly constraining the scenarios in which they can be effectively used.

Our proposed method directly performs an image classification to detect loose bolts and only takes 4.70 milliseconds for a single detection of a multi-bolt connection, which is much faster than traditional methods. On the other hand, the proposed method does not require the prior information of the initial angles of the bolts, has strong robustness to the perspective angle, and can detect unseen bolts. These findings underscore the efficacy and simplicity of the proposed method. In conclusion, the proposed method has great potential in real-time bolt looseness detection for multi-bolt connections, even connections in complex structures.

5. Conclusions

In this study, we proposed a precise, automated, and real-time method for detecting loose bolts in rigid multi-bolt connections. Firstly, the DSSPI system was established to capture shearing speckle patterns under different looseness cases. Secondly, fringe patterns were derived using the DSSPI subtraction algorithm. Thirdly, these fringe patterns were labeled and utilized for training the RNN model. In the actual detection of loose bolts, the pre-trained RNN classifier directly identifies the input fringe patterns to detect the loose bolts in multi-bolt connections.

The method can detect all the bolts within the measured surface at one time, which is efficient. It achieved an impressive detection accuracy of over 99% and a quick completion time of 4.70 millisecond for a single judgment (multiple bolts at one detection), which is quite accurate and thousands of times faster than other methods in the literature. Compared to other vision-based method, the proposed method neither requires recognition and localization of the bolts from the original images previously nor does it require prior information such as the initial angle of the bolts. It is also not perturbed by overlapping bolt heads that mask other loosened bolts. It can also detect unseen bolts in multi-bolt connections, making it applicable for connections in complex structures in which occlusion often occurs. Furthermore, this method skips the process of distortion rectification and has a wide detection angle (at least from 90° to 30°). The trained RNN classifier maintains a detection accuracy over 99% at any perspective angle (90°, 60°, and 30°).

Overall, the effectiveness, robustness to perspective angles, and robustness to occluded bolts of the method, which have been evidenced in the experiments, illustrate its great potential in multi-type connection looseness detection, even for connections in complex structures. Additionally, the method achieves real millisecond detection, which can be used in fast, real-time monitoring. That is a promising property of integrating it with webcams for the remote, real-time, and multi-scenario monitoring of connection looseness in future research. The application of the method enables bolt connections’ lifecycle detection and management, thereby enhancing the accuracy and automation level of the detection work.