Detection of Anomaly in a Pretensioned Bolted Beam-to-Column Connection Node Using Digital Image Correlation and Neural Networks

Abstract

Bolted connections, commonly applied in civil engineering structures, have many advantages. According to current trends, bolted connections in steel structures are designed as prestressed ones. Unfortunately, precise control of the prestressing forces is difficult, while the loosening (due to, e.g., dynamic interactions) may be dangerous for the entire structure. There are many control methods applied in the determination of the tightening level, among which are vision-based methods. The methods described so far enable—thanks to image processing—damage detection in connections with visible connectors. The level of the considered loosening was significant—in many cases, changes in connectors were visible with the naked eye, whereas the procedure presented here enables the detection of very small changes, impossible to detect without manual inspection of every single connector. It is not necessary to observe the connectors directly, but the near surrounding of the node should be visible. As a measurement technique, Digital Image Correlation (DIC) was used. The applied measurement method and the high sensitivity of the presented procedure makes the presented research original. The currently presented procedure, employing Artificial Neural Networks, based on the laboratory examination of an example of one selected beam-to-column connection of a two-story steel portal frame, was perfect in the detection of a change and in the determination of the number of loosened rows, 95%, and their location, 94%, with the number of false alarms below 1%.

1. Introduction

Bolted connections are commonly applied in civil engineering structures, and they are easy to implement both in the workshop and on the construction site. They allow for easy replacement of damaged components, do not require highly qualified workers (as in the case of welded joints), and are usually cheaper. According to current trends, bolted connections in steel structures are designed as prestressed ones; various methods are used to preload the connectors in order to increase the load capacity. Phares et al. [1] describe various prestressing methods together with their accuracy and the techniques of verification of forces in bolted joints according to the United States’ rules. Moreover, the authors of [2] compared 16 measurement methods to control the prestressing forces in bolts and proposed the division of these methods into direct and indirect ones.

Precise control of the prestressing forces is difficult, while the loosening (due to, e.g., dynamic interactions) may be dangerous for the entire structure. This subject was studied by You et al. [3]; for a plate girder connection, the relation between the pretension loosening in bolts and its deflection was analyzed.

The identification of forces in bolts may be performed using different approaches, among which there are elastic wave propagation phenomenon-based methods as well as ultrasonic and vibration-based methods. Some attempts to determine the value of forces in bolts in a complex flange connection using elastic wave propagation, via Principal Component Analysis (PCA) and Artificial Neural Networks (ANNs), are presented in [4]. Guided waves were applied by Yang et al., see [5], to the issue of the detection of changes in the lap joint. The correlation of the signals recorded by piezo-transducers located in the vicinity of the tested connectors with the base signals was considered in the detection of loosening bolts by Ruan et al. The method was described as easy to use, operating in real time, and insensitive to interference, see [6]. Amerini and Meo (see [7]) presented the results of the detection of changes in lap-bolted connections obtained using ultrasonic measurements and three different approaches: the acoustic moment method, the second harmonic method, and the sidebands method. The loosening/tightening state of bolted structures was described by newly developed indices, which allowed for the precise determination of the health state with an average error lower than 5%.

Another group of methods used to measure prestressing forces in bolts are vibration-based measurements. In [8], the relation between the first bending natural frequency of a single connector and its prestressing force is given. The procedure applied in laboratory tests enabled the very precise detection of loosened fasteners. However, this approach is not ready to be applied in field tests since high measurement accuracy is required but is difficult to be obtained in real-life structure measurements. The article [9] shows the impact of loosening connectors in the Frequency Response Function (FRF), which changes not only its shape but also decreases the resonance frequencies values. The publication also highlights the complex non-linear nature of bolted connections. A non-linear approach in the frequency domain used to detect loosened fasteners was presented by Li and Jing in [10]. A very interesting application of the vibration-based method for the detection of loosening was presented in [11,12]. The authors, basing on acceleration signals and convolutional neural networks, developed a method that allows for the detection of loosening, both in a single node and in several nodes simultaneously. Although this approach “can be easily applied for real-time structural health monitoring of any engineering structure” [11], the size of detected loosenings was greater (loosenings were visible with the naked eye) than that considered in this paper. Another approach to detect changes in bolt tightening relies on the application of intelligent washers, see, e.g., [13]. Changing the bolt tightening affects the frequency of free vibrations of a cantilever pad with piezoelectric sensors (PZT) mounted on it. The analysis of the signals excited and measured by the PZT sensors makes it possible to assess these frequencies.

Due to the vigorous development of vision-based techniques and image processing algorithms, vision-based methods are also increasingly used in the subject under consideration. Verification of the proper tightening of fasteners in steel joints is presented in [14], where convolution networks are applied to analyze images of the monitored connection. Although the authors declared a high efficiency of fault detection (over 90% of cases) and the possibility of using the procedure for remote monitoring of the connection status in real time, it should be noted that the system can detect the loosening of fasteners when the height of the protruding part of a bolt exceeds 0.5 cm, which, in the case of prestressed connections, may not be sufficient. The problem of bolted connection change detection, based on image analysis, was also dealt with in [15], where the proposed approach combined a cascade bolt detector with Support Vector Machines (SVMs). Work [16] presents algorithms for the automatic identification of a loosened bolt location; the algorithm used the comparison of photos taken during various inspections (undamaged and with a loose connector). Some other algorithms that are based on the detection of washer edges are presented in [17,18]; the identified location of washers in relation to the initial state enabled the detection of changes in bolt tightening.

It is not only the detection of fastener loosening that is important—some other phenomena may also be dangerous for bolted connections; these include stress concentration points, cracks (propagating from the edge of the holes), and corrosion. The work [19] analyzed the change of stress (measured by strain gauges) in bolts, depending on the change in the bolt head surface (the head was cut off to simulate a corrosion defect). Stress concentration issues, especially around welds of connections subjected to cyclic loading (e.g., seismic), were analyzed in [20].

The problem of detecting changes in connections is also dealt with in the aviation industry. For example, in work [21], looseness of connectors in the turbine-plane connection was detected, and in work [22] a decision system for monitoring the condition of bolted joints for large-size aluminum plates used in aircraft plating was proposed. The system was developed on the basis of Lamb wave propagation analysis; the waves were recorded by a piezoelectric transducer system.

The described vision-based methods allow for damage detection, as a result of image-processing, on which the connectors are visible. The level of loosening, considered in these papers, was significant. In many cases, changes in connectors were visible with the naked eye, whereas the procedure presented here enables the detection of very small changes, impossible to be detected without manual inspection of every connector. It is not necessary to observe the connectors directly, but the near surrounding of the node should be visible. These two reasons make the presented research original. A similar problem of anomaly detection in nodes has been admittedly presented in [23], where the anomaly detection was supported by SVMs, but in the now proposed procedure a different tool is used, namely, Artificial Neural Networks (ANNs).

2. Formulation of the Problem

The proposed procedure enables the detection of anomaly occurrence in pretensioned bolts of steel connections, based on Digital Image Correlation (DIC) in-plane measurements. The basis for assessing the condition of the connectors is observation of changes in the selected area neighboring the investigated node, during harmonic vibrations of the system. Many parameters were taken into account, e.g., the vertical and horizontal displacements of selected points, the linear and rotational displacements of segments, as well as the varying variants of plane figures. The aim of the procedure is not only the detection of changes in connections, but also the determination of their types and the location of loosened bolts. For the purpose of better visualization, the consecutive steps of the proposed procedure are shown in Figure 1.

The applied measurement technique—Digital Image Correlation—is a vision-based measurement method enabling contactless registration of the observed sample shape and dimensions, as well as the full field displacement measurements within the observed area of the tested sample. On the basis of the data collected using DIC, it is possible to determine the deformations and accelerations of points in the observed area and the material constants: Young’s modulus or Poisson’s ratio [24].

The method is used for both static and dynamic tests [25]. It may be applied to analyze long-lasting processes (e.g., fatigue tests [26]), as well as fast-changing phenomena (e.g., vibrations and/or cracking).

The main idea of DIC is to track a digitally registered random pattern placed on the tested surface; the pattern may be either natural or created by a DIC operator before the measurements. The tested surface changes its shape during loading, and so does the pattern on it; thanks to algorithms based on the correlation of photo sequences taken during the sample deformation process, it is possible to determine the location of selected (or even all) points in the consecutive photos (and in the consecutive steps of the deformation phase) [27]. One camera enables the determination of in-plane displacements; the use of a two-camera system allows for the observation of displacements of a plane sample in 3D.

DIC is a rapidly growing measuring method [28]. In recent years, some new multi-camera DIC systems that allow for the observation of samples from many sides simultaneously (e.g., a cylinder observed from all around) have been developed. These systems require at least four cameras for recording changes on the edge of the two planes, or observing two opposite sides of the sample. The increase in the number of cameras allows for the observation of objects with increasingly complex shapes [29]. Unfortunately, the number of cameras used during the measurement determines the level of complication of the task.

DIC has found application in both destructive and non-destructive testing, on both macro and micro scales. Grygierek et al. [30] studied deformations of a road and pavement under a heavy truck load, while Rusinek and Kopernik [31] used 3D DIC for microscopic examination of biocompatible materials. Kujawińska et al. [32] proposed the use of DIC in a hybrid system of simultaneous measurement of temperature distribution and displacement of engineering objects. The ability to measure displacements in all three directions was used in dynamic tests of aircraft plating components [33], as well as for inventory and monitoring of the “Monte Seco” railway tunnel [34] and for the measurement of cracks in reinforced concrete beams [35]. In [36], the authors present the use of DIC as a method enabling verification of the anchoring of composite samples in thw jaws of testing machines; the obtained results were additionally used to determine the properties of tested materials and for the validation of numerical models. Validation of numerical models using the data obtained from DIC was also presented in [37,38]. DIC helps in understanding the mechanism of cracking [26,39] and can be used to monitor the deflection of a bridge [40] or to detect damage [23]. The works [41,42] compared the DIC method with laser vibrometry in the application for dynamic measurements. Due to the fact that the result of DIC measurements is information about displacements or deformations of the studied area, this method was readily used to obtain data for the validation of numerical models [43]. Other examples of DIC usage are described in [44,45,46,47].

On the basis of the numerical examination of a two-story steel portal frame (see Figure 2a), the hypothesis was formulated that observation of rotations of elements during dynamic tests can be useful for the detection of damage in nodes. However, numerical studies are not the sole subject of consideration, but they underlie the research presented here. In this paper, 2D DIC was used—indirectly—as a rotation gauge.

The dynamic measurement of the angle of rotation is difficult or impossible to perform using the usual equipment. Even in static examination (e.g., in measurements of rotational stiffness of connections in steel structures), it is rather complicated [48,49], but the changes in the angle of rotation can also be indirectly measured by observation of changes in the shape of the selected area. During the motion of the element recorded in photos, all types of displacements are combined (vertical and horizontal displacements as well as rotations). When using Cartesian coordinates, the description of a number of subsequent planar transformations (translations, rotations, scaling, etc.) requires that single transformation matrices (describing subsequent transformations) have to be combined through addition and/or multiplication. In computer graphics, this problem is avoided by the application of so-called homogeneous coordinates [50], which allow for a description of all the transformations using a single $3\times 3$ transformation matrix. Each point with Cartesian coordinates $(x_C,y_C)$ is represented by the $(x,y,W)$ homogeneous coordinates (for $W=\mathrm{c}onst$, the points are located on the same projective plane). Transformation from homogeneous to Cartesian coordinates is simple:

$$\begin{array}{c}{x}_C=x/W\\ y_C=y/W\end{array},$$

(1)

but there is no unique transformation in the other direction, since there are many possible projective planes (values of W). Inspired by affine transformation in homogeneous coordinates, the connection between the new location of points can be estimated according to Equation (2), if only the transformation is affine and the coefficient matrix is known.

$$\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ 1\end{array}\right]=\left[\begin{array}{ccc}a& b& c\\ d& e& f\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$$

(2)

where

• $x\prime ,y\prime ,1$—new homogeneous coordinates;
• $x,y,1$—old homogeneous coordinates;
• $a,b,c,d,e,f$—transformation matrix coefficients.

The problem can be inverted: when the old and new homogeneous coordinates of three points are known, the coefficients of the transformation matrix (the so-called affine parameters) can be calculated and describe the transformation that has taken place (neglecting the type of motion). Making a decision about the loosening of connectors based on six transformation coefficients means working in a 6D space, which is quite a complicated task. Therefore, artificial intelligence in the form of single layer feedforward neural networks was involved.

ANNs are widely implemented in the field of Structural Health Monitoring (SHM). Thanks to their generalization ability, they help to solve both classification and regression problems, even those based on inaccurate and/or noisy data sets. Though many types of ANNs are available, simple feedforward networks are still very popular for the structure condition monitoring of buildings. An exemplary application of ANNs in the considered field can be found in many areas, e.g., in damage detection based on time signals registered during dynamic measurements [51], in an assessment of building damage and safety after an earthquake [52], and in the identification of internal forces in pretensioned bolts [4].

3. Laboratory Experiment

3.1. Measurements

Two dimensional DIC measurements, applying one high-speed Phantom v341 camera (see Figure 2b) were conducted during vibrations of the frame caused by harmonic excitation at the frequency of 106 Hz (The recorded data are available at the reader’s request. Please note that these are huge files (tens of Gigabytes)). The decision regarding the excitation frequency required the analysis of all natural frequencies of the investigated structure; possible changes of the natural frequencies (caused by some introduced damages) had to be taken into account. Finally, it was decided that an excitation with a frequency of 106 Hz would be applied, because this frequency was offset by at least 10% from all natural frequencies corresponding to global in-plane mode shapes. One of the local in-plane mode shapes (with moving beams, without any movement of columns) was very close to the chosen excitation frequency (the difference was below 1%); however, the excitation point was chosen so that this particular mode shape was not induced and no resonance appeared.

The measurements were focused on one selected beam-to-column connection, marked in Figure 2c, made with eight pretensioned bolts of the M8.8 type (see Figure 2d,e). During DIC measurements, the bolts in the selected connection were loosened according to the scenarios shown in Figure 2d. The symbols used for the description of applied scenarios, namely Sxy, were the combination of the letter “S” (scenario) and two numbers, $x\in \{1,2,3,4\}$ and $y\in \{1,2,3,4\}$, which denote the row with loosened bolts (where 1 is the top row, 4 is the bottom row); in cases where bolts were loosened only in one row, $x=y$ (e.g., S44 means that only the 4th row of bolts was loosened). Despite the loosening of nodes, the changes in the whole connection, as well as in every bolt were invisible with the naked eye and their detection was possible only by manual verification. Three measurement series were carried out and all scenarios were implemented in each of them. Each single measurement (in one connection state) lasted one second, with the recording frequency equal to 2120 Hz.

According to the requirements of the applied method, the surface surrounding the node was prepared by painting with a speckled pattern, shown in Figure 2e. Only the left area, indicated by the red oval, was considered in this paper. The right one will be analyzed in the future. Figure 2f shows point OO together with the other points on the crossing of red lines, in which the displacements were analyzed with an accuracy of 0.0007 mm. The obtained accuracy was the result of the specified measurement parameters (dimension of the region of interest: 51.48 × 102.94 mm, picture size: 756 × 1560 px) and the accuracy of the ISTRA 4D software (0.01 px) [53].

3.2. Preprocessing

The analysis of picture (It should be noted that each of the pictures recorded in the series should be called a frame; however, in order to avoid confusion with the considered structure (a steel frame), the word picture is used throughout the paper.) sequences was made using commercial software ISTRA 4D [53]. The obtained displacements were then exported to the MATLAB environment [54], where all other calculations were made. Every registered photo sequence was divided into smaller samples, which consisted of 20 consecutive pictures (called here “a specimen”). The displacement of the OO point in every picture of each specimen was studied, and two extreme states were selected, the changes in location of the point OO in photos of one exemplary specimen, as well as the selected steps (pictures), are shown in Figure 3a. The net of black dots, shown in Figure 3b, presents all points for which the displacements were measured. Due to the multitude of data describing the connection condition, their number had to be reduced; the idea presented here was to use affine parameters as a description of the observed area changes occurring between two extreme locations (see Figure 3a). As mentioned in Section 2, in order to calculate six affine parameters, the information about the location of three points in two states is necessary. The selected points are shown in Figure 3b as the vertexes of a triangle. In this figure, two states were compared, but to highlight the differences the parts in the neighborhood of vertexes were enlarged and shown in Figure 3c. After the conversion of Equation (2), six parameters for every specimen were calculated and created “a pattern”. This was the first, but not the only set of parameters describing the connection condition. The following variants were considered:

• ${\mathbf{x}}_{(6x1)}=\{a,b,c,d,e,f\}$—coefficients of homogeneous coordinates transformation, see Equation (2);
• ${\mathbf{x}}_{(6x1)}=\{\Delta x_1,\Delta x_2,\Delta x_3,\Delta y_1,\Delta y_2,\Delta y_3\}$—differences in Cartesian coordinates of every triangle vertex (see Figure 3);
• ${\mathbf{x}}_{(8x1)}=\{\Delta x_1,\Delta x_2,\Delta x_3,\Delta x_4,\Delta y_1,\Delta y_2,\Delta y_3,\Delta y_4\}$—differences in Cartesian coordinates of every quadrangle vertex; the quadrangle was created by adding an additional point.

3.3. Damage Detection

Feedforward ANNs were used for anomaly detection. All patterns were divided (evenly for each case of the considered scenario) into learning and testing sets in the ratio of 2:1. The division remained constant throughout the experiment. Every set of patterns P was described according to the following formula:

$$P={\{{\mathbf{x}}^i,{\mathbf{t}}^i\}}^n$$

(3)

where

• ${\mathbf{x}}_{\left(Nx1\right)}^i$—the input vector for the i-th pattern, the number of inputs $N\in \{6,8,9\}$;
• ${\mathbf{y}}_{\left(Kx1\right)}^i$—the target vector of the i-th pattern, the number of outputs $K\in \{1,4\}$;
• $n$—the number of patterns in the considered set; in the learning set $n=1826$ and in the testing one $n=913$

A number of variants of input and output parameters were considered, see

Table 1. Input (x) and output (y) vectors, as well as targets (t) and the value of the best success ratio (SR) obtained for Artificial Neural Networks (ANNs) considering their type.

Input	Target	Output	Success Ratio
$\mathbf{x}$	$\mathbf{t}$	$\mathbf{y}$	$SR$
ANN TYPE 1
$\mathbf{x}=\{a,b,c,d,e,f\}$	$\mathbf{t}=\left\{C\right\}$	$\mathbf{y}=\left\{C\right\}$	$87\%$
affine transformation parameters	C - integer parameter	C - continuous value
	describing the class	from the range
	of anomaly
	$C\in \{0,1,2,3,4,5,6,7,8\}$	$C\in <0,8>$
ANN TYPE 2
$\mathbf{x}=\{a,b,c,d,e,f\}$	$\mathbf{t}=\{R1,R2,R3,R4\}$	$\mathbf{y}=\{R1,R2,R3,R4\}$	94%
affine transformation parameters	$R1,R2,R3,R4\in \{0,1\}$	$R1,R2,R3,R4\in <0,1>$
or	$R_r$ - parameter
$\mathbf{x}=\{\Delta x_1,\Delta x_2,\Delta x_3,\Delta y_1,\Delta y_2,\Delta y_3\}$	describing the condition
differences in location	of r-th row of bolts
of every triangle vertex
or
$\mathbf{x}=\{\Delta x_1,\dots ,\Delta x_4,\Delta y_1\dots ,\Delta y_4,\}$
differences in location of every
vertex of the quadrangle
ANN TYPE 3 - two-step procedure
$\mathbf{x}=\{a,b,c,d,e,f\}$	$\mathbf{t}=\left\{D\right\}$	$\mathbf{y}=\left\{D\right\}$	100%
affine transformation parameters	$D\in \{0,1\}$	$D\in <0,1>$	appearance
or
$\mathbf{x}=\{\Delta x_1,\Delta x_2,\Delta x_3,\Delta y_1,\Delta y_2,\Delta y_3\}$	D - parameter
differences in location	describing occurrence		95%
of every triangle vertex	of anomaly in the selected type		size
	$\mathbf{t}=\left\{C\right\}$	$\mathbf{y}=\left\{C\right\}$
$\mathbf{x}=\{D^{S00},D^{S33},D^{S22},D^{S23},D^{S44},$	$C\in \{0,1,2,3,4,5,6,7,8\}$	$C\in <0,8>$
$D^{S34},D^{S14},D^{S11},D^{S12}\}$	in task		92%
	of location determination		location
	or
	$C\in \{0,1,2\}$	$C\in <0,1,2>$
	in task
	of size determination

:

$${\mathbf{y}}_{\left(Kx1\right)}^{i,j}={\mathrm{ANN}}^j({\mathbf{x}}^i,{\mathbf{w}}^j)$$

(4)

where

• ${\mathbf{y}}_{\left(Kx1\right)}^{i,j}$—the output vector of the i-th pattern and the j-th neural network of the specified architecture;
• ${\mathrm{ANN}}^j$—the ANN of the $N—Hi—K$ architecture, with $Hi$ hidden neurons;
• ${\mathbf{w}}^j$—the free parameters vector for the j-th neural network of the analyzed architecture $N,Hi,K=const$.

The output vectors took one of the following forms (see Figure 2d and Table 1):

• ${\mathbf{y}}_{(1x1)}=\{C\}$, where C was the class of anomaly; adopted classes:

  –

  concerning the determination of anomaly location:

  $C=0$ for Class S00, $C=1$ for S33, $C=2$ for S22, $C=3$ for S23, $C=4$ for S44, $C=5$ for S34, $C=6$ for S14, $C=7$ for S11, and $C=8$ for S12,

  –

  concerning the determination of size of anomaly:

  $C=0$ for Class S00, $C=1$ for S11, S22, S33, and S44, and $C=2$ for S12, S14, S23, and S34;

• ${\mathbf{y}}_{(4x1)}=\{R1,R2,R3,R4\}$, where $Rr$ is the described condition of the r-th row of bolts, starting from the top and ending with the bottom one; $Rr=0$ means that the r-th row is properly fastened, $Rr=1$ means that the r-th row is loosened;
• ${\mathbf{y}}_{(1x1)}=\{D\}$, where D determines the occurrence of the anomaly of the selected type, $D=0$ means that the anomaly in the selected type does not exist, $D=1$ means that the anomaly in the selected type exists.

It should be noted that the elements of output vectors were continuous values in specified ranges, while the elements of target vectors were discrete values (see Table 1). The number of input and output parameters determined the network architecture.

The learning process was repeated 30 times for each architecture, because of the random nature of weight adoption in the first step of iteration. All networks of the analyzed architecture were used for the purpose of anomaly appearance detection and the output value ${\widehat{\mathbf{y}}}^i$ for a single pattern was given after the statistical calculation, conducted using Equations (5) though (8). The averaged k-th element of the output vector y for the i-th pattern, for the j-th neural network of the specified architecture, is defined as

$${\overline{y}}_k^i=\frac{{\sum }_{j=1}^{30}{y}_k^{i,j}}{30}.$$

(5)

Considering the standard deviation,

$${\sigma }_k^i=\sqrt{\frac{{\sum }_{j=1}^{30}{(y_k^{i,j}-{\overline{y}}_k^i)}^2}{30}},$$

(6)

the set of k-th outputs for the i-th pattern without outliners $B_k^i$ is specified as

$$B_k^i=\{y_k^{i,j}\}for{\overline{y}}_k^i-3{\sigma }_k^i\le y_k^{i,j}\le {\overline{y}}_k^i+3{\sigma }_k^i,$$

(7)

and ${\widehat{y}}_k^i$ for the k-th element of the output vector for the i-th pattern can be calculated using the following formula:

$${\widehat{y}}_k^i=\mathrm{round}\left(\frac{\sum y_k^{i,j}\in B_k^i}{\overline{\overline{B_k^i}}}\right),$$

(8)

where $\mathrm{round}(·)$ returns the nearest integer treated as the class of anomaly, and $\overline{\overline{B_k^i}}$ is the number of elements in the $B_k^i$ set.

The correctness of the classification was checked according to

$$CL(i)=\left\{\begin{array}{c}1for{\mathbf{t}}^i-{\mathbf{y}}^i=\mathbf{0}\\ 0for{\mathbf{t}}^i-{\mathbf{y}}^i\ne \mathbf{0}\end{array}\right.,$$

(9)

where $CL(i)$ is a proper classification coefficient.

The number of neurons in the hidden layer ($Hi$) also varied. It was optimized every time to increase the success ratio of anomaly detection $SR$:

$$SR=\frac{{\sum }_{i=1}^{n}CL(i)}{n}·100\%.$$

(10)

Obviously, $SR\to 100\%$ means success, and $SR\to 0\%$ means failure.

Three types of network architecture, designed for solving the problem, are shown schematically in Figure 4. The attempt to detect the type of anomaly implied the application of ANN Type 1 (see Figure 4a). For the six-element input vector, the elements of which were affine transformation parameters, the information about the class of anomaly (specified by its number) was searched. The results of the architecture optimization process, presenting SR that was dependent on the number of hidden neurons for learning and testing sets are shown in Figure 5a. It is evident that the increase of hidden neurons above 12 did not cause an improvement of SR, which remained at a level of 87%. However, it was observed that the scattering of results for individual classes (see Figure 5b) is smaller for a greater number of hidden neurons.

In order to improve the classification results, ANN Type 2 was used (see Figure 4b). On the basis of the same input vectors as used previously, networks provided the information about the appearance of loosening in each single row of bolts. The values of the elements of the four-element output vector determined the condition of bolts in four rows of the investigated connection, from the uppermost and ending at the lowest row. The results of optimization of the hidden neurons number, as well as the success ratio for ANNs of the architecture 6-35-4 are presented in Figure 6a. In this case also, the SR level is rather constant for $Hi\ge 12$, but it is higher than before and equals 94%. In Figure 6b, it can be seen that some anomalies are precisely recognized, while the others are not, so the problem was more carefully studied, and the results (for the ANN architecture 6-35-4) are shown in Figure 7. In this figure, the value that equals 0 corresponds to the correct tightening of the fasteners (no defect identified); when it equals 1, the bolts in the considered row are loosened (a defect is found). A careful analysis of the charts shows that some different anomalies provide the same output vectors; e.g., Class S11 (damage only in the first row, ${\mathbf{t}}^i=\{1,0,0,0\}$, see Table 1, in many cases gives ${\widehat{\mathbf{y}}}^i=\{1,0,0,1\}$) is similar to Class S14 (damage in the first and fourth row, ${\mathbf{t}}^i=\{1,0,0,1\}$, but in some cases ${\widehat{\mathbf{y}}}^i=\{1,0,0,0\}$). The efficiency of the precise determination of the type of anomaly by the networks of this architecture remained at the same level regardless of the adopted input parameters (parameters of affine transformation or the difference in the position of the triangle or quadrangle vertexes).

Considering the fact that some of the anomalies produce very similar effects, a different approach was considered. The two-step procedure, called “ANN Type 3”, in which ANN Type 1 was combined, was implemented. Firstly, nine ANN Type 1 procedures were adopted, each trained to recognize only one type of anomaly ($\mathbf{y}=D$). Each time, the number of neurons in the hidden layer was optimized in an analogous manner, as described previously. The results of damage detection in nine analyzed cases are presented in Figure 8. It is also clearly visible in this case that some of the anomalies are precisely detectable when others are not. Therefore, the combination of all nine outputs should be the base for decision making about the connection condition, and they were combined into a new vector of input parameters for ANNs on the second step of the procedure. One output described the class of damage (0, 1, …, 8). The results are shown in Figure 9. The efficiency of this approach is a little bit lower than that for ANN Type 2 and equals 92%. The diagram in Figure 9b shows the percentage distribution of the network output for patterns in different classes, considering learning (blue color) and testing (red one).

Slightly better are the results of anomaly size determination (see Figure 10), based on nine outputs (like in previous case). This time, only three classes were marked out: without anomaly (Class 0), with anomaly in one (Class 1) or in two rows (Class 2). The detection of the anomaly’s appearance is perfect, but the determination of its size is on the level of 95%.

4. Conclusions

In this paper, a procedure of anomaly detection in bolted connections based on displacement obtained using DIC during dynamic measurements is presented. The usage of the information on horizontal or vertical displacements enabled the proper detection of failure in up to 80% of the analyzed cases. Enrichment of the data with rotations, however indirectly measured (the direct estimation of rotations in a form of the first derivative of displacement fields was not considered), increases the procedure efficiency: in the detection of a change to 100% and in the determination of the number of loosened rows to 95% and their location to 94%, with the number of false alarms below 1%. In every considered type of ANN, the least recognizable anomalies were S11, S14, and S44. The differences between them were too small to precisely separate these cases. Despite that, the obtained results are considered to be very good, especially taking into account the size of the studied anomalies. Taking into account the influence of the dimensions of the observed area and the camera’s resolution, it can be stated that the proposed method is suitable for local applications, as it is in this paper. Although DIC measurement application in in situ examinations still requires more testing, its usage in SHM for civil engineering structures affords new possibilities. DIC is a contactless measurement technique, so it is especially useful for monitoring parts of structures that are hard to reach. There is no necessity to mount signal cables, so the influence of long-length cables on the registered signals has been eliminated. The sensors’ power supply problem, as in the case of wireless sensors, does not exist. Depending on the optical system applied (e.g., a type of adopted lens), it is possible to observe places that are either far away or very close (and are very small).

The proposed procedure is not a replacement of existing ones (as, e.g., the simple but very efficient signal-based SHM), but it can be rather an alternative and complement for traditionally applied tools. It has to be taken into account that the procedure does not work in real time, since the DIC calculations have to be done off-line; the observed area should be visible and properly prepared (random pattern). Moreover, the approach discussed in the paper needs rather expensive hardware, since the excitation frequency was quite high compared to those that can be found in real buildings or bridges. In consequence, the use of a high-speed camera was required and very good, with stable lightning conditions for registering pictures with extremely short exposition time; in case the excited/monitored frequency is lower (a few hertz), such sophisticated hardware will not be required and the system will be much cheaper.