*3.3. Evaluating Accuracy and Precision of Microsoft Kinect V1*

Even though Microsoft Kinect V1's overall accuracy and precision were previously evaluated by Khoshelham et al. [30]; in this work, its evaluation was performed for the specific working conditions required for structural health monitoring. To this end, the deflections at 32 reference points of the wing were measured using Kinect V1 sensor, and they were compared with the measurements from a laser ranger. To track the locations of reference points in the image plane, eight ArUco [34] markers, each having four corners were used. The patterns known as ArUco markers are small two-dimensional (2D) barcodes often used in augmented reality and robotics, as shown in Figure 8. ArUco was developed by Garrido-Jurado et al. [34], where they showed the superiority of their work to other known markers in literature such as ARTOOLKIT [35], ARToolKit Plus [36], and ARtag [37]. The locations of these 32 corners in image plane were detected in subpixel accuracy using the algorithm provided by Garrido-Jurado et al. [34] and the obtained results are shown in Figure 8.

**Figure 8.** (**Left**) Sample ArUco Markers. (**Right**) Corner extraction in subpixel accuracy from ArUco markers.

### 3.3.1. Mapping Depth and RGB Images of Microsoft Kinect V1

To use ArUco markers for tracking the corners during deflection measurements, the RGB and depth images obtained from Kinect V1 were mapped using an affine transformation. Using *n* known points of the wing, such as its four corners, the unknown parameters of the transformation can be calculated, as follows:

$$C = AD \tag{11}$$

$$A = \mathbb{C}D^{-1} \tag{12}$$

where *<sup>C</sup>* ∈ 2×*<sup>n</sup>* contains the locations of a point in the colored image, *<sup>D</sup>* ∈ 3×*<sup>n</sup>* contains the corresponding locations in the depth image, and *<sup>A</sup>* ∈ 2×<sup>3</sup> is the affine transformation matrix.

By following the above steps, depth images were successfully mapped to the RGB images that were acquired from the Kinect V1, as shown in Figure 9. As seen from this figure, the color images were changed to grayscale, so as not to work with unnecessary color channels. This way, the corners of the ArUco markers, which act as reference points, can be used for tracking the deflection changes at these points both by Kinect V1 and a laser meter, which will act as ground truth.

**Figure 9.** Kinect V1 depth and RGB images in grayscale (**Left**) Before mapping. (**Right**) After mapping.

3.3.2. Microsoft Kinect V1 vs Leica DISTO X310 Laser Meter

After the RGB and Depth images were mapped, the difference between the deflections that were measured by Kinect V1 and laser ranger at corner points were calculated. To account for errors in corner detection and mapping, for each detected corner pixel from RGB image, a 3 × 3 window of pixels was chosen from the corresponding depth image. A Leica DISTO X310 laser meter [38] with ±1 mm accuracy was used for evaluating the accuracy of Kinect V1. Loads with magnitudes of 2.45, 4.9, 7.35, 9.81, 12.26, and 14.71 N were placed on position 4 of the wing and the deflections were measured using both the Kinect V1 and Leica DX310 sensors. The obtained results are shown in the left image of Figure 10 for 2.45 N load, in which the mean absolute error for the 32 corner points was measured to be 2.0556 mm with a standard deviation of 1.74 mm. This was repeated for 100 frames, and the obtained results are shown in the left image of Figure 11, in which the mean of absolute errors was measured to be 2.0642 mm with a standard deviation of 0.21 mm. Similarly, the results obtained for 14.71 N load are shown in right images of Figure 10 and Figure 11, in which the mean absolute error for 32 corner points was calculated to be 2.6875 mm and 2.1520 mm for a single and 100 frames, respectively, with standard deviations of 1.615 mm and 0.26 mm. The same procedure was repeated for all of the aforementioned loads that range from 2.45 to 14.71 N, and the obtained results are summarized in Table 1. These results show that for the required working conditions, the Kinect V1 has an accuracy of ±2.25 mm with a standard deviation of 0.28 mm and a resolution of 1 mm. Therefore, since the deflections due to different loads range from 1 mm to 19 mm in this work, the Kinect V1 is proved to be suitable for acquiring the deflection measurements reliably. Moreover, neural networks already take the noise of the sensors into account when they are trained with the data acquired from them; therefore, the small noises in the acquired data are taken care of by the proposed method.

**Figure 10.** Absolute errors at 32 corner points of the wing for (**Left**) 2.45 N load. (**Right**) 14.71 N load.

**Figure 11.** Mean of absolute errors in 100 frames for (**Left**) 2.45 N load. (**Right**) 14.71 N load.


**Table 1.** Accuracy and Precision of Microsoft Kinect V1.

#### *3.4. Visualization of Depth Features Acquired from Microsoft Kinect V1*

Here, the visualization of depth features that were obtained from Kinect V1 for loads acting on the wing is shown. As mentioned, the spanwise deflections of the wing due to the application of the external loads were captured by the depth camera. Figure 12 illustrates these deflection measurements for the distinct loads acting on position 1 and 8. These illustrations were made by reconstructing the surface of the wing from the acquired depth images after the application of each load. These images show how the overall bending profile of the wing changes due to the magnitudes and locations of the external loads. The Kinect V1 sensor can measure deflections due to loads as small as 2.45 N acting on position 1 of the wing, as seen from these measurements. Deflection profile is especially visible for loads acting on Position 8, since it is the furthest point from the fixed part of the wing.

**Figure 12.** Wing deflections due to loads placed at (**Left**) Position 1 (**Right**) Position 8.

Moreover, in order to show the effect of the same load applied on different sections of the wing, loads of 2.45 and 14.71 N were applied on all eight distinct partitions of the wing, and the resulting depth images were plotted in Figure 13. These results clearly indicate that the wing deflection profile is different even when the same load is applied on different locations. Furthermore, by observing these results one can see a strong relationship between the behavior of the deflection of the wing for loads placed in parallel, namely loads placed at Positions 1 and 5, 2 and 6, 3 and 6, 4 and 8. These results prove the effectiveness of Microsoft Kinect V1 sensor for wing deflection measurement under various loading conditions acting on distinct sections of the wing.

**Figure 13.** Wing deflections due to (**Left**) 2.45 N and (**Right**) 14.71 N loads applied on distinct positions of the wing.

### **4. Dataset Creation and Experimental Results**

#### *4.1. Dataset Creation*

A dataset of wing deflections due to loads was constructed by extracting the depth features from the Microsoft Kinect V1 sensor operating at 30 Hz by following the procedure explained in Section 3.2 in order to train and test the proposed ANN models. To construct the training dataset, 100 samples for each loading case was acquired. Initially, depth images were acquired when no load was acting on the wing. Afterward, data were collected for six different load magnitudes acting on 21 distinct sections of the wing in sequence, which included the concentrated and distributed loading scenarios. Therefore, 12,700 samples were collected for the training dataset. As for the test dataset, the depth images were acquired for the duration of only one second, thus resulting in 30 samples for each loading case for this dataset. Unlike the training dataset, the six distinct loads on the 21 sections of the wing were applied at random in this case, so as to evaluate the robustness of the proposed approach. This resulted in the acquisition of 3810 samples for the test dataset. The details of the constructed training and test datasets are tabulated in Table 2.



The ( ) show the number of samples in the test dataset.

In this paper, TensorFlow [39] software was used to build and test the proposed ANN models. TensorFlow is an open source platform developed by Google Brain team, and it is widely used in literature to conduct machine learning based research due to its highly efficient computation framework. The computer used for developing the proposed ANN models had an Intel Xeon 3.6 GHz, twelve thread central processing unit (CPU) with 16 GB of RAM, and the whole network was trained on the CPU only, without the need of GPU.

#### *4.2. Experimental Results and Discussions*

In this section, the performance of the proposed framework for load localization and estimation are analyzed and discussed in detail. Initially, autoencoders were used to extract informative data from the depth images at a much smaller scale. subsequently, the proposed ANNs were trained using the training set described in Section 4.1, and their performance was evaluated and compared with a modified version of Castigliano's theorem (The readers can refer to the Appendix A for the details of this algorithm.) on the constructed test set.

#### 4.2.1. Load Localization From Depth Images Using ANN

An autoencoder was used to obtain informative data at a much smaller scale from the acquired depth images to be fed into the localization and estimation networks, as stated in Section 2.1. The autoencoder was run in series with the classification network so as to use its accuracy as a measure of performance in order to determine the smallest size of the encoded features required for load localization. The encoded feature size was initially set to be 400 and then was increased with increments of 200 until satisfactory load localization performance was obtained. As described in Section 3.1, two different loading scenarios were considered in this work. In the concentrated loading scenario, the output locations were labeled positions 1 to 8 and, in the distributed loading scenario, they were labeled positions 9 to 21. Moreover, 0 label was chosen to represent the no load condition. Therefore, the total number of distinct positions in the output layer amounts to 22. By making use of one hot encoding technique, the aforementioned 22 distinct positions were encoded using only nine

classifiers in the output of the localization network by setting *K* to 9 and *P* to 22. Labels 1 to 21 were one hot encoded using the first eight classifiers, and the no load case was encoded using the ninth classifier.

The proposed autoencoder and classification network was trained by varying the encoded data size. The activation function of the autoencoder was chosen to be a sigmoid and the number of neurons in the first and second hidden layers of the classification network were set to 60% and 30% of the encoded data size, respectively. The mean and standard deviation of the encoded data to be used in Equation (3) were chosen as 0.5 and 0.2, respectively. The *β* coefficient for regularization was chosen to be 0.1, and the dropout ratio of 0.8 was chosen in order to increase the network's generalization capabilities. Moreover, the starting learning rate was chosen to be 0.0005 and the network was trained using Adam optimizer. The training of the network was performed for 8000 iterations for each encoded data size, and the results are tabulated in Table 3. From these results, it is observed that, as the size of encoded data increases, the difference between the training and test accuracies decrease. This suggests that more distinctive data is being extracted as encoded data size increases. Moreover, an encoded data size of only 1200 was enough for obtaining very high accuracies of 96.4% and 94.3% when evaluated on the training and test datasets, respectively.

**Table 3.** Accuracy of the proposed localization ANN with varying encoded data size.


As for load localization that is based on the modified Castigliano's theorem, its performance was also evaluated on the constructed test dataset. The proposed localization ANN was trained with the load locations shown in Figure 14. Outputs of the proposed localization network with an encoded data size of 1200 and the one based on Castigliano's theorem are plotted against the ground truth positions in Figure 15. In this figure, the label denoted as zero represents the no load case, labels 1 to 8 represent the concentrated load positions, labels 9 to 21 represent the distributed load positions, and label 22 represents misclassified outputs that do not belong to any of the aforementioned load positions. From these results, one can see that both frameworks are able to discern the locations of the loads causing different kinds of bending and twisting deflections on the wing rather successfully. However, the accuracy of the proposed ANN based framework is superior to the one that is based on Castigliano's theorem, as visible from the results obtained in Figure 15 and the obtained accuracies given in Table 4. More importantly, the proposed neural network based method exhibits invariance to the type of the applied load and is able to successfully localize both concentrated and distributed loads causing wing deflections, which the framework based on modified Castigliano's theorem fails to do so properly.

**Table 4.** Localization performance of the proposed ANN and modified Castigliano's theorem, evaluated on test dataset.


**Figure 14.** Load positions used for training the proposed localization ANN.

**Figure 15.** Position predictions based on the proposed localization ANN and modified Castigliano's theorem, evaluated on the test dataset.

The relationship between the ground truth and the estimations of both frameworks is illustrated as a confusion matrix in order to get more insight into the localization performance of both methods. Figure 16 shows the confusion matrix for the proposed localization ANN, and Figure 17 is for the modified Castigliano's theorem. From these plots, it can be seen that the correct estimations are on the diagonal and any points not located on this line represent misclassified outputs. For the ANN framework, all of the position estimations have accuracies of higher than 90%, except for Positions 8 and 14. Nonetheless, their accuracies are still high, being 76.7% and 86.1%, respectively. As observed from Figure 16, Position 8, which is a concentrated load, is misclassified 16.1% of the time as Position 19, which is a distributed load. This observation is due to the wing deflection profiles under these loading conditions having similar patterns. However, these observations seem to be worse for the results obtained through the modified Castigliano's theorem, especially in the case of distributed loads, as seen in Figure 17. This is due to the fact that measured deflections exhibit very similar patterns at different points and, therefore, can not be accurately captured unless a more robust model such as the proposed ANN one is used to classify them in an appropriate manner.

**Figure 16.** Confusion matrix for localization based on the proposed ANN, evaluated on test dataset.

**Figure 17.** Confusion matrix for localization based on modified Castigliano's theorem, evaluated on test dataset.

#### 4.2.2. Load Estimation from Depth Images Using ANN

In this subsection, the proposed autoencoder and regression ANN's performance for load estimation is evaluated on the same dataset as the localization one. Unlike the localization networks' output, the output of this network is a single continuous variable that represents the magnitude of applied load under both concentrated and distributed loading conditions. Similar to the localization part, the proposed load estimation network was trained with varying encoded data size and the same parameters of the autoencoder network. As for the regularization coefficient *β* and dropout rates, they were fine tuned to be 0.1 and 0.35, respectively. This time the training was performed for 30,000 iterations for each encoded data size, and the obtained results are shown in Table 5. Based on the obtained results, it is seen that, again, an encoded data size of 1200 neurons was enough for obtaining very high accuracies of 97.3% and 92.7% when evaluated on the training and test datasets, respectively. Besides, it is observed that training such a network with good performance requires significantly more iterations when compared with the classification one, since the output in regression problems is a continuous variable.

**Table 5.** Accuracy of the proposed load estimation ANN with varying encoded data size.


Moreover, the performance of load estimation that is based on modified Castigliano's theorem was also evaluated on the constructed test dataset. The proposed estimation ANN was trained with the loads that are shown in Figure 18. The outputs of the proposed load estimation ANN with 1200 encoded data size and the modified Castigliano's theorem are plotted against the ground truth loads presented in Figure 19. These results show that magnitudes of loads causing bending and twisting deflections can be estimated with very high confidence, regardless of where the load is acting on the wing. However, the accuracy of the proposed ANN based framework is far more superior when compared with the one that is based on the modified Castigliano's theorem, as visible from the results obtained in Figure 19 and the obtained accuracies tabulated in Table 6.

**Figure 18.** Load magnitudes used for training the proposed estimation ANN.

**Figure 19.** Load estimation based on the proposed ANN and modified Castigliano's theorem, evaluated on test dataset.

**Table 6.** Load estimation performance of the proposed ANN and modified Castigliano's theorem, evaluated on the test dataset.


These results show that the proposed method can localize and estimate the loads acting on an aircraft wing with very high accuracies under both concentrated and distributed loading conditions. It should be noted that, since the proposed method was trained with elastic loading cases, it will only work with elastic loads. If the wing is damaged, then the relationship between the deflections and the load will change; therefore, the proposed method will not work unless it is trained for the damaged cases as well. Even though the training of the proposed networks is performed offline, the proposed method requires only 0.02 s for data reduction from a full frame image with 640 × 480 = 307,200 pixel features using an autencoder with an encoded data size of 1200. As for data reduction from the cropped images used in this work that had 247 × 166 = 41,002 pixel features, an autoencoder with an encoded data size of 1200 requires only 0.008 s. Another 0.001 second is required for localization or estimation of loads from the encoded data. Therefore, the proposed method can operate at 0.021 s or around 47 Hz for a single full frame image, thus realizing real time performance.

#### **5. Conclusions**

In this work, a robust structural health monitoring system based on depth imaging and artificial neural networks for localization and estimation of bending and twisting loads acting on an aircraft wing in real time is proposed. The proposed framework is based on the usage of depth images obtained from a depth camera as input features to an autoencoder and load location or magnitude as output labels of supervisory neural networks placed in series with the autoencoder. Initially, the Microsoft Kinect V1 depth sensor's accuracy and precision were evaluated for monitoring of aircraft wings by making use of ArUco markers and a Leica DISTO X310 laser meter having an accuracy of ±1 mm. The Kinect V1 proved to be reliable for SHM purposes, since it provided full field measurements with accuracy and standard deviation of 2.25 mm and 0.28 mm, respectively, when compared with the single point measurements provided by the laser meter.

As for the proposed method, first, an ANN consisting of an autoencoder and two hidden layer classification network with ReLU activation functions was proposed for estimating the location of loads. Second, an autoencoder and logistic regression network of two hidden layers with tanh and sigmoid activation functions was proposed for estimating the magnitude of these loads. Both of the proposed networks were trained and validated on an experimental setup, in which the application of concentrated and distributed loads were applied on a composite UAV wing.

In addition, a comparison with an approach based on Castigliano's theorem was performed, and the proposed method proved to have superior performance in terms of the localization and estimation of loads. The proposed localization and estimation ANNs achieved accuracies of 94.3% and 92.7%, while the framework based on Castigliano's theorem achieved average accuracies of 57.7% and 64.1%, respectively, when both of the methods were evaluated on a dataset containing randomly applied concentrated and distributed loads. As demonstrated, the proposed ANN based framework can localize and estimate the magnitudes of loads acting on aircraft wings with very high accuracies from a single depth sensor in realtime.

In the near future, it is planned to extend the current study for the localization and estimation of highly dynamic loads on larger aircraft wings.

**Author Contributions:** Conceptualization, D.K.B., M.U., M.Y., and B.K.; methodology, D.K.B. and M.U.; resources, M.U.; data curation, D.K.B. and M.U.; visualization, D.K.B. and M.U., writing—original draft preparation, D.K.B.; writing—review and editing, M.U., M.Y., and B.K.; supervision, M.U. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no funding.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A. Localization and Estimation of Loads Based on Castigliano's Theorem**

Based on Castigliano's second theorem [40,41], for a linearly elastic structure under loads *P*1, *P*2, *P*3,..., *Pm* at points 1, 2, 3, . . . , *n*, the deflection *Wj* of point *j* can be calculated as follows:

$$\mathcal{W}\_{\bar{\jmath}} = \sum\_{\mathfrak{m}} a\_{\mathfrak{m}\times\mathfrak{m}} P\_{\mathfrak{m}} \tag{A1}$$

where *an*×*<sup>m</sup>* is the deflection at point *n* due to a unit load applied at point *m*, normalized by the magnitude of the load. Based on this theorem, given loads at *m* points, one can calculate deflections at *n* points, as given in matrix form in Equation (A2). In order to construct *A* matrix, the concentrated loads mentioned in Section 2.1 were placed at eight different positions of the wing and the corresponding deflections *D* were recorded. Afterward, these measured deflections were normalized by the magnitude of the loads (*L*) acting on them to calculate the *A* matrix as given in Equation (A3). Moreover, this was performed for all load magnitudes i.e, [2.45, 4.9, 7.35, 9.81, 12.26, 14.71] N. Afterward, the average of all these magnitudes was calculated to be used as the final *A* matrix due to the nonlinear nature of the wing material as shown in Figure A1 and the inevitable sensor noise.

$$\mathcal{W} = AP \tag{A2}$$

$$A = \frac{D^T}{L} \tag{A3}$$

where *<sup>W</sup>* ∈ *n*, *<sup>A</sup>* ∈ *nm*, *<sup>P</sup>* ∈ *m*, *<sup>D</sup>* ∈ *mn*, and *<sup>L</sup>* ∈ .

**Figure A1.** Deflection measurements for loads acting at point 2 of the wing.

However, the objective of this work is the inverse of the above mentioned problem i.e., given deflections at *n* points, we want to calculate loads at *m* points. To do that, one can calculate the pseudo inverse of *A* to obtain *K*. The calculated *K* is shown in Table A1. This is valid since Castigliano's first theorem is basically the inverse of its second theorem. Therefore, loads can be estimated by using the following equation:

$$P = \mathbb{KW} \tag{A4}$$

$$
\begin{bmatrix} P\_1 \\ \vdots \\ P\_8 \end{bmatrix} = \begin{bmatrix} 4244.729867\dots - 1615.667686 \\ \vdots \ddots \vdots \\ -2459.317023\dots 1233.037717 \end{bmatrix} \begin{bmatrix} W\_1 \\ \vdots \\ W\_8 \end{bmatrix} \tag{A5}
$$

where *K* = *A*−<sup>1</sup> is the unit load at point *m* due to the deflection measured at point *n*, normalized by the magnitude of the deflection.


**Table A1.** Calculated *K* matrix.

After calculation of *K* matrix, one can now estimate loads at different points of the wing by providing the deflections measured at various points of the wing. Below are the results for the cases when the load is acting at point 2 of the wing with varying magnitudes of [2.45, 4.9, 7.35, 9.81, 12.26, 14.71] N, and the loads at all of the 8 points were calculated using Equations (A4) and (A5), for which the results are plotted in Figure A2.

**Figure A2.** Load estimations for concentrated loads acting at point 2 of the wing.

#### *Appendix A.1. Load Localization Based on Castigliano's Theorem*

From the above results, it is seen that the load estimation fails as it is since it is estimating that loads with different magnitudes at different points of the wing are responsible for the measured deflections. In order to overcome this shortcoming, this work proposes the loads to be first localized, and then their magnitudes can be calculated. In order to localize the loads, the measured deflections at any time are compared with the constructed deflection *D* matrix and then the Pearson correlation[42] is used for finding the maximum correlation between them as follows:

• Given a new measurement as the one in Table A2.

**Table A2.** New deflection measurements (mm) at 8 points of the UAV wing.


• Compare the new measurements with the constructed deflection *D* matrix given in Table A3.


**Table A3.** Constructed deflection (*D*) matrix.

• Find the maximum Pearson correlation coefficient between the current measurement given in Table A2 and the constructed deflection matrix *D* given in Table A3, which in turn will correspond to the actual load position as shown in Table A4.


**Table A4.** Calculated Pearson correlation coefficients.

From these results, it is seen that the load can be successfully localized using this method. Besides its obvious usage for load localization, this information is vital for the correct estimation of load magnitudes based on the proposed methodology for load estimation based on Castigliano's theorem as formulated in the following subsection.

#### *Appendix A.2. Correction of Load Estimation Based on Castigliano's Theorem via Localization and Optimization*

Now that the loads are localized, one can correct the estimated loads by using the localization information. This can be done by first eliminating all the loads other than the localized one. Second, since the theory estimates that the measured deflections are due to a combination of loads acting at different points, one can conclude that the actual load applied at a single point is the cumulative sum of all the loads acting on the wing. Therefore, once the load is localized, its magnitude will be equal to the sum of all the estimated loads. The effectiveness of this can be seen in the results obtained in Figure A3 for loads acting again at point 2 of the wing.

**Figure A3.** Corrected load estimations for concentrated loads acting at point 2 of the wing.

However, as seen from the obtained results in Figure A3, the estimated loads' accuracy is low in general. This is due to the calculated *K* matrix. Since the *K* matrix was calculated through pseudo inverse, it may not provide the optimal relationship between the measured deflections and the applied loads. Sensor noise also affects this, since the *A* matrix was constructed using measurements acquired at different loading cases. Therefore, this work proposes to optimize the calculated *K* matrix using a backpropagation algorithm such as Adam [20] optimizer. The initial solution to the optimizer was provided by the calculated pseudo inverse, and the optimization was defined as follows:

$$
\mathcal{F} = \mathcal{K}V\tag{A6}
$$

$$\mathcal{C}F = ||(F - \mathcal{F})||\_2 \tag{A7}$$

where *<sup>K</sup>*<sup>ˆ</sup> ∈ *mn* is the matrix to be optimized, *<sup>V</sup>* ∈ *nN* is the deflection matrix with *<sup>N</sup>* samples measured by the depth sensor, *<sup>F</sup>*<sup>ˆ</sup> ∈ *mN* is the estimated load matrix for *<sup>N</sup>* samples, *<sup>F</sup>* ∈ *mN* is the ground truth load matrix obtained from the applied loads' magnitudes and *CF* is the cost function defined to be the Frobenius norm of the matrix.

The optimized *K*ˆ is shown in Table A5. Using this optimized matrix, the loads can now be estimated quite accurately, as shown in Figure A4. The same procedure can be applied to distributed loads by considering them as new load locations and constructing a 21 × 21 deflection matrix. A sample result obtained for distributed loads is shown in Figure A5, in which the deflection matrix was extended to include the new 13 distributed points.


**Table A5.** Optimized *K*ˆ matrix.

**Figure A4.** Optimized and corrected load estimations for concentrated loads acting at point 2 of the wing.

**Figure A5.** Optimized and corrected load estimations for distributed loads acting at point 9 of the wing.
