*Article* **Camera-Based in-Process Quality Measurement of Hairpin Welding**

**Julia Hartung 1,2,\*, Andreas Jahn 1, Oliver Bocksrocker <sup>3</sup> and Michael Heizmann <sup>2</sup>**


**Abstract:** The technology of hairpin welding, which is frequently used in the automotive industry, entails high-quality requirements in the welding process. It can be difficult to trace the defect back to the affected weld if a non-functioning stator is detected during the final inspection. Often, a visual assessment of a cooled weld seam does not provide any information about its strength. However, based on the behavior during welding, especially about spattering, conclusions can be made about the quality of the weld. In addition, spatter on the component can have serious consequences. In this paper, we present in-process monitoring of laser-based hairpin welding. Using an in-process image analyzed by a neural network, we present a spatter detection method that allows conclusions to be drawn about the quality of the weld. In this way, faults caused by spattering can be detected at an early stage and the affected components sorted out. The implementation is based on a small data set and under consideration of a fast process time on hardware with limited computing power. With a network architecture that uses dilated convolutions, we obtain a large receptive field and can therefore consider feature interrelation in the image. As a result, we obtain a pixel-wise classifier, which allows us to infer the spatter areas directly on the production lines.

**Keywords:** hairpin; laser welding; semantic segmentation; dilated convolution; sdu-net; spatter detection; quality assurance; fast prediction time

#### **1. Introduction**

In the production of electric motors, the automotive industry relies on a new technique known as hairpin. Instead of twisted copper coils, single copper pins that are bent like hairpins are used, which give the technology its name. These copper pins are inserted into the sheet metal stacks of a stator and afterward welded together in pairs. As with conventional winding, the result is a coil that generates the necessary magnetic field for the electric motor. This method replaces complex bending operations and enables a more compact motor design while saving copper material [1,2]. Depending on the motor design, between 160 and 220 pairs of hairpins per stator are welded. If at least one weld is defective, the entire component may be rejected. Therefore, continuous quality control is necessary and the weld of every hairpin pair should be monitored [1].

In most cases, a laser is used to weld the pins. The laser welding process enables a very specific and focused energy input, which ensures that the insulation layer is not damaged during the process. In addition, unlike electron beam welding, no vacuum is required and laser welding is a flexible process that can easily be automated in a short cycle time [1]. The lower-cost laser sources that are scalable in the power range emit in the infrared wavelength range, which is comparatively difficult for working with copper. At this wavelength of about 1030 nm or 1070 nm copper, is highly reflective at room temperature, so very little incoming laser light is absorbed [1,3]. Just before reaching the melting temperature the absorption level rises from 5% to 15% and reaches almost 100%

**Citation:** Hartung, J.; Jahn, A.; Bocksrocker, O.; Heizmann, M. Camera-Based in-Process Quality Measurement of Hairpin Welding. *Appl. Sci.* **2021**, *11*, 10375. https:// doi.org/10.3390/app112110375

Academic Editor: Andreas Fischer

Received: 13 September 2021 Accepted: 19 October 2021 Published: 4 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

when the so-called keyhole is formed. Based on this dynamic, the process is prone to defects and spattering [4]. A spatter occurs when the keyhole closes briefly and the steam pressure causes the material to leak out of the keyhole. If the ejected material gets into the stator, it may cause short circuits or other defects [1]. In addition, less material will be used to form the weld, which often leads to a loss of stability. For these reasons, it is extremely important to prevent spatter as much as possible. Various processes can improve the welding result on copper. Three approaches are briefly touched upon below. By moving the laser spot fast and simultaneously during forwarding motion (wobbling), stable dynamics can be created in the weld pool. This can improve the process quality when welding with an infrared laser. Another approach is welding with different strengths of inner and outer fiber core. This means that the inner fiber core is used to create the desired welding depth with high intensity, while the molten pool is stabilized by an outer fiber core—the fiber ring. In addition, there is the possibility of using a visible wavelength of a green laser, which results in higher absorption of the laser light and thus higher process reliability [5–7]. Furthermore, there may also be external causes that lead to spattering. These include, for example, contamination, gaps, misalignment or an oxidized surface.

The correct setting of the laser welding parameters such as laser power, speed and focus size is very important in copper welding. In addition, the process may not drift or this must be detected at an early stage. The presence of spatter on the component can be used as an indicator of an unstable situation in the welding process, as its occurrence is closely related to the quality of the weld seam [8,9]. Due to the briefly mentioned reasons, it is essential to monitor the welding process while focusing particularly on spattering. This allows a conclusion about the quality of individual welds, the occurrence of defects, as well as the overall quality of the stator. An important requirement is also fast process time which is a prerequisite for a system to be used in large-scale production. The welding of an entire engine takes just a bit more than one minute and quality monitoring should not slow down the process [1,2].

Currently, there are only a few machine learning applications that are used for quality assessment in laser welding [10]. Some approaches are presented by Mayr et al. [11], including an approach for posterior quality assessment based on images using a convolutional neural network (CNN). They use three images in the front, back, and top view of a hairpin, to detect quality deviations [12]. In [13] a weld seam defect classification with help of a CNN is shown. They achieve a validation accuracy of 95.85% in classifying images of four different weld defects, demonstrating the suitability of CNNs for defect detection. Nevertheless, some defects cannot be seen visually on the cooled weld seam. For example, pores in the weld seam or a weld bead that is too small due to material ejection can not be visually distinguished from a good weld seam.

That is why imaging during the process of hairpin welding offers more far-reaching potential for machine learning than subsequent image-based inspection of the weld. Important criteria are the mechanical strength of the pin and the material loss [12]. Both criteria are in correlation with a stable welding process and the occurrence of spatter. For spatter detection, a downstream visual inspection of the component is also possible [14]. However, this approach is problematic for hairpins since there is little material around the hairpins that can be verified for spattering.

Therefore, this paper presents an approach that enables spatter detection during hairpin welding. One of the main challenges of spatter detection directly during the welding process is the fast execution time on hardware with low computing power. The algorithm should be executed directly in the production line, where the installed hardware is often fanless and only passively cooled due to ingress protection. Another important issue is the amount of training data. Since this is an application in an industrial environment, training should only be done on a small data set so that the labeling effort is low and the algorithm can be quickly adapted to new processes. These two aspects are considered in the following.

In Section 2 the data basis and the analysis methods are presented in detail. On the one hand, the network architecture is discussed, but also comparative algorithms, such as morphological filters with their configurations, are presented. Subsequently, in the result section, the training parameters and the results are shown. Finally, the results are discussed and summarized in Section 4.

#### **2. Materials and Methods**

To obtain a comprehensive data basis, images were recorded from two different perspectives while welding hairpins. The captured images were then used to perform different approaches to monitor the occurrence of spatter. An automated solution for spatter detection is recommended because it ensures consistent evaluation and only in this automated way can spatter be detected reliably. The image-based approach contributes to the detection of spatter caused by external factors. By using artificial intelligence, a high feature variance in the data is covered. Additionally, the required properties of the categories do not need to be defined in detail. Images were continuously recorded in the process to implement in-process quality monitoring. Since a visual inspection of the hairpins after the welding process is not always clear, a post-weld inspection is not precise enough.

#### *2.1. Data Basis*

We use three different approaches for generating data sets for the observation of the welding process using an industrial camera. These differ firstly in the perspective from which the data were recorded and secondly in the type of preprocessing.

In the first approach, the images taken during the welding process were summed up to a single image per process. The superimposition of the individual images provides information about the spatter occurrence during the entire process [15] and can thus be used to evaluate the welding behavior. When summing up the images, those taken at the time the laser beam pierced the material must be removed. In this process step, a white glow occurs, which takes up so much space on the image, that it would hide possible spatters. An example is shown in Figure 1.

**Figure 1.** Summed image with and without piercing.

On the one hand, the images were taken with a high-speed camera mounted on the side of the welding process. This provides a lateral view in which the welding process and spatter are visible. The data set containing the summed images of a welding process is referred to as *laterally complete* in the following.

The second view of the images is coaxial through the laser optics. Often, a camera is already installed in a laser welding system that captures images through laser optics. These images are used for example to determine the component position before the welding process. To achieve more accurate results in this process, the magnification optics are often selected to provide a good imaging ratio for the component. In contrast, when monitoring the welding process for spatter, the lowest possible magnification would be optimal. Since the camera is usually part of an existing system, the magnification cannot be adjusted specifically, which often means that a higher magnification must be used. For our test setup, we use a gray-scale camera with a recording frequency of 2 kHz. The summed images, which were generated based on the coaxial view, are called *coaxial complete*.

For a third approach, we evaluate the individual images acquired by the coaxial view. The third data set is called *coaxial single*. While spatters are shown as lines on the summed image, they are usually visible as dots on the single images, depending on the exposure time.

With semantic segmentation, labeling is time-consuming and error-prone. Many developments in the industry are carried out specifically for a customer project on customer data. The labeling effort, which means time resources and therefore costs, is recurring for each customer project. In addition, there are often confidentiality agreements and data flow for a large database is difficult. Therefore, especially in the industry, an attempt should be made to work with a small amount of training data. To teach the network the desired invariance and robustness properties even in training with only a few training data sets, data augmentation is essential [16]. Various network architectures, such as the U-Net architecture, are designed for strong data augmentation. We enlarge our training data set using rotation, vertical and horizontal shift, vertical and horizontal flip, adjustment of the brightness range, zoom and shear. Through the strong use of data augmentation we also have the advantage of avoiding overfitting. Dosovitskiy et al. [17] have shown in this context the importance of data augmentation to learn a certain invariance. Despite a small data set, the network never gets the same image with an identical setting presented multiple times. Thus, it cannot learn the image by memory.

The segmentation masks could not be exact for each pixel. Especially in the summed images where the spatter is shown as a line, the labels are not accurate for each pixel. As shown by Tabernik et al. [18] inaccurate labeling is sufficient for predictions aimed at a quality assurance based on defect detection. The paper processes the topic of surface-defect detection. They used a segmentation-based approach to detect cracks on an image and transferred the problem afterwards to binary image classification with the classes *defect is present* and *defect is not present*. In an experiment, it was shown that the segmentation results are better if larger areas around the crack are marked in the annotation mask. Although pixels that are not part of the crack are marked as defect class, they achieve better results than with an accurate annotation mask. Our use case is very similar to the use case of Tabernik et al., which is why their result can be applied to our task. Instead of surface defects, we detect spatter in our application.

We train separate models, one for each data generation approach. For the two models based on the summed images, we use 14 images each in the training process. Each of these images represents the complete welding process, whereas with the *coaxial single* data set we have about 700 images per welding process. There, we use 500 images from different processes for training. We categorize the pixel-wise class assignment into the background, process lights, and spatter.

#### *2.2. Network Architecture*

The images are evaluated using a neural network. We use a segmentation network to localize the process light and the spatter pixel by pixel. Compared to the object detection, which could be done for example with YOLO [19] or SSD [20], this has the advantage that through the pixel-based loss function each pixel can be considered as an individual training instance [18]. This increases the effective number of training data massively and thus also counteracts overfitting during training.

Unlike many other network architectures, the U-Net architecture is very well suited for small data sets, as shown by Ronneberger et al. [16]. Therefore a neural network whose architecture is based on the stacked dilated U-Net (SDU-Net) architecture presented by Wang et al. [21] is used to evaluate our images. The abbreviation SD is short for stacked dilated. To avoid confusion considering the SDU-Net, it should be mentioned that various other U-Net modifications exist, which are also called SDU-Net. However, in these nets the abbreviation SD describes other concepts. For example, there is the spherical deformable U-Net (SDU-Net) from Zhao et al., which was developed for medical imaging in the inherent spherical space [22]. Because there is no consistent neighborhood definition in the cortical surface data, they developed another type of convolution and pooling operations, especially for this data. Another U-net modification, which is also called SDU-Net, is the structured dropout U-Net presented by Guo et al. [23]. Instead of the traditional dropout for convolutional layers, they propose a structured dropout to regularize the U-Net. Gadosey et al. present the stripping down U-Net, with the same abbreviation, for segmentation images on a platform with low computational budgets. By use of depth-wise separable convolutions, they design a lightweight deep convolutional neural network architecture inspired by the U-Net model [24].

As mentioned before, we used a SDU-Net modification with stack dilated convolutional layers. This U-Net variant adopts the architecture of the vanilla U-Net but uses stacked dilated convolutions. Instead of using two standard convolutions in each encoding and decoding operation, the SDU-Net uses one standard convolution followed by multiple dilated convolutions which are concatenated as input for the next operation. Thus the SDU-Net is deeper than a comparable U-Net architecture and has a larger receptive field [21]. In Figure 2 our architecture is shown in detail. We used a gray-scale image of the size 256 × 256 pixels as input. In designing the network architecture we set the concatenate output channel numbers to *<sup>n</sup>*<sup>1</sup> = 16, *<sup>n</sup>*<sup>2</sup> = 32, *<sup>n</sup>*<sup>3</sup> = 64 and *<sup>n</sup>*<sup>4</sup> = 128 instead of *<sup>n</sup>*<sup>1</sup> = 64, *<sup>n</sup>*<sup>2</sup> = 128, *<sup>n</sup>*<sup>3</sup> = 256 and *<sup>n</sup>*<sup>4</sup> = 512 like the original implementation of the paper. Since our images are far less complex compared to the medical images used in the paper from Wang et al., this number is sufficient. So all in all we have 162,474 trainable parameters instead of 6,028,833. With this comparatively small amount, we achieve a fast inference time, which is about 20 ms on CPU. This is important to be able to run the network prediction on an industrial computer directly at the production line, where often no GPU is available.

**Figure 2.** Our optimized small SDU-Net architecture. The input size is 256 × 256 pixels and we have 32 × 32 pixels at the lowest resolution. The number of channels is denoted at the bottom of the boxes.

#### *2.3. Evaluation*

An alternative approach is to implement spatter detection with a morphological filter. We choose the opening operation, which involves an erosion of the data set *I* followed by dilation, both with the same structural element *H*:

$$I \diamond H = (I \ominus H) \oplus H. \tag{1}$$

In the first step, the erosion, the opening process eliminates all foreground structures that are smaller than the defined structural element. Subsequently, the remaining structures are smoothed by the dilation and thus grow back to approximately their original size. With the opening filter, we identify the process light in the images based on the structure element *H*. In Figure 3 the process is visualized. An input image is shown in Figure 3a and the corresponding filtered image in Figure 3b. Afterwards, we define the spatters by subtracting the filtered image from the original image. The remaining image elements represent the spatters, shown in Figure 3c. Figure 3d shows an overlaid image, where the process light is painted in green and the spatters in red. For this algorithm we use the original image size with 480 × 640 pixels for the coaxial images and 840 × 640 pixels for the lateral images. We choose the structure elements as ellipse with *<sup>H</sup>* <sup>=</sup> <sup>45</sup> <sup>×</sup> 45 pixels for the *coaxial single*, *<sup>H</sup>* <sup>=</sup> <sup>90</sup> <sup>×</sup> 90 pixels for *coaxial complete*, and *<sup>H</sup>* <sup>=</sup> <sup>60</sup> <sup>×</sup> 60 pixels for *lateral complete* images. The definition of the structural element is based on the average size of the process light, which is estimated in the images. The spatters usually represent smaller elements and can thus be distinguished from the elements found by the filter.

**Figure 3.** Gray-scale opening, (**a**) gray-scale image (**b**) process light (**c**) spatter (**d**) overlaid image with spatters colored in red and the process light in green.

We have also used other network models, such as a comparable small version of the U-Net model according to Ronneberger et al. [16]. We trained this model equivalent to the SDU-Net architecture with the same input images and the same parameters, shown in the next section.

#### **3. Results**

We trained different models of the small SDU-Net architecture for each input data generation approach, *coaxial single*, *coaxial complete*, and *lateral complete*. All models were trained with a batch size of 6, an input of gray-scale images in the size of 256 × 256 pixels, and 500 steps per epoch. We used the Adam Optimizer and started the training process with a learning rate of 0.001. The learning rate was reduced by 5% after 3 epochs without any improvement until a learning rate of 0.000005 is reached. The training process was stopped when no further improvement has occurred in 20 consecutive epochs. This results in different long training times for the different models. The loss value and the accuracy of the different models can be seen in Table 1. To verify the results during training, we used validation data sets. These contained 3 images each for *coaxial complete* and *lateral complete* and 18 images for *coaxial single*, according to the small database. The validation data sets were also enlarged with strong use of data augmentation. After the training, we

used a separate test data set, each containing 50 images with the corresponding ground truth image.

Because the number of pixels per class is very unbalanced, and especially the less important background class contains the most pixels, we used the loss functions weighted dice coefficient loss (DL) and the categorical focal loss (FL) [25]. The network results are shown in Figure 4 and in Table 2.

The advantage of focal loss is that no class weights have to be defined. The loss function, which is a dynamically scaled cross entropy (CE) loss, down-weights the contribution of easy examples and focuses on learning hard examples:

$$FL(p\_t) = -\mathfrak{a}(1 - p\_t)^\gamma \log(p\_t) \tag{2}$$

The two parameters *α* and *γ* have to be defined. The parameter *α* represents the balancing factor, while *γ* is the focusing parameter. The CE loss is multiplied by the factor (<sup>1</sup> <sup>−</sup> *pt*)*γ*. This means that with the value *<sup>γ</sup>* <sup>=</sup> 2 and a prediction probability of 0.9, the multiplier would be 0.12, i.e., 0.01, making the FL in this case 100 times lower than the comparable CE loss. With a prediction probability of 0.968, the multiplier would be 0.001, making the FL already 1000 times lower. This gives less weight to the easier examples and creates a focus on the misclassified data sets. With *γ* = 0 the FL works analogously to the cross entropy. Here the values *α* = 0.25 and *γ* = 2 were chosen.

For comparison, we used the weighted dice coefficient loss, where the loss value is calculated for each class, weighted with the respective class weighting, and then added up. The class weights were calculated based on the pixel ratio of the respective class in the training images. The classes that contain only a few pixel values, such as the spatter class, must be weighted more heavily so they are considered appropriately during training. Since the values are calculated based on the number of pixels in the training data, these weights vary between the different input data sets.

Besides training individual models for each input data approach, we also trained one model for the prediction of all data, the coaxial and lateral view, summed, and single images. Since the different data sets have the same classes and a similar appearance, one model approach is also possible. The advantage of this approach is that a higher variance of data can be covered in one model and therefore we do not need to define new models or parameters for each data type. To train the global model we used 14 images of each *coaxial* and *lateral complete* data set and 34 *coaxial single* images.

Another advantage is that additional classes can be added to the model. We have introduced a new class, which includes the cooling process of the welding bead. From the moment the process light turns off, the weld is assigned to the cool-down class. This class cannot be identified via the previously described structure recognition with the subsequent exclusion procedure using the morphological filter. Only image elements of different sizes can be detected and distinguished from each other. For elements of similar size with different properties, the method reaches its limits.

The result of training the small SDU-Net as a single model for all data is also shown in Table 1. All four classes were considered in the training process. In the summed images, the cooling process is not visible. For comparison, a SDU-Net model with twice the number of filters was trained. This net has more trainable parameters, but in our test, no significantly better results in loss, accuracy as well as in evaluation could be obtained. In addition, the results of the comparatively small U-Net model are shown.

The classification results are compared using the Intersection over Union metric (*IoU*). The metric compares the similarity between the predicted classification mask and a drawn annotation mask as ground truth by dividing the size of the intersection by the size of the union:

$$\operatorname{Iol} I(A, B) = \frac{A \cap B}{A \cup B}. \tag{3}$$

**Coaxial Single Coaxial Complete Lateral Complete All Data SDU-Net Double Filter Size, All Data U-Net, All Data** Steps per epoch 500 Batch size 6 Input size [pixels] 256 × 256 DL 0.173 0.196 0.198 0.156 0.146 0.163 Accuracy 0.993 0.966 0.995 0.980 0.980 0.975 Validation DL 0.488 0.347 0.489 0.361 0.339 0.428 Validation Accuracy 0.976 0.831 0.953 0.963 0.963 0.951

**Table 1.** Training results of the neural network models based on the SDU-Net architecture. Three different models are trained for each input data generation approach. The fourth model was trained for the prediction of all data. In comparison the results of an SDU-Net architecture with twice the number of filters and of the small U-Net are shown.

> In Table 2 the evaluation results of the different approaches are shown. The second value in the rows shows the IoU for all pixels in the entire image. The dark area around the weld is most correctly classified as background. Especially for the coaxial single dataset, the background takes up the main part of the image, so the IoU over the entire image is very high for all methods. Therefore, the specific class pixels are again considered separately. This consideration can be seen in the first values in the table. This represents the IoU based on the pixels assigned to a specific class, except for the background class. This value gives more information about the actual result than the total IoU. However, the larger the area of the background, the fewer pixels are included in the calculation of the class-specific IoU. As result, the value is more influenced by individual misclassified pixels.

> In Figures 4 and 5 the first value, without consideration of the background pixels, is used. As shown in Figure 4 the two weighted loss functions, FL and DL, result in comparable distributions in which the DL performs only marginally better.

> Using a single model trained on all three data sets, an outlier with IoU close to 0 can be seen in each of our test sets in Figure 4. There, a shot during the cooling process with spatter was misclassified as process light. When using different models per data set, this error case did not occur. On the one hand, the error can be attributed to an underrepresentation of the cooling class in the overall data set, since this only occurs in the *coaxial single* images. On the other hand, the occurrence of spatter on images at this point in the welding process is very rare, which is why the case was not sufficiently present in the training. In productive use, it is assumed that the data is taken only from one perspective. Nevertheless, this experiment can show that the model generalizes well even on different input data with only very few training data and thus covers a high data variance.

> Figure 5 shows the IoU without considering the background pixels of the different data sets all trained with the dice coefficient loss. This graph shows that the largest deviations are contained in the *coaxial single* data set. In these images, the background occupies the largest image area, which makes small deviations of the other classes more significant, as shown in Table 3.

> In all three input data sets, *coaxial single*, *coaxial complete*, and *lateral complete*, the SDU-Net provides the best results compared to the other methods. The disadvantages of the U-net architecture arise from the fact that only simple and small receptive fields are used, which leads to a loss of information about the image context. In our use case, it leads to the fact that the classes cannot always be clearly distinguished from each other. The SDU-Net processes feature maps of each resolution using multiple dilated convolutions successively and concatenates all the convolution outputs as input to the next resolution. This increases the receptive field and both, smaller and larger receptive fields are considered in the result.


**Figure 4.** Comparison of the IoU of the classes process light, cool down, and spatter for the different loss functions dice coefficient loss and focal loss, as well as the approach with three different models and one model for all data.

**Figure 5.** Comparison of the IoU of the different input data sets. All three models are trained with the weighted dice coefficient loss.

Visualized results of the different methods are shown in Table 3. In comparison to the small SDU-Net, results of small U-Net, binary opening, and gray-scale opening are shown. The models of the SDU-Net and the U-Net are trained on all data and with four classes, while the morphological filter on the one hand requires a structural element of different sizes per data set and also cannot distinguish between process light and cooling process. For better visualization, the pixel-by-pixel classification of the neural networks is displayed in different colors and superimposed on the input image. The class of the process light is shown in green color, the spatters in red, and the cooling process in blue. The resulting images of the morphological filter are displayed analogously to Figure 3.

With the morphological filtering, small regions always remain at the edge of the area of the process light, since the structural element can never fill the shape exactly. As a result, the exclusion method would always recognize some pixels as spatter, which must be filtered in post-processing. Even small reflections, which occur mainly in the lateral images, are detected as spatter by the exclusion procedure. On the other hand, spatter that is larger than the defined structural element is detected as process light and not as spatter, which also leads to a wrong result. Compared to the binary opening, the steam generated during welding, which is mainly visible on the lateral images, is usually detected as process light in the gray-scale opening. With the binary opening, the steam area is usually already eliminated during binarization.

Without runtime optimizers, our classification time of the small SDU-Net model on the CPU is about 20 ms. In comparison, the binary and gray-scale opening reached 12 ms for coaxial single, 1.61 ms for coaxial complete, and 40 ms for the lateral complete images. The deviating process times of the opening operation are caused by the different image sizes and the different sizes of the structural element. By using a larger or differently shaped structural element, the process times can be further improved, but the resulting detection quality suffers.

The production time of a stator is about one minute for all 160 to 220 pairs of hairpins. In the best case, 270 ms are needed for welding one hairpin. The quality assurance with the SDU-Net needs 20 ms which is not even 10% of the welding time and 0.33 per mill of the whole welding process. With a time-delayed evaluation of the images, the previous pin can be evaluated each time the next pin is welded. Thus the time sequence of the welding of a stator remains unaffected by this setup due to the fast prediction time. The evaluation was deliberately calculated on the CPU since the model is to be executed directly at the production plant on an industrial PC, where GPUs are not always available. Thus, a strong spatter formation, which can indicate a drifting process or contaminated material, can be reported directly to the user. This allows the user to react directly and stop or adjust the process.

**Table 3.** Overlaid result images of the different methods. The green color marks the process light, red the spatters, and blue the cooling process. Average IoU value of the pixel which was assigned to process light, spatters, or cooling process.



**Table 3.** *Cont.*

#### **4. Discussion and Outlook**

By training one model on all three input data sets, it could be shown that a high variance of data can be covered with this approach. The data varies greatly in terms of both recording position and spatter optics. This high level of data variance will not occur in a production line, where a position of data acquisition, as well as a type of pre-processing, will be determined. However, this experiment suggests that it will be possible to use one model for different applications and with slightly different recording mechanisms. We obtain an average IoU of the specific classes without background class of 0.759 in this approach. In comparison, the IoU values of the morphological filter are 0.544 and 0.542, even though these methods were parameterized specifically for the particular data set. This generalization opens the possibility of using one model for different optics without having to make adjustments. With the execution time of 20 ms we are also in a similar range as the execution times of the morphological filter. This requires 12 ms, 1.61 ms, or 40 ms depending on the input data. For some input data, especially *coaxial complete* with 1.6 ms, this process takes less time, but for other input data it takes even longer.

The spatter prediction works on the summed images as well as on the single images. These show the average IoU value for the specific class with 0.821 for *coaxial single*, compared to 0.810 for *coaxial complete*. To record single images, where every spatter is shown punctually, a high recording frequency is required. Cheaper hardware usually has a lower capture frequency, which means that individual spatters would be missed. To counteract this, the exposure time can be increased so that the spatters are visible as lines on the images, similar to the cumulative images we used. The tests showed that even in this application, the spatter can be detected in the image and thus cheaper hardware can be used for quality monitoring.

By using a segmentation approach and a model architecture which works well with strong data augmentation, it is possible to work with very small training data sets. This makes the labeling effort for new processes, e.g., to new customer data, manageable, and thereby saves time and costs. By using a small network architecture with few parameters, both the training time as well as the prediction time are short. Thanks to the short prediction time, the application can be run directly on the production line on a conventional industrial computer. By analyzing the data during production, it is possible to react interactively, which is more efficient than a completely downstream analysis. The algorithm can also be continuously optimized by feeding new data into the neural network under defined monitoring conditions and then training it further. Further knowledge can be

generated through the proper application of data feedback. However, in this application, it is important to ensure that the application is not retrained by a drifting process. In addition, an online learning approach for the laser parameters would also be conceivable. The algorithm can be used to check whether spatter occurs with a certain configuration and thus readjust the laser settings.

The data can be recorded coaxially through the laser optics or laterally to the welding process. Spatter detection works well with both recording methods. In the average IoU of the process light and the spatter class, we achieve 0.850 for the coaxial view, while we only achieve 0.688 for the lateral images. It should be noted that the input images in both cases look very different and the relevant image area has different sizes. In the lateral images, a larger area is covered. In both cases, care must be taken to ensure that the distance to the weld seam is large enough to ensure that the spatter is still within the camera's field of view. The coaxial camera setup is often already available on production lines and can therefore be integrated more easily. In this case, the spatter detection could be upgraded in a production line with the help of a software update.

When considering the entire welding process, welding monitoring with a focus on spatter can be seen as just one part of a desired automated 100% inspection of the welding result. This step could be integrated into a three-stage quality monitoring system: in the first step, a deviating position of the hairpins can be detected in the process preparation and thus the welding position can be corrected. In addition, it makes sense to integrate a check of the presence of both hairpins and their correct parallel position. In the second step, spatter monitoring can be carried out directly in the process. This provides information on whether the welding process is unstable and enables rapid response. In the third step, subsequent quality control of the welding results can be carried out. Due to the in-process monitoring, random samples are sufficient in this step.

However, if 100% monitoring for spatter occurrence is to be implemented, additional hardware is required. As mentioned before, an industrial camera installed at production lines usually does not have such a high frame rate that images can be recorded without short times were spatters can be missed. This can be counteracted with the help of an extended exposure time and a larger field of view in which the spatter can be detected, but a 100% view during the process is unrealistic. In this case, an event-based camera or other sensor technology would have to be used. The approach presented in this paper focuses on quick and easy integration into an existing production system without the need for investment in additional hardware. This is often very costly and can lead to additional calibration effort.

Further on, the presented approach can be extended by an additional consideration of laser parameters or other sensor technology, which is already installed in the system. With the help of the information fusion in which the camera-based in-process monitoring for spatter is integrated, it is also possible to control the process even more comprehensively with existing hardware.

**Author Contributions:** Conceptualization, O.B.; methodology, J.H., A.J. and M.H.; software, J.H.; validation, J.H., A.J. and M.H.; formal analysis, J.H.; data curation, O.B., J.H. and A.J.; writing original draft preparation, J.H.; writing—review and editing, all authors. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


**Andreas Tausendfreund 1,\*, Dirk Stöbener 1,2 and Andreas Fischer 1,2**


**Abstract:** In the concept of the process signature, the relationship between a material load and the modification remaining in the workpiece is used to better understand and optimize manufacturing processes. The basic prerequisite for this is to be able to measure the loads occurring during the machining process in the form of mechanical deformations. Speckle photography is suitable for this in-process measurement task and is already used in a variety of ways for in-plane deformation measurements. The shortcoming of this fast and robust measurement technique based on image correlation techniques is that out-of-plane deformations in the direction of the measurement system cannot be detected and increases the measurement error of in-plane deformations. In this paper, we investigate a method that infers local out-of-plane motions of the workpiece surface from the decorrelation of speckle patterns and is thus able to reconstruct three-dimensional deformation fields. The implementation of the evaluation method enables a fast reconstruction of 3D deformation fields, so that the in-process capability remains given. First measurements in a deep rolling process show that dynamic deformations underneath the die can be captured and demonstrate the suitability of the speckle method for manufacturing process analysis.

**Keywords:** speckle photography; in-process measurement; deep rolling process

#### **1. Introduction**

The functionality of technical surfaces is significantly influenced by the manufacturing process [1–3]. This applies in particular to processes which aim to change the surface layer properties of the workpiece in a suitable manner. However, the targeted generation of, e.g., depth hardness gradients is challenging—even under laboratory conditions [2]. The reasons are that material loads occurring during the machining process in the form of displacements and strains, which are responsible for the material modifications, cannot be measured and that a lack of process knowledge exists. Therefore, the targeted material modifications are only adjusted via the machining parameters or process variables of the machine tools in an iterative way [4–6]. A remedy can be provided by the introduced concept of process signatures [7], which focuses on the internal mechanisms and the influencing quantities [8], leading to a material change to describing the workpiece modification. To gain a deeper understanding of the manufacturing process, the study of process signatures, which correlate the material changes occurring with the acting internal mechanical material loads, is ongoing. The use of process signatures not only enables a better understanding of the process, but also allows a comparison of seemingly different manufacturing processes. As a result, processes and process chains can be optimized to improve functional properties and reduce manufacturing costs [7].

Although a variety of characterization methods for the resulting material modifications exists, the metrological recording of the material loads in the ongoing production process remains a needed prerequisite. Mechanical workpiece loads can be calculated by comparing or correlating images taken before and during the occurrence of the deformation. In contrast

**Citation:** Tausendfreund, A.; Stöbener, D.; Fischer, A. In-Process Measurement of Three-Dimensional Deformations Based on Speckle Photography. *Appl. Sci.* **2021**, *11*, 4981. https://doi.org/10.3390/app11114981

Academic Editor: Kijoon Lee

Received: 6 May 2021 Accepted: 25 May 2021 Published: 28 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

to Digital Image Correlation (DIC), the method of Digital Speckle Photography (DSP) [9] is much more precise, especially on low-contrast surfaces. In terms of its maximum achievable resolution, the method is limited only by Heisenberg's uncertainty principle [10]. The inprocess capability and suitability of the speckle photography method for two-dimensional deformation measurement in the image plane with resolutions of less than 20 nm [11] has already been demonstrated in several manufacturing applications [12–15]. However, since mechanical machining causes at least a biaxial stress state and, thus, a triaxial strain state in the workpiece via the machining forces, a reconstruction of three-dimensional deformation fields is required for a complete metrological description of the loads.

Even though original speckle photography is only suitable for two-dimensional inplane measurements, various method derivatives exist to obtain out-of-plane deformation information as well. In this context, Khodadad et al. [16] propose a combination of speckle photography and speckle interferometry. The robust setup uses only one camera and can determine three-dimensional displacements with a resolution of 0.6 μm. The method introduced by Fricke-Begemann [17] uses the decorrelation of the speckle patterns associated with the out-of-plane motion, i.e., the speckle correlation functions changing with the motion are evaluated. The determined measurement uncertainties here are significantly lower than in the approach of Khodadad et al. [16] and are in the range of one tenth of the wavelength.

Therefore, the aim of this paper is the realization of a solely speckle-photographic in-process measurement system for the acquisition of three-dimensional load fields. In a first practical implementation, the deformation fields in a manufacturing rolling process are to be measured. The open questions in the context of speckle-photographic 3D deformation measurement in manufacturing processes are to what extent such a measurement system is suitable for use in the harsh manufacturing environment and to what extent the proposed algorithms are sufficient for in-process measurement. Furthermore, it must be clarified in which functional relationship the speckle decorrelation is related to displacements in the out-of-plane axis and how a calibration of this axis needs to be performed.

Accordingly, this paper is organized as follows: Section 2 describes the challenges of three-dimensional deformation measurement by speckle photography, especially in deformation evaluation according to Fricke-Begemann [17]. Moreover, an alternative approach and its implementation in the context of sub-pixel interpolation is presented. After the demonstration of calibration strategies, the first in-process measurements during deep rolling follow in Section 3. Section 4 discusses the reconstruction of the three-dimensional deformation or displacement field, followed by a summary and an outlook in Section 5.

#### **2. Materials and Methods**

#### *2.1. In-Plane Speckle Photography*

When illuminating a rough surface, a so-called speckle pattern is formed in the image plane of an imaging camera system by constructive and destructive interference (see zoomed area *Weval* in Figure 1). Here, each surface point on the object surface, with the coordinates *ξ* and *ψ*, is assigned to a point (*x,y*) on the camera chip. If a point (*ξ,ψ*) of the surface is shifted in-plane by Δ*ξ* or Δ*ψ* due to deformation, the corresponding point (*x*,*y*) in the image plane is shifted by Δ*x* or Δ*y*, respectively. This shift can be analyzed via the maximum of the cross-correlation function according to Figure 1. In order to measure a global deformation field of many object points in a region of interest *WFOV*, a small local evaluation window *Weval* is rasterized across the overall image *WFOV*.

**Figure 1.** Evaluation principle of speckle photography for the measurement of the two-dimensional in-plane deformation.

In order to determine the position of the maximum of the cross-correlation that is shown in Figure 1 with sub-pixel accuracy, interpolation methods are used based on curve approximation. The currently used sub-pixel interpolation algorithm, according to Nobach and Honkanen [18], determines the shift of the maximum (Δ*x*,Δ*y*) using an elliptic Gaussian function

$$g\_{\mathcal{B}}(\pounds, \mathfrak{y}) = a \cdot \exp\left\{ b\_{20} (\pounds - \Delta \mathbf{x})^2 + b\_{11} (\pounds - \Delta \mathbf{x}) (\hat{y} - \Delta \mathbf{y}) + b\_{02} (\hat{y} - \Delta \mathbf{y})^2 \right\},\tag{1}$$

with the variables (*x*ˆ, *y*ˆ), which is fitted into to the cross-correlation function. It can be rewritten as:

$$\log(\pounds, \mathfrak{Y}) = \exp\left\{c\_{00} + c\_{10}\mathfrak{X} + c\_{20}\mathfrak{X}^2 + c\_{01}\mathfrak{Y} + c\_{11}\mathfrak{X}\mathfrak{Y} + c\_{02}\mathfrak{Y}^2\right\} \tag{2}$$

with

$$\alpha\_{00} = \ln(a) + b\mathbf{\bar{z}}\boldsymbol{\alpha}^2\mathbf{x} + b\_{11}\boldsymbol{\Delta x}\boldsymbol{\Delta y} + b\mathbf{\bar{z}}\boldsymbol{\Delta}^2\mathbf{y},\tag{3}$$

$$x\_{10} = -2b\_{20} \Delta x - b\_{11} \Delta y\_{\prime} \tag{4}$$

$$c\_{20} = b\_{20} \,\tag{5}$$

$$x\_{01} = -b\_{11} \Delta \mathbf{x} - 2b\_{02} \Delta y\_{\prime} \tag{6}$$

$$c\_{11} = b\_{11\prime} \tag{7}$$

$$c\_{02} = b\_{02}.\tag{8}$$

The shift components Δ*x*, Δ*y* of the maximum are then obtained from the Equations (3)–(8) as follows [18]:

$$
\Delta \mathbf{x} = \frac{c\_{11}c\_{01} - 2c\_{10}c\_{02}}{4c\_{20}c\_{02} - c\_{11}^2},
\tag{9}
$$

$$
\Delta y = \frac{c\_{11}c\_{10} - 2c\_{01}c\_{20}}{4c\_{20}c\_{02} - c\_{11}^2}.\tag{10}
$$

The relationship between the displacements Δ*x*, Δ*y* in the image plane and the respective object displacements Δ*ξ*, Δ*ψ* within the measuring area are finally obtained according to the magnification *A* of the imaging system:

$$
\Delta \xi = A \,\Delta x,\tag{11}
$$

$$
\Delta \psi = A \,\, \Delta y.\tag{12}
$$

#### *2.2. Three-Dimensional Speckle Photography in Out-of-Plane Direction*

For the initially targeted objective of an extended understanding of the manufacturing process, the acquisition of information about the 3D deformation field is essential. Speckle photographic approaches are in principle also suitable for this purpose, since the recorded speckle patterns basically represent sections through the three-dimensional structure of the

wave field. Figure 2a shows an example of the measured three-dimensional structure of the speckle pattern, for a *ζ*-shift of the measurement plane.

**Figure 2.** Three-dimensional speckle pattern measured by shifting the camera and illumination system in the *ζ*-axis (**a**) and the calculated cross-correlation function of two evaluation windows shifted against each other in the *y*-direction (**b**) or in the *ζ*-direction (**c**).

The cross-correlation for pure in-plane displacements Δ*ξ*, Δ*ψ* is shown in Figure 2b. Deformations and displacements of the surface in the out-of-plane direction (Δ*ζ*), however, lead to a decorrelation of the speckle patterns, which means a reduced amplitude of the cross-correlation maximum (compare Figure 2b,c). This decorrelation was previously considered as a disturbance of the measurement effect and was minimized by using higher measurement rates. The approach of Fricke-Begemann uses the decorrelation and derives the out-of-plane deformation of the measurement surface via an elaborate shape analysis of the power spectrum of the cross-correlated speckle patterns [17]. A maximum likelihood estimation is first used to determine the tilt angles of the surface points and based on this, comprehensive three-dimensional deformation measurements are finally reconstructed. However, laboratory investigations of simultaneous tilts and in-plane displacements of the measurement surface illustrate the problem that arises in separating tilt and in-plane displacement in this form of evaluation, see Figure 3.

**Figure 3.** Power spectra of two speckle patterns for tilted or additional displaced measurement surfaces.

Especially in the range of very small in-plane displacements (<1 pixel), as they are predominantly present in the in-process load measurements of manufacturing processes, the differences are hardly recognizable. While in-plane displacements in the cross-correlation (see Figure 1) are clearly recognizable by the displacement of the maximum without significantly influencing the shape of the curve, in the evaluation via the power spectrum both the in-plane displacement and the out-of-plane movement affect the shape of the curve. Even though the power spectrum is very sensitive to the tilts, complex fit and filter algorithms are required to separate the two components. First approaches for the analysis of a two-axis motion, consisting of a tilt in the *ξ*-axis and a displacement in *ψ*-direction, resulted in computation times in the double-digit minute range for the smallest evaluation windows. With regard to the parallelization of the evaluation algorithms and the goal of a fast in-process measurement, an alternative approach is therefore pursued. The cross-correlation allows for separating the in-plane and out-of-plane displacement information by evaluating the shift of the correlation maximum (see Figure 2b) and the broadening of the normalized cross-correlation function (see Figure 2c), respectively. The shape or the width of the correlation function is already evaluated in the context of the performed sub-pixel interpolation. Inherently hidden in the time-optimized interpolation is the width *σ* of the approximated Gaussian function ℊ(*x*ˆ, *y*ˆ). It can be considered as a value for the speckle decorrelation and thus for the out-of-plane deformation. According to the Equations (1), (5) and (8), the width components for the *x*ˆ- or *y*ˆ-direction read

$$
\sigma\_{\chi} = \sqrt{\frac{1}{-2c\_{20}}},
\tag{13}
$$

$$
\sigma\_y = \sqrt{\frac{1}{-2c\_{02}}}.\tag{14}
$$

While the functional relationship between the displacement of the maximum of the cross-correlation function and the in-plane displacement of a surface point is linearly related via the imaging scale *A*, the functional relationship between the out-of-plane shift Δ*ζ* and the width of the correlation function *σ<sup>x</sup>* or *σ<sup>y</sup>* is unknown yet. For the establishment of a respective calibration function, the relation Δ*ζ*(*σx*,*σy*) is investigated experimentally in the following.

#### *2.3. Relationship between the Width of the Correlation Function and the Out-of-Plane Shift*

In order to analyze the functional relationship Δ*ζ*(*σx*,*σy*), the laboratory setup of a speckle-photographic measurement system was realized, see Figure 4a.

**Figure 4.** Speckle-photographic laboratory experiment: (**a**) laboratory setup with indicated bending of the workpiece sheet over a locating dowel pin (yellow) fixed in the camera system; (**b**) field of the respective local widths of the correlation functions, at a linear axis displacement of 28 mm.

The measurement object, a sandblasted sheet metal strip (400 mm × 50 mm) is illuminated with a collimated widened Omicron laser beam (*λ* = 405 nm). The illuminated area is observed and imaged with an Imaging Source camera (DMK 21AU04) through a Pentax 25 mm lens. Camera, laser and the center of the sheet metal strip (in the center of the measurement field) are fixed on a linear axis from PI company, Germany. The outer ends of the sheet metal strip, on the other hand, are fixed to the laboratory bench. Thus, a movement of the linear axis causes a bending of the sheet metal strip (yellow line in Figure 4a) and thus a relative movement between the center and its outer ends.

Figure 4b shows the global field of the respective locally calculated correlation function widths

$$
\sigma = \sqrt{\sigma\_x^2 + \sigma\_y^2} = \sqrt{\frac{1}{-2c\_{20}} + \frac{1}{-2c\_{02}}} \tag{15}
$$

for a linear axis displacement of 28 mm in *ζ*-direction with a total measurement field size of 30 mm × 40 mm. The parabolic bending around the dowel pin can be seen very well in the center of the image. The width of the correlation function increases significantly from the original 1.3 pixels to about 4 pixels. If looking in each case at the mean value of the width of the cross-correlation function along the *y*-axis at the position *x* = 0 in Figure 4b, one can see the linear relationship between the respective adjusted displacement of the linear axis and the broadening of the interpolated correlation function *σ* shown in Figure 5. The functional relationship between a changed loading condition with a resulting displacement Δ*ζ* and the resulting speckle decorrelation or change in the width Δ*σ* of the Gaussian fit ℊ from Equation (1) is thus linear in nature, i.e.,

$$
\Delta \zeta = \pm B \,\Delta \sigma.\tag{16}
$$

**Figure 5.** Linear relationship between the width of the correlation function *σ* calculated in the sub-pixel interpolation and the shift of the linear axis in *ζ*-direction.

The indefinite sign for the calibration factor *B* results intuitively from Figure 2. The shift by +Δ*ζ* leads to the decorrelation of the speckle pattern to the same extent as a shift by −Δ*ζ*. While the calibration factor can be defined prior to a respective measurement on the basis of a defined displacement, the sign can only be determined via prior knowledge of the observed deformation process. In the case of the bending test, for example, the outer ends of the sheet metal strip will move away from the camera in negative *ζ*-direction, so that negative signs are to be expected. While the in-plane deformation (Δ*ξ*,Δ*ψ*) is linearly related to the displacement (Δ*x*,Δ*y*) of the maximum of the cross-correlation via the scaling factor *A* (Equations (11) and (12)), it is shown by Equation (16) that a linear relationship also results for the out-of-plane deformation Δ*ζ* via the factor *B* and the change in width of the cross-correlation function Δ*σ*.

#### **3. Results of In-Process Measurements in a Deep Rolling Process**

*3.1. Measurement Setup and Experimental Realization*

Due to the expected strong out-of-plane movement in the edge area of the machined surface, the deep rolling process was selected as the manufacturing process to investigate first. The realized manufacturing process with the corresponding measurement setup is shown in Figure 6. Contrary to the intuitive idea, the setup is upside down and the rolling tool presses on the workpiece from below. The design was therefore chosen so that any cooling lubricant escaping from the hydraulic rolling tool is prevented by gravity from contaminating the measuring field located on the front surface and disturbing the measurement. The rotation axis of the cylindrical tungsten carbide tool with a diameter of 13 mm, which protrudes a few millimeters above the workpiece, is located in the *ζ*-axis of the specimen or measuring system. The *ξψ*-plane lies in the face of the hardened martensitic 42CrMo4 workpiece (150 mm × 70 mm × 20 mm). The tool moves while rolling with a loading of 350 bar in the negative *ξ*-direction with a feed rate of v = 0.02 mm/s (see Figure 6a).

**Figure 6.** Test setup for deep rolling: Basic measurement setup (**a**) and practical implementation with enlarged view from the opposite *ζ*-direction to the camera (**b**).

The 8 mm × 6 mm measuring field is illuminated by a short pulse laser (*tp* = 1 ns, *λ* = 532 nm) from Horus via two deflection mirrors and observed with a 4K high speed camera CP70 from Optronis via a 90 mm Apo-Rodagon lens from Rodenstock (see Figure 6b). In the rolling process with its moderate feed rate, full-frame recordings could be made at a recording rate of 166 frames per second. The implemented manufacturing setup is also used for in situ X-ray diffraction analysis and residual stress measurement and is described in detail in [19].

#### *3.2. Results of 3D-Deformation Measurements*

The results of a three-dimensional load measurement for the deep rolling test are shown in Figure 7. Assuming a homogeneous workpiece, the evaluation range in *ξ*-direction is extended from 8 mm to 20 mm. For this purpose, the speckle-photographic deformation measurements of the time series—in the direction of movement of the tool—were stitched into an extended evaluation field [15,20]. The calibration of the *ξ*- and *ψ*-axis is done according to Equations (11) and (12) from the magnification *A* of the lens system.

**Figure 7.** In-process measurement in the deep rolling process: The occurring shifts in *ξ*-direction (**a**) and Ψ-direction (**b**) as well as the reduction of the calculated width of the correlation function resulting from the out-of-plane shift as a value for the *ζ*-shift (**c**) are shown.

> The load measurements in the running deep rolling process show maximum deformation differences for the *ξ*- and *ψ*-direction at a depth of 800 μm. Accordingly, the equivalent stress maximum also occurs at this depth, which is in line with the expectations [19]. At the machined surface, the material is moved up to 100 μm in the direction of the tool movement (see Figure 7a). The minimum displacement in *ψ*-direction is located about 1 mm behind the contact point of the tool at position 14 mm and is -40 μm (see Figure 7b). In addition to the measured in-plane deformations in *ξ*- and *ψ*-direction, Figure 7c shows for the first time the out-of-plane component of the in-process characterization in the direction of the camera axis. The determined deformation in *ζ*-direction corresponds to the broadening of the correlation function or a decorrelation of the speckle patterns between unloaded and loaded state.

#### **4. Discussion**

For a complete description of the three-dimensional displacement field, the only remaining task is to perform the calibration between the Δ*ζ*-displacement and the decrease of the correlation width Δ*σ* according to Equation (16). In this case, the scaling or calibration of the *ζ*-axis is performed by a subsequent reference measurement with a stylus instrument (Mahr LD-120). For this purpose, the tactilely measured burr (see Figure 8a) was compared with the determined width of the correlation function (see Figure 8b). The approximately linear deformation of the burr in the section shown is reflected proportionally in the linear behavior of the correlation width Δ*σ* and confirms the preliminary tests from Figure 5. Therefore, a linear correlation is identified and the *ζ*-axis is accordingly scaled for the speckle photographic measurement in Figure 8c. The evaluation of the speckle decorrelation basically does not provide any information about the sign of the measured Δ*ζ*-deformation (see Section 2.3). However, the evaluation succeeds due to the prior knowledge gained from the calibration measurements that the material is pressed outward in the direction of the camera.

**Figure 8.** 3D deformation measurement in the deep rolling process. (**a**) Tactilely measured burr; (**b**) width of the correlation function as a function of the out-of-plane displacement of the burr; (**c**) speckle-photographically measured deformation field in the *ζ*-direction after calibration.

Figure 8c clearly shows the strong burr formation at the edge of the workpiece with an extension of up to 80 μm, caused by the high rolling pressure of 350 bar applied to the 42CrMo4 sample. Highlighted in the measured data is the dynamic deformation front, which propagates like a 'bow wave' in front of the tool. It is exactly this dynamic deformation front that has so far caused problems in the interpretation of the measured data, since the accompanying decorrelation of the speckle patterns could not be explained. The acquisition of load dynamics shows that speckle photography is suitable for measuring and evaluating three-dimensional deformation fields within ongoing manufacturing processes. In particular, the analysis of the load dynamics is a key to the establishment of process signatures described above as well as for a comparison with corresponding simulations of the manufacturing process and illustrates the great potential of this in-process measurement technique.

#### **5. Conclusions and Outlook**

Laboratory investigations in the context of the analysis of the functional relationship between the decorrelation of measured speckle patterns or the broadening of the correlation function Δ*σ* and the out-of-plane deformation Δ*ζ* confirm a linear behavior. Moreover, the additional computational effort for the determination of Δ*σ* in the sub-pixel evaluation is negligible, which means that the parallelizability of the algorithms remains given and three-dimensional in-process deformation measurements during the rolling process become possible. The measurements during deep rolling show the dynamic deformations below the tool. From these deformations, the strain and thus the load during the die engagement can be calculated directly via the formation of the gradients. In particular, the 'bow wave' of out-of-plane deformation moving ahead of the tool shows that the local mechanical material loads can be much more pronounced than the workpiece modifications remaining in the material. In order to better understand manufacturing processes, 3D speckle photography can thus make an important contribution in the context of the sought-after process signatures.

While the calibration of the *ξ*- and *ψ*-displacement can be performed via the imaging scale of the camera system, this is only possible for the *ζ*-axis via a subsequent reference measurement with a stylus instrument. The ambiguity with regard to the direction of the displacement can also be eliminated by the additional information from the reference measurement. In order to ensure in-process capability in various manufacturing processes, the realization of a self-calibration of the measuring system is planned for the future. For this purpose, a three-axis piezo stack is to be used, via which the system can be calibrated before each individual measurement by means of defined, homogeneous

displacements. In addition to the calibration, it is thus simultaneously possible to eliminate optical distortions—for example due to misalignments of the measuring system—already during the measurement and to carry out quantitative 3D deformation measurements in running manufacturing processes.

**Author Contributions:** Conceptualization, A.T.; investigation, A.T.; methodology, D.S. and A.F.; supervision, A.F.; writing—original draft, A.T.; writing—review and editing, D.S. and A.F. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the German Research Foundation (DFG) for subproject C06 'Surface optical measurement of mechanical working material loads' within the Transregional Cooperative Research Center SFB/TRR136.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data presented in this study are not available.

**Acknowledgments:** The authors thank Heiner Meyer (SFB/TRR136, subproject C01) and their colleagues from the Leibniz Institute for Materials Engineering IWT for the friendly support during the conduction of the in-process measurements.

**Conflicts of Interest:** The authors declare no conflict of interest. The founding sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, and in the decision to publish the results.

#### **References**

