Real-Time Optical Detection of Artificial Coating Defects in PBF-LB/P Using a Low-Cost Camera Solution and Convolutional Neural Networks

Abstract

Additive manufacturing plays a decisive role in the field of industrial manufacturing in a wide range of application areas today. However, process monitoring, and especially the real-time detection of defects, is still an area where there is a lot of potential for improvement. High defect rates should be avoided in order to save costs and shorten product development times. Most of the time, effective process controls fail because of the given process parameters, such as high process temperatures in a laser-based powder bed fusion, or simply because of the very cost-intensive measuring equipment. This paper proposes a novel approach for the real-time and high-efficiency detection of coating defects on the powder bed surface during the powder bed fusion of polyamide (PBF-LB/P/PA12) by using a low-cost RGB camera system and image recognition via convolutional neural networks (CNN). The use of a CNN enables the automated detection and segmentation of objects by learning the spatial hierarchies of features from low to high-level patterns. Artificial coating defects were successfully induced in a reproducible and sustainable way via an experimental mechanical setup mounted on the coating blade, allowing the in-process simulation of particle drag, part shifting, and powder contamination. The intensity of the defect could be continuously varied using stepper motors. A low-cost camera was used to record several build processes with different part geometries. Installing the camera inside the machine allows the entire powder bed to be captured without distortion at the best possible angle for evaluation using CNN. After several training and tuning iterations of the custom CNN architecture, the accuracy, precision, and recall consistently reached >99%. Even defects that resembled the geometry of components were correctly classified. Subsequent gradient-weighted class activation mapping (Grad-CAM) analysis confirmed the classification results.

1. Introduction

The technology of additive manufacturing (AM) enables the production of highly complex and qualitative components for a wide range of industrial and commercial applications such as automotive, aerospace, medical devices, dentistry, electronic components, and even microstructural applications [1,2,3,4,5,6,7,8]. In particular, the laser-based powder bed fusion of polymers PBF-LB/P and metals PBF-LB/M, in which powder is applied in layers and sintered or fused via a laser [9,10], has a wide range of applications and research areas [11,12,13,14,15,16,17,18,19,20]. Despite the many advantages of this technology, there are still certain limitations when it comes to reproducible component quality across multiple build jobs. For example, the quality of AM-produced parts varies from batch to batch and from machine to machine. There is also a lack of standards developed for AM processes, as compared to the tolerance requirements defined in DIN 16742:2013 [21] for the injection molding process, where the dimensional errors of AM exceed the defined ranges [22]. Another limitation is the inconsistent dimensional error, which is often outside the standard tolerance range [23]. When it comes to PBF-LB/P, any defect is critical, both in terms of endangering the process itself, which could lead to a process stoppage, and in terms of compromising the quality of the part in every way. Further investigation is required as part of process monitoring to understand the thermal interactions, ensure the part quality, and reduce the number of failures. PBF-LB/P has already made a remarkable breakthrough in certain industrial applications by reaching high standard requirements, but successful integration into the industrial process flow and supply chain is still limited by the lack of sustainable process control. The integration of suitable real-time quality control is difficult to implement due to the largely closed software and hardware solutions and the fact that it is almost impossible to interfere with the parameters during the running process [24,25]. Several simulation designs for real-time process control systems in powder bed fusion are proposed [26,27,28,29,30,31]. A good overview of important parameters and influencing factors is given in [32] by Vlasea et al. This review describes a testbed for implementing process monitoring methods, including simulation approaches for real-time process control algorithms. A measurement and process control strategy is presented based on an organizational structure that includes pre-processing, in situ defects or fault detection, in situ continuous feedback control, and signature deviation control. When this organizational structure of Vlasea et al. and the individual components are examined in more detail, a concept for the in situ defect detection component can be developed on the basis of the current study. For this defect detection concept, certain parameters, as shown in Figure 1, need to be defined and implemented in order to be integrated and processed within an overall process control structure.

Regarding the failures and defects that can occur in the processing of polyamides as well as metallic powders, there are synergies as well as independent effects. In addition to similar defects like curling, shrinkage, coating failures, part shifting, particle drag, powder contamination, and powder short feed, the manufacturing of metal powders might lead to additional defects like cracks, spatters, and porosity [25,33,34]. Figure 2 shows the effect of part shifting during the ongoing sintering process in the FORMIGA P 110 PBF-LB/P system via the EOS (Krailling, Germany) used for this work. As can be clearly seen, this event immediately leads to a complete stoppage of the build job.

Previous work has shown several different experimental approaches for the detection of defects in both metallic and plastic powder processing [24,25,28,35,36,37,38,39,40,41,42,43,44,45]. However, despite the promising results, these are always scientific experiments with expensive measuring instruments or complex experimental set-ups that are not yet suitable for industrial applications.

In this paper, a new approach to coating defect detection is proposed by installing a cooled camera system inside the PBF-LB/P system and evaluating the data via deep learning using a custom convolutional neural network (CNN). Since the artificial and reproducible induction of various defects in the field of powder bed fusion represents a major challenge without direct process termination [24,25,35], this approach represents a new experimental solution.

2. Background and Methodology

Recent studies and literature reviews show that the use of deep learning is becoming increasingly common, especially in the manufacturing industry, to monitor processes, detect defects, and predict component quality [46,47,48,49]. According to the literature, the application of supervised CNN in particular has often been used to check relevant components. The use of supervised CNN has shown promising results for tasks such as defect classification and defect segmentation. Classification works at the image level and recognizes the object in the image. Segmentation operates at the pixel level and detects the object type in each pixel of the input [46]. The following are a few examples of the use of supervised CNN in industrial and safety-related applications. In [50], Chen and Jahanshahi proposed a CNN to detect cracks in safety-relevant parts of a nuclear power plant. Dong et al. [51] addressed the identification of small abnormalities in applications such as industrial inspection. A steel plate defect inspection was proposed by He et al. [52] for real-time quality control of the manufacturing process. In [53], Shi et al. showed the effective deployment of CNN for automated underwater pipeline defect inspection. Tabernik et al. [54] introduced a segmentation-based deep learning approach for surface defect detection, which was designed to detect and segment surface anomalies and showed that the proposed approach is able to learn on a small number of defects, using only about 25–30 defective training samples instead of hundreds or thousands. Another industrial non-destructive testing (NDT) example is given by Tang et al. in [55], who inspected X-ray images of castings using spatial attention bilinear CNN. In [56], an online sequential classifier and transfer learning (OSC-TL) method was presented by Yang et al. for the training and classification of Mura defects in the manufacturing process of flat panel displays.

In the literature, unsupervised CNN models are mainly categorized into anomaly detection, GAN-based models (Generative Adversarial Networks), and hybrid models. Compared to supervised CNN architectures, unsupervised models are able to perform image labeling, pixel-level, defect image classification, and defect area localization. External and internal defect detection is possible within several computer vision tasks [46]. The following are a few examples of unsupervised CNN applications for anomaly detection, GAN-based, and hybrid models. In [57], Chow et al. proposed a convolutional autoencoder (CAE) to implement the anomaly detection of defects on concrete structures and further facilitate the visual inspection of civil infrastructure. Another example of an autoencoder is given by Yang et al. in [58]. A multi-scale, fully convolutional autoencoder (FCAE) was used, considering only defect-free images for surface defect detection. Ruff et al. [59] used a deep support vector description (D-SVDD) CNN architecture for semi-supervised and unsupervised anomaly detection.

Regarding unsupervised GAN models, Lian et al. [60] presented a model where GAN is combined with CNN to enable flawless image and identify miniature surface defects. Another approach for a surface defect-generation adversarial network (SDGAN) was presented by Niu et al. in [61]. The model was applied on accurate images to increase the dataset of defective images. For high-dimensional datasets, Zenati et al. in [62] presented a trained GAN-based model using a score function to increase the efficiency of defect detection. Another example for high-dimensional datasets was given in [63] by Deecke et al. by investigating a new anomaly method based on searching for a good representation of this sample in the latent space of the generator.

The final category of unsupervised CNN models for defect detection is the hybrid variant. A generic method for automated surface defect detection based on a bilinear model has been proposed by Zhou et al. in [64] by extracting the features globally and locally, applying subnetworks based on visual geometry and labelled Double-VGG16. Another approach of a hybrid unsupervised CNN model was presented by Tsai et al. [65] for automatic defect detection in material surfaces. The study included a two-stage deep learning scheme for pixel-wise defect detection on textured surfaces using two CycleGAN models and a U-Net semantic network. The previous studies showed that the use of deep learning is well suited for industrial manufacturing purposes in order to classify and identify both external and internal defects. The targeted use of CNN in additive manufacturing, especially in powder bed fusion, has already been discussed in several applications and has shown very good results in process monitoring, defect detection, and the prediction of component quality [47]. What is striking, however, is that most of the research is being carried out in the field of powder bed fusion of metals, and there are only a few cases of CNN being applied to the manufacturing of polymers. This could be due both to the nature of the defects, which are more visible and occur in greater numbers in the metal fabrication sector [33,34], and presumably to the interest from the manufacturing industry [66].

Our previous approach [35] showed good results in detecting curling defects on the same PBF-LB/P system using thermal imaging and defect classification using a VGG-16 CNN. An average accuracy of 98.54% for the detection of curling defects was achieved and the results encouraged the effective use of DL for the non-destructive, in situ quality control of powder bed fusion processes. Another study by Westphal and Seits [67] demonstrated the application of non-destructive defect detection in the field of powder bed fusion of polymers using complex transfer learning (TL) methods. Their approach used a VGG16 and Xception CNN model with pre-trained weights from the ImageNet dataset as the initialization. In [68], Arslan et al. demonstrated a DL defect detection system called “SLS-ResNet” to detect defects such as curling, part shifting caused by curling, and powder short feed on a DTM Sinterstation 2500 Plus PBF-LB/P system. Unfortunately, no further information on defect generation was given. Another approach on the same Sinterstation was presented by Xiao et al. in [69] to detect the same defects as in the previous study, but in this case, it was explained in detail how certain parameters such as process temperature were manipulated to create the defects. In [70], Schlicht et al. showed a method for the in-line assessment of part porosity to detect limitations in the reproducibility of manufactured parts by using deep residual neural networks. Their study used the same EOS PBF-LB/P system as the current study. Baturynska et al. [23] showed the influence of component positioning in the installation space of an EOS P 395 PBF-LB/P system on part dimension accuracy. This work aimed to predict the scaling ratio for each part separately, depending on its placement, orientation, and CAD characteristics, using artificial neural networks (ANN) such as Multi-Layer Perceptron (MLP) and CNN.

3. Materials and Methods

3.1. PBF-LB/P System

Just like in previous studies [24,35], all experimental trials were carried out on the same PBF-LB/P system. This laser-based powder bed fusion system for polymer powders uses high performance PA (PA2200) nylon fused via a CO₂ laser and is capable of producing small batch prototypes as well as larger quantities of industrial-grade components. According to the technical description of the system [71] and the product information of the powder [72], the parameters are listed in

Table 1. Parameters of EOS FORMIGA P 110 (EOS, Krailling, Germany).

Laser type	CO2
Laser power	30 W
Laser wavelength	10.6 µm
Scanning speed	5 m/s
Exposed area (xy)	200 mm × 250 mm
Maximum part height (z)	300 mm
Layer thickness	50 µm–200 µm
Defined layer thickness	100 µm
Powder type	PA2200 nylon

.

A standard unadjusted laser scan speed and hatch spacing from the EOS default exposure strategy was used, but its parameters cannot be viewed or adjusted as EOS does not provide any further information about them.

3.2. Simulation of Artificial Coating Defects as Part Shifting and Particle Drag

As described in [24,25,35,68,69], it is hardly possible to intervene in any kind of process parameters of industrial PBF-LB/P systems, to adjust them, or to obtain information about exposure strategies. Consequently, it is difficult to simulate any defects in a reproducible manner during the running process without causing a complete process termination. The studies showed that with the EOS system, it is only possible to manipulate the process temperature before the start of the process in order to artificially cause a curling defect. Curling, which is the thermally induced in-process distortion of the free edges of build parts [73], can lead to the shifting of whole components if too pronounced. Depending on the size of the displaced parts and the current process progress, this can lead to an immediate process termination. However, as the height of curling can only be determined using complex measuring methods, as described in [24], the actual occurrence of this effect is subject to chance, making reproducible generation impossible. The effect of particle drag can also only be caused accidentally by a manual contamination of the powder. One of the main goals of this study was to artificially generate these defects efficiently, in order to generate as much data as possible from both the defect-free and defective process sections, and then train a custom CNN for automated defect detection.

To determine the penetration depth of the foreign bodies at which the actual components are displaced from the powder bed, different wire thicknesses of copper and aluminum were applied to the coater blade, as shown in Figure 3, and the sintering process was then started normally.

A total of seven different thicknesses from 0.3, 0.5, 0.6, 1, 1.5, 2, and 3 mm were applied to the coater. As well as the depth of penetration into the powder bed, the current state of the process was also important, making it difficult to make an explicit statement, so a penetration depth between 0.1 mm and 0.5 mm was defined for further investigations. After these initial trials, however, it quickly became apparent that a stationary assembly of a simulated defect on the coater blade was not practical for further investigation, as the defect would be continuously implied with each layer. In order to train a CNN effectively, sufficient data from both the correct and defective process steps must be available. When it comes to the real-time monitoring of the process, and especially the simulation of randomly occurring defects, the defects must not occur permanently.

Based on these findings, a mechanical assembly was developed that is mounted on the coater blade and can be activated externally outside the machine. In order not to damage the machine, the activation of this defect simulation could only take place when the coater blade was in a certain area of the powder bed. To define the end positions of the coating blade, two limit switches have been fitted to the actuator cylinder inside the machine, as shown in Figure 4.

Within this time window, the coating defect can then be randomly simulated, controlled via a Raspberry Pi. The number of new layers after which the defect occurs can also be randomized. The mechanical actuation of the defect on the powder bed surface was achieved using components made from stereolithography-produced high temperature material mounted on the coating blade to withstand temperatures of up to 170 °C during the ongoing sintering process. This mechanical design includes a lever structure that can be operated from outside the machine via mechanical Bowden cables, as shown in Figure 5.

When the system is then activated during the application of a new layer of powder, contact is made with the powder bed surface and an artificially induced coating defect is created. Figure 6 shows the system in the activated state. In this study, a needle was used to simulate particle drag and the part shifting of small components. Depending on the depth of penetration into the powder, the intensity of the defect can then be individually adjusted.

The Bowden cables were routed inside the machine and then exited through a customized inspection hatch on one side of the machine. The mechanics were driven via two stepper motors using a lever system, which enabled very precise programming and control. The system could be used continuously under real conditions during the sintering process to produce a variety of coating defects, as shown in Figure 7.

This approach clearly demonstrates a novel and reproducible method of artificial defect simulation in PBF-LB/P without the risk of process termination. It was shown that the simulation could be run for many hours over several construction jobs to randomly generate defects without human supervision, in order to generate as much data as possible for both training the CNN and the subsequent classification of defective and defect-free images.

3.3. Camera Set-Up and Machine Integration

One of the main objectives of this study was to implement a suitable and low-cost process monitoring system for PBF-LB/P that allows both a simple set-up and operability as well as a permanent monitoring of the process without the need for complex experimental set-ups. It was also clearly shown that the non-machine-learning approaches to process the monitoring of PBF-LB/P and PBF-LB/M described in Section 1 were able to achieve the targeted results, but a complex test set-up with particularly cost-intensive measurement equipment was always required.

Based on the comparison of different camera systems, it was decided to use the Raspberry Pi Camera Module 2 for further investigation in this study. This HD camera module uses a Sony IMX219 sensor with 3280 × 2464 pixels and a maximum frame rate of 30 fps [74]. However, this inexpensive and readily available model has an operating temperature range of −20 °C to only 60 °C, which in turn raises the first considerations for a cooling option within the machine, as the temperature during the ongoing production process is more than 170 °C throughout. When the maximum operating temperature is reached, these components lose the signal due to their characteristics [75].

To summarize the key requirements for the cooling system, the camera housing should be additively manufactured and able to withstand a continuous operating temperature of 180 degrees. In addition, the camera lens must be flushed with nitrogen (N₂) to avoid coating defects during the process due to evaporation effects. A camera housing was designed that is cooled via the machine’s internal N₂ supply. There is virtually no information in the literature on solutions for cooling cameras or other measuring equipment used in the PBF-LB/P build chamber during the process. Only the study by Sillani et al. [25], in which the powder bed is measured using laser profilometry, mentions that the sensor is cooled inside the build chamber; unfortunately, no further information is available on the implementation of the cooling concept.

The components for the camera housing were produced on a stereolithography system using a high-temperature polymer. When cured, the resin can withstand sustained temperatures of just over 200 °C [76]. The camera window is a 20 × 20 × 0.13 mm piece of glass glued in place with high-temperature silicone. An additional line was installed on the N₂ generator and routed into the machine’s build chamber. This leads into the body of the camera housing so that the camera is cooled throughout. The N₂ then exits the other side of the housing and is returned to the housing via an additional line to flush the glass with an air-knife. The design of the cooling system is shown in Figure 8.

Tapered and airtight, 1/8” stainless steel fittings were installed, as shown in Figure 8. The positioning on the heating system was such that the entire powder bed could be captured at the best possible camera angle, as can be seen in Figure 9.

In summary, this approach provides a low-cost solution to capture the complete powder bed surface of the PBF-LB/P system with a standard RGB camera. The design of the housing allows the camera to be cooled, ensuring long-term use under the process conditions, even over multiple build jobs. By using the machine’s internal N₂ generator, no external supply of coolant is required. This system enables permanent process monitoring without complex and cost-intensive test set-ups and the generation of high-quality data for evaluation using CNN.

4. Design and Optimization of the CNN Architecture

4.1. Data Quality and Interfaces

A custom Convolutional Neural Network (CNN) designed to identify particle drag is the method of choice for this evaluation. These deep learning networks are specifically designed for image classification and offer a high degree of customization via the layered structure and tuning of hyperparameters. The design is optimized for portability, so it can be used on devices such as standard laptops, workstations, and Raspberry Pi. Seven datasets were generated using the proposed method described in Section 3, containing various types of defects such as part shifting, particle drag, and powder contamination on different manufactured part shapes and also including an additional defect called overheating. Overheating can be caused by the manual cover of the internal pyrometer and leads to uncontrolled fusion of the powder bed surface.

Table 2. Datasets and characteristics.

Dataset	Ok Frames	Defective Frames	Resolution	Defect Info
1	16.506	4.827	480 × 640	Overheating
2	3.007	n.a.	480 × 640	Various defects
3	22.449	2.356	480 × 640	Various defects
4	24.823	388	960 × 1280	Various defects
5	40.609	1.044	960 × 1280	Various defects
6	79.563	3.400	mixed	Consolidated
7	3.000	3.000	mixed	Evaluation

shows the datasets by characteristics.

The data was automatically labelled by interfacing with the defect induction mechanism using GPIO pins on the Raspberry Pi and Arduino Mega, which control acquisition and induction, respectively. This allowed a lot of data to be labelled automatically without time-consuming sorting and the viewing of frames. The consolidated dataset was used for training purposes only. Evaluation results are derived exclusively from the test dataset. It was checked both manually and automatically using a custom function that the latter did not contain any duplicates found in the training data, so as not to bias the results. Figure 10 shows the applied interface structure with the communication, the data hierarchy, and the concept for automated labelling by signaling via the mechanical structure for defect generation.

As can be clearly seen in Table 1, there is a large difference in the amount of data divided into classes. More precisely, this imbalance can lead to classification problems in the field of computer vision [77]. This phenomenon can be avoided via a method proposed by Cui et al. [78], which uses a class-balanced loss by designing a reweighting scheme that uses the effective number of samples for each class to rebalance the loss, resulting in a class-balanced loss. By implementing an imbalance factor, the weights are adjusted based on the class imbalance, whereas a cross-entropy loss function does not address the unbalanced classes. Further steps are described in more detail in Section 4.3.

4.2. Initial Setup

The proposed setup has been evaluated for use with TensorFlow 2.0 (TF), Keras, and PyTorch (PT) open-source libraries, which are used to develop CNN in Python. All three libraries use both tensor data storage and neural networks as computational operational graphs, as well as automatic differentiation. TensorFlow implements more complex learning but provides a broader feature library than PT. The Keras library is implemented on top of TF and simplifies the learning process, albeit with slightly higher test environment accuracy but longer training times [79]. However, TF’s installation process is more complex than PT’s due to the lack of native GPU support [80]. PT, developed by Meta, has built-in GPU support for Apple Silicon, making it easier to install [79], and beats TF in the operating time cycles. Despite these differences, there is no objective superiority between the libraries [80]. In the end, TF was chosen to develop a custom CNN for defect identification, with portability for use in a variety of field devices, such as a Raspberry Pi or a standard laptop. Development was carried out in a virtual environment using Python 3.10.10.

4.3. Model Architecture

The data preprocessing involves the removal of corrupt images and the loading of the data as a data set object using a directory function. This provided a quick way to ingest new data without a custom pipeline. The basic model used in this study adopts a training, testing, and validation split of 6:2:2. The input image shape is 480 × 640 pixels, with three channels following the RGB color chart. At the beginning of the model architecture, there is a Conv2D layer with a set of 32 filters. These filters are 3 × 3 in size and a convolution of stride 1, which acts as a feature identifier. Stride indicates how many steps are moved in each step of the convolution. This layer uses the ReLU activation function to introduce non-linearity, which is critical for handling real-world non-linear data. Following the Conv2D layer, a max-pooling operation is implemented for 2D spatial data for non-linear down-sampling, which divides the image into non-overlapping rectangles and outputs the maximum value for each sub-region, reducing the resolution of the feature map to simplify the computation and minimize overfitting. The model uses two more such pairs of layers, containing a series of 64 filters in the Conv2D section. These layers are used to recognize higher-level features, such as shapes or defined objects, via progressive learning and abstraction. A transformation of the pooled feature map is performed via a flattening layer after abstraction. The single column output of the transformation is then transferred to the final fully connected layers. High-level inference on the extracted features is then performed via a subsequent dense layer of 256 neurons using a ReLU activation function. Softmax activation is used by the final dense layer of the basic model to produce a two-class probability distribution indicating the model’s prediction of the input image. Figure 11 shows the design of the basic model architecture, which was then used as the basis for further adaptation.

As mentioned in Section 4.1, the class imbalance was tackled using the “Class-Balanced Loss” by Cui et al. [78]. Based on cross-entropy, this proposed loss function quantifies the performance of a model architecture that outputs the probabilities between two values (0/1). It can be used for both multi-class and binary classification. It shows an increase when the predicted probability deviates from the actual label [81]. If there is an imbalance, the classes are treated in the same way, which can lead to overfitting on the output. The weighted cross-entropy (WCE) used in the proposed method is calculated by applying these weights to the cross-entropy, a widely used loss function in classification problems that does not take into account the probabilistic distribution [82].

The required scalar value of the WCE loss is obtained by calculating the mean of the elements across the tensor’s dimensions. A study by Smith [83] identified the learning rate as a critical hyperparameter to be iteratively adjusted in deep neural networks to improve the model performance. The technique of “cyclic learning rates” describes a process where one starts with a lower learning rate that increases exponentially with each batch. The final iteration of the model trains an edge detection model using a Sobel filter. The idea was to explicitly detect grooves in the powder bed, i.e., sharp edges. Since defects such as part displacement or particle drag leave ridges in the powder bed, this approach produces a classification. Figure 12 shows the original image of a simulated defect and the image after Sobel filtering.

The final proposed model architecture based on the previous model is shown in Figure 13.

5. Results

To avoid distorting the results or creating a bias, only the test dataset was examined in more detail and used for the evaluation. However, TF does not include a function to output the F1 score, so a separate category had to be implemented for this. An evaluation script was used to compare the results of all model iterations and the results are shown in

Table 3. Results of all model iterations.

Model Architecture	Loss	Accuracy	Precision	F1 Score	Epochs
Basic	0.0002	0.7233	0.7233	0.7233	15
+ Weighted cross-entropy	<0.0001	0.9838	0.9846	0.9835	10
+ Learning rate scheduling	<0.0001	0.9852	0.9886	0.9847	15
+ Sobel layer	<0.0001	0.9912	0.9910	0.9911	10

.

The results in the table clearly show how the performance of the model could be improved by adding new layers and adjusting the parameters. Accuracy was increased from 72.32% in the base model to 99.11% in the final model architecture by adding the WCE loss function, the learning rate scheduling, and the Sobel filter layer. After 10 epochs, the final model also achieved a 99.1% precision and an F1 score of 99.1%.

During the evaluation of the model, Grad-CAM heat maps were rendered for each image to provide insight into the model’s predictions. Gradient Weighted Class Activation Mapping is a class-discriminative localization technique that makes any convolutional neural network model more transparent by providing visual explanations [84]. This method, implemented via the “make_gradcam_heatmap” function, highlights sections that have significantly influenced the model’s prediction. This function takes four inputs as the image to be scored, the last layer of the model, the trained network, and the value of the predicted class to generate the explanatory heatmap. Gradients are calculated using the TF’s GradientTape function, a tool that documents operations for automated differentiation. It determines the gradients of the predicted class in relation to the output of the previous layer. The function averages the calculated gradients over each feature map in the output of the last convolutional layer, producing a set of individual weights for each feature map used. The feature maps are then multiplied by the assigned weights to produce weighted feature maps. A visual heatmap per class is then created by using the output images from the previous layer and calculating their weighted sum by averaging the resulting maps across all channels. The Grad-CAM images generated for each frame during the evaluation provided a visual explanation of the classification process. The randomized evaluation of these heatmaps clearly showed that the correct areas were marked and that the accuracies listed in Table 2 were correct and not due to false convergence. The Grad-CAM heatmaps make the classification of the CNN comprehensible and visually highlight the areas that were crucial for the prediction results.

In order to assess the actual usability of the custom CNNs used, a random comparison of the original frames with the Grad-CAM heat maps was performed, as shown in Figure 14. As can be clearly seen, all images without defects were classified as “True: ok” and “Predicted: ok”. In contrast, the images with a detected defect were classified as “True: defect” and “Predicted: defect”. During the various build jobs carried out as part of this work, attempts were also made to produce component geometries that looked as close to a defect as possible, and these geometries were also not classified as defects according to the Grad-CAM heat maps.

6. Discussion

This paper proposes an approach for real-time process monitoring and in situ defect detection in PBF-LB/P using a low-cost camera solution and a custom Convolutional Neural Network for data analysis and defect prediction. As previous studies have shown, the artificial creation of defects in powder bed fusion is severely limited by the ability to manipulate the process parameters, so new ways must be found to simulate these defects and further investigate their effects on the process. These approaches have shown promising results in the generation of curling and the detection of this effect via sensory measurement of the powder bed using laser profilometry, as shown in [24,25], and process monitoring using thermal imaging and CNN [35]. In order to investigate the effects of other defects on the process, a new method had to be developed that would influence the process in an independent mechanical way without causing a process termination. This is the only way to take a further step towards holistic process monitoring via defect detection. A mechanical assembly was therefore developed and mounted on the coater blade, which then penetrates the powder bed to a defined depth in a reproducible manner via an external actuator system. Controlled via a Raspberry Pi and triggered by an end switch on the coating blade actuator, the defects were randomly generated and the test series could then be run over several build jobs without the need for human intervention.

Many of the approaches discussed in Section 2 have already shown that deep learning computer vision is a mature method for monitoring various manufacturing processes, using supervised, unsupervised, GAN, and hybrid convolutional neural networks to detect or predict various defects. The use of a CNN enables the automated detection and segmentation of objects by learning spatial hierarchies of features from low to high-level patterns. In this case, a camera solution based on a Raspberry Pi Camera Module 2 was used to generate the data for training a CNN. In order to generate as much and as varied data as possible to train the CNN, several processes were filmed and the individual frames were automatically labelled via a separate interface triggered by the defect generation. The chosen camera system is low cost but offers a resolution of up to 8 MP. However, it was found that the resolution had less of an impact on the defects detected than the illumination, so the full capabilities of the camera were never used. This can be explained via the use of several filters and abstraction by means of a Sobel filter, depending on the model chosen. If, for certain applications (e.g., very small defects), the resolution were to be blamed for the poor performance, this approach would allow it to be increased simply by modifying the code, but in the setting chosen, this was not necessary, and both the file size and processing speed were favored.

There are already many well-known CNN model architectures [85] that can be used to classify the different tasks [86] and have been proposed in several studies as already discussed. In this study, a custom CNN has been proposed whose architecture is based on a standard model that includes Conv2D layers, MaxPooling2D layers, flattening, and fully connected dense layers to output two possible classes, indicating the model’s prediction of the input image class (0 or 1). The basic architecture was then gradually adapted and extended to achieve continuous performance improvements. To avoid class imbalance, the weighted cross-entropy loss function was added, increasing the accuracy from 72% for the base model to 98%. This method has already been proposed by Cui et al. [78] and leads to the avoidance of overfitting in the case of class imbalance caused by a large number of frames without defects and a relatively small number of frames with a detectable defect. In the first step, the standard cross-entropy values were calculated, followed by the extraction of each class label, resulting in the output of the highest value on the tensor axis. The (predefined) weights are then determined according to the class labels. The pre-defined weights provide the ability to highlight classes during training. The next step was to add learning rate scheduling to the model architecture. This method of cyclic learning was introduced by Smith [83] and results in an increase in the learning rate with each batch of data by starting at 0.001 for the first five epochs and then increasing exponentially. Accuracy has been improved by a further 0.5% to 98.5%. The scheduler is based on a mathematical decision of the fastest loss minimization. The final adjustment to the model architecture was the introduction of a Sobel filter which resulted in a final accuracy of over 99%. The simulated defects leave edges in the powder bed, and these sudden image discontinuities are detected via this edge detection [87]. The assumption was that edge detection would be well suited to this, as the abstraction makes defects visible rather than the actual components in the powder bed. The direction in which the intensity increases the most is represented by an image gradient. Consequently, a discrete variation is used to distil the concept of a derivative or gradient. Greyscale conversion is performed before the images pass through two directional Sobel filters stacked on top of each other.

The results of this study show that with the implemented method of artificially generating coating defects, a large dataset of defect-free and defective frames could be generated to successfully train a custom CNN for classification with very high accuracy. Manual comparison of the automatically labelled data with real labels has shown that random defects are correctly labelled. This is only true in any chosen setting and after manual validation—while an educated guess can be made about the detection rate, the best results are only expected when a manual check is performed. As the defects tend to occur randomly, the likelihood of a recurring defect being mislabeled is low—however, if it were to occur, the impact on a real-use case would be proportional to the severity of the mislabeled defect. While a decrease in the accuracy is likely to be small, a serious problem that goes undetected because it was mislabeled during training could affect the entire manufacturing process. A continuous increase in the accuracy was observed by tuning the hyperparameters and by adding and combining previously presented and well-established additional functions. This approach has clearly shown that it is not necessary to use a particularly high-resolution industrial camera system for monitoring; as long as the camera is positioned in the right place, the interfaces for the data transfer are correctly defined and the CNN architecture is properly designed.

7. Conclusions

The evaluation of the results clearly shows that the application of the custom CNN developed in this study is capable of detecting the artificially generated defects with an accuracy of over 99%, and a further step has been taken towards possible process monitoring. It has once again demonstrated and confirmed that the application of deep learning is suitable for monitoring industrial processes and providing data that can then be used in further steps to actively intervene in the process. As it is otherwise hardly possible to intervene in the running process, the developed mechanical set-up shows a new possibility to simulate defects. Compared to the experimental sensor-based methods of process monitoring, this study proposes an effective method of defect detection and process monitoring that only requires sufficient training data with many different component geometries and scenarios of the process. This approach proved capable of providing real-time data analysis via the CNN and an online interface. This is followed by a discussion of the next steps in a complete process monitoring system to define the parameters responsible for the failure and derive appropriate countermeasures and continuous interaction with the in situ process control to modify the process inputs based on the measurements. However, this would require intervention in the machine parameters such as the active control of the temperature or exposure strategies, which was not part of this study.

Once again, deep learning computer vision technology has demonstrated its ability to monitor industrial processes without human intervention. However, it should be emphasized that this approach has been adapted to the current PBF-LB/P system and that, from today’s perspective, it is not possible to say how the CNN would react if a different additive manufacturing technology were used and whether the high error detection rate could then be maintained. It would be useful to further investigate the use of deep learning to monitor the ongoing process, including additional data such as physical influences or recordings of other AM processes, to compare defect detection results, or possibly to implement online parameter optimization.