Synthetic Image Data Generation for Wiring Harness Component Detection Using Machine Learning ^†

Abstract

Machine learning is a powerful tool for computer vision tasks in manufacturing, as features are automatically extracted and a high variety of components or failures are reliably detected. A focal prerequisite for high-performing machine learning models is a database that is large in quantity as well as quality, and representative for the computer vision task in the manufacturing environment. In addition, manufacturing applications require a domain-specific dataset. Thus, we generated and integrated synthetic data for object detection using convolutional neural networks, specifically for wiring harness component detection. A synthetic data generation pipeline for images was developed and implemented. Experiments were conducted to assess the domain gap between synthetic and real images and to determine factors that are beneficial to synthetic data generation. The experimental findings demonstrate relevant training approaches to integrate synthetic data, factors that have a positive impact on training, and high-performance results comparable to using real data only.

1. Introduction

The potential of machine learning for manufacturing applications has been widely addressed and proven by researchers [1]. Machine learning models automatically extract relevant features from Euclidian and non-Euclidian data. Convolutional neural networks (CNNs), a type of machine learning model, are well known for processing image data as they can exploit shift-invariance and local connectivity in image data [2]. However, manufacturing applications are highly specific, meaning that off-the-shelf datasets lack relevance for CNN training. Thus, a new computer vision task for a manufacturing application results in the need to create a dataset with a latent feature space specific to the application.

The challenge of data scarcity and high resource investment in database creation is also found in the manufacturing application of wiring harness component detection. Researchers addressing this application generate CNNs by investing a high amount of time and manpower into generating and labeling images. To increase the efficiency of data generation, synthetic data are generated and integrated into model training. This facilitates the rapid development of robust CNN models and the adaptation of existing models to data drift in real-world production environments. Synthetic data are created in a virtual environment by, e.g., reusing computer-aided design (CAD) models or utilizing generative models. Therefore, using synthetic data reduces the amount of time and the costs involved because physical setups, manual image capturing, and manual labeling are no longer necessary. We present the synthetic image data generated and processed in this study.

The remainder of this article is structured as follows. First, an overview of the state of the art of traditional and machine-learning approaches for wiring harness component detection is provided. Second, the data generation and processing pipeline for wiring harness images is presented in detail. Experiments are conducted to evaluate the value-added through synthetic data in contrast to using real data only. Third, the last section concludes this article and gives an outlook on future research endeavors.

2. Machine Vision for Wiring Harness Component Detection

The wiring harness consists of thousands of components such as wires, connectors, clips, tubes, tapes, and grommets [3]. Assembled into the final product, wiring harnesses serve to connect sensors, actuators, and control units, e.g., in vehicles, planes, or electronic devices. The detection of the components in wiring harness manufacturing is difficult to realize, due to the presence of high component variety, deformable and rigid objects, and product complexity [4]. A door wiring harness is shown in Figure 1.

2.1. Traditional Machine Vision

A variety of proposed solutions for wiring harness component detection stem from traditional computer vision algorithms. Focus is often directed towards assembled wires due to their deformable character and high occlusion between wires. Thus, a combination of multiple algorithms is proposed. Algorithms for thresholding [5], manual feature extraction [6], and edge detection [7] are often applied for component detection. The addition of algorithms for template matching facilitates the determination of the presence and position of tape, clips, wire bundles, and wheel-shaped components [8]. While the proposed solutions are successfully applied to the individual research use cases, the robustness to data drift, e.g., inducted by environmental changes and novel components, is not given. Thus, adapting algorithms to data drift necessitates the adjustment of a multitude of parameters. Matching algorithms focus on wiring harness localization for wire manipulation. Matching algorithms rely on a topological representation form of the wiring harness to determine wire branches and components [9].

2.2. Machine Learning Approaches

Wiring harness branch detection is realized with a CNN processing image as well as depth data, and the experimental findings show the benefit of processing data from several modalities to achieve a high CNN performance [10]. Reference [11] focuses on connector detection and pose estimation using a single-shot detector. These works rely on manual image data collection and labeling. Only a few researchers rely on synthetic data as the primary database for CNN training. The real-time deformable linear objects instance segmentation (RT-DLO) approach addresses instance segmentation for wires using synthetic data [12,13]. A CNN is proposed for semantic segmentation to generate a binary mask for additional algorithms for instance extraction. Synthetic data generation is based on the copy-and-paste approach. Therefore, wires are extracted from existing images and pasted on a variety of complex backgrounds. The results show the successful application of RT-DLO to real data.

Synthetic data can be generated without relying on real data for feature extraction. Wiring harness simulation and rendering allow data generation using digital artifacts. Wire instance segmentation of control cabinet wires and high-voltage wires is accomplished using models trained with synthetic data [14]. The synthetic data generation process encompasses a model generator to generate geometric models of wires considering their deformable character; scene generation, as the wires to be detected are placed in small load carriers; and scene rendering using Blender. A similar approach was applied in [15] to generate synthetic point clouds for point cloud segmentation. The simulation involved individual wires and wire bundles in an assembled state, the scene composition entailed wire assembly on an assembly board, and the rendering was implemented using Blender. The researchers investigated different training approaches to achieve a high performance. In [16,17], semantic part segmentation was conducted using geometric deep learning trained with meshes deducted from CAD data.

3. Synthetic Data Generation

Synthetic data generation is used to train a CNN for semantic segmentation of images containing wire harness assemblies. Following our previous works, we implemented and assessed semantic segmentation for wiring harness assembly states [18]. Previous research investigated the feasibility of machine learning-based component detection. We investigated the added value through synthetic data generation for semantic segmentation. The classes are (1) ‘connector’, (2) ‘clip type 1’, (3) ‘clip type 2’, (4) ‘clip type 3’, (5) ‘grommet’, (6) ‘fully taped wire bundles’, (7) ‘spiral taped wire bundles’, (8) ‘untaped wire bundles’, and (9) ‘background’. Synthetic data generation, the ideal approach to synthetic data integration in the training process, and the determination of impacting factors for synthetic data generation are explored in this study. The real dataset for wire harness assembly states has been introduced in previous works [18].

3.1. Data Generation

The synthetic data generation approach and required input are visualized in Figure 2. The main steps are ‘wiring harness model generation’, ‘scene composition’, and ‘rendering’. The starting point for wiring harness model generation was a structured bill of material (SBOM) with the assembly sequence and component material number. Furthermore, a CAD database with a geometric representation of the components was mandatory. The components possess meta information pertaining to the color, material, and label. As wiring harnesses are customizable products, a configuration list provided information on which components need to be retrieved from the database for specific wiring harness configurations. In conjunction with the SBOM, different assembly states, specifically work in progress and finished assembly, were derived. The wiring harness models were generated in accordance with the proposed steps using the tools ‘Siemens NX’ [19] as well as ‘IPS Cable Simulation’ [20]. The latter was required to simulate the deformable characteristics of wires and wire bundles.

Next, the manufacturing scene was created. The wiring harness model required an expansion to include the manufacturing equipment and environment. The background of the scenes can be composed of digital models of the manufacturing or background images. We used CAD models of the manufacturing equipment and a variety of images of manufacturing backgrounds. The final model, consisting of wiring harnesses and the scene, was then enriched by rendering relevant parameters. Therefore, a material library was required to assign material and texture information to individual components according to component meta information. Scene composition and rendering were realized using the tool ‘Blender’ [21].

For image generation, a parameter space for lighting, camera position, and background was created in this study. Lighting sources were located in the scene and varying lighting intensities were defined. Furthermore, the camera for image synthesis was set to match the camera of the physically existing setup. A parameter space for the camera location was set to capture the scene from different perspectives and to capture images of different light intensities. The light intensity was set in Blender and parametrized with 5, 25, and 50 W. Defined parameter spaces were implemented since the field of view was known for manufacturing applications. Lighting conditions did not vary significantly in the manufacturing environment. Therefore, a random function was implemented to select the camera location within the parameter space and for background retrieval. The background library consisted of a real image of the physically existing laboratory setup and background images of manufacturing environments retrieved from the internet. Thus, the background possessed contextual relevance. We used the rendering engine ‘Cycles’ provided by Blender. The output was images with a size of 1120 × 640 pixels. As the images contained metadata on the components and label annotations, the masks for semantic segmentation were automatically extracted. Masks were generated simultaneously with the image and saved as .jpg according to a naming convention to maintain an unambiguous assignment of image and mask for training. The images and masks generated for a wire harness assembly state are shown in Figure 3.

3.2. Data Processing

The generated images and masks were downscaled to 384 × 384 pixels and data augmentation was implemented before training using the library “Albumentations” [22]. Data augmentation techniques applied were horizontal flip, shifting, scaling, rotation, image cropping, and padding. Moreover, contrast, brightness, gamma, blur, and hue saturation were randomly applied. The hold-out method was applied by splitting the dataset into a training dataset for training and validation according to a ratio of 80:20. The size of the test dataset was limited to the amount of real data available. A total of 166 real images with associated masks were available. The resulting number of images according to experiments conducted is shown in

Table 1. Overview of experiments and datasets.

No	Experiments	Training and Validation		Testing
		Domain	Images	Domain	Images
1	Benchmark	Real	132	Real	34
2	Domain gap	Synthetic	10,728	Real	166
3	Parameter study	Synthetic	445	Real	166
4	Mixing	Synthetic Real	747 83	Real	83
5	Fine-tuning	Synthetic Real	10,728 83	Real	83

. Semantic image segmentation relied on the U-Net and LinkNet architecture with the backbone models SEResNet50 and EfficientNetB3, according to [23]. Training was performed for 50 epochs, using the Adam optimizer, adaptive learning rate adaptation, and early stopping. Hyperparameters were optimized using grid search for the parameters batch size (2, 4, 8, 12), learning rate (0.001 and 0.0001), architecture, and model backbone. The hardware used was Intel Core i7-8565U CPU and NVIDIA GeForce RTX 2080TI.

4. Results

Experiments were conducted to understand the domain gap between synthetic and real data, as well as the added value provided by synthetic data for training. The goal for synthetic data implementation is to rely on as little real data as possible while achieving an equally high or higher performance. The results were obtained on the evaluation metrics of intersection over union (IoU) and F1 score. The IoU evaluates the spatial accuracy. The F1 score allows the precision and recall to be concluded. The first experiment was designed to establish a benchmark for comparative analysis when utilizing the generated synthetic data. This benchmark serves as a reference for evaluating the performance and generalizability of the following experiments. Training resulted in a mean IoU score of 0.7976 and a mean F1 score of 0.8375.

4.1. Domain Gap and Parameter Study

The experiment ‘Domain gap’ was conducted to identify the domain gap between real and synthetic data. CNNs were trained with synthetic data only and tested on real data. The training curves for the Unet architecture and EfficientNetb3 backbone trained with solely synthetic data reached an accuracy of 99.08% and a mean IoU of 0.6248. Contrastingly, testing with real data resulted in a mean IoU of 0.5531. A domain gap between the real dataset and the synthetic dataset was observed. The overview of IoU by classes is depicted in

Table 2. Evaluation of IoU by class.

No	Experiments	Classes
		1	2	3	4	5	6	7	8	9
1	Benchmark	0.900	0.794	0.794	0.647	0.840	0.633	0.746	0.782	0.993
2	Domain gap	0.692	0.484	0.777	0.675	0.220	0.248	0.452	0.482	0.975
3	Parameter study	0.671	0.618	0.235	0.265	0.378	0.281	0.466	0.125	0.982
4	Mixing	0.874	0.181	0.664	0.613	0.845	0.543	0.748	0.515	0.991
5	Fine-tuning	0.894	0.759	0.747	0.687	0.880	0.628	0.787	0.506	0.991

. The classes (1) connector, (4) clip type 2, (5) clip type 3, and (9) background achieved an IoU over 0.5. The class background displayed a superior performance because of the underlying class imbalance. The classes (5) grommet and (6) untapped wire bundle achieved an IoU score below 0.25. A comparison between ground truth and prediction mask shows the misclassification of background objects as a grommet and incomplete segmentation of untaped wires, especially at the edges of the objects (Figure 4). Thus, the model trained with synthetic data could not entirely detect all wire harness components of the real dataset.

To create a more in-depth understanding of the parameters applied during synthetic data generation, the parameters were determined according to the ‘Design of Experiment’ methodology. Parameters of the process steps ‘scene composition’ and ‘rendering’ were selected for the experiment, specifically light intensity, color, texture, and background. The parameter for light intensity was defined at three levels: 5, 25, and 50 W. The light intensity was chosen because this factor significantly impacts image generation time. Whereas an image with 5 W light intensity was generated in approx. 27 s, the 50 W image took 179 s to generate. The background was differentiated to distinguish between the real background and a background retrieved from the internet. Furthermore, images were generated with and without color and texture information. A full factorial design was employed. Thus, 24 datasets were created for all possible combinations of factors and levels. In the experiments, 14 blocks were tested, whereby the number of blocks and combinations reflected the hyperparameter optimization runs. Each dataset consisted of 355 images for training and 90 images for validation. The experiments were conducted with the LinkNet architecture.

Regression analysis and main effect analysis results showed the relevance of the factors of light intensity and color information. Figure 5 illustrates the main effect plot, showing that light intensity has a slight impact on IoU, while color has a significant impact on the performance. Texture and background have a minor impact on the performance. Consequently, the texture information can be neglected during rendering and the background selection does not require a systematic approach. The highest performance was a mean IoU of 0.2645 and a mean F1 score of 0.3337 with a parameter setting of 5 W, color, texture, and random background.

4.2. Mixing and Transfer Learning

Mixing datasets and transfer learning were implemented to overcome the domain gap between synthetic and real data. First, the amount of real data was reduced by substituting real data with synthetic data in the training and validation dataset. A mixing ratio of 10% real data and 90% synthetic data for training and validation was applied. The best-performing model resulted in a mean IoU of 0.6440 and a mean F1 score of 0.7000. While the IoU scores of most classes were slightly below the benchmark experiment, Class (2) clip type 1 yielded poor results, since the IoU was 0.181.

Fine-tuning was conducted to transfer the model pretrained with synthetic data to the domain of real data. The best-performing model generated in the experiment ‘domain gap’ was fine-tuned with real data through training with small learning rates. The mean IoU was 0.7496 and the mean F1 score of 0.7849. While classes (4) clip type 3, (5) grommet, and (7) fully taped wire bundles achieved higher IoU scores compared to the benchmark experiment. The class (8) spiral-taped bundles exhibited lower performance. As shown in Figure 6, thin wire bundles pose a challenge for some images in the testing dataset. The performance of the remaining classes is comparable to the performance achieved during the benchmark experiment. The model trained through fine-tuning achieved results equivalent to those of the benchmark model, despite utilizing only half the amount of real data for training. The experiment results indicated the importance of synthetic data for CNN pretraining and the model performance for fine-tuning with real data.

5. Conclusions and Outlook

The potential of synthetic data to overcome the data scarcity for domain-specific machine learning applications was validated specifically in the detection of wire harness components. The proposed synthetic data generation approach was implemented. The experimental results present factors in the steps of ‘scene composition’ and ‘rendering’ that are critical to object detection. The texture and type of background can be neglected, and color information is critical to performance when the CNN is transferred to a real dataset. The identification of critical factors allows faster synthetic data generation because factors without impact are eliminated. Even though a domain gap has been identified, different dataset compositions and training approaches show that the need for real data is significantly reduced for this approach, but it can still achieve a high CNN performance equal to that achieved by training with real data only. By fine-tuning a CNN that has been trained with synthetic data, the use of real data leads to higher results compared to mixing datasets. However, further research is needed to explore other mixing strategies beyond a 1:9 ratio.

The limitation of the proposed synthetic data generation approach and the experimental findings is its applicability to the specific use of wire harness component detection and the dataset used for evaluation. The dataset’s specific features, such as data distribution and sample size, significantly influence the experimental results. It is necessary to improve and extend the approach to synthetic data generation for wire harness component detection. Testing the approach on a bigger dataset validates its robustness and allows for the identification of optimization measures. The generalizability of the approach to other use cases and datasets also needs to be addressed. Extensive evaluation that includes diverse use cases is important to holistically understand the method’s strengths and weaknesses with regard to its implementation in manufacturing applications.