A Deep Learning Labeling Method for Material Microstructure Image Segmentation

Abstract

In the existing deep learning modeling process for material microstructure image segmentation, the manual pixel labeling process is time-consuming and laborious. In order to achieve fast and high-accuracy modeling, this work proposes a convenient deep learning labeling method and a workflow for generating a synthetic image data set. Firstly, a series of label templates was prepared by referring to the distribution of the material microstructure. Then, the typical textures of different microstructures were box-selected in the images to be segmented to form texture templates. The manual pixel labeling was simplified to the box-selection of the typical microstructure texture. Finally, a synthetic data set can be generated using the label and texture templates for further deep learning model training. Two image cases containing multiple types of microstructures were used to verify the labeling method and workflow. The results show that the pixel segmentation accuracy of the deep learning model for the test images reaches 95.92% and 95.40%, respectively. The modeling workflow can be completed within 20 min, and the labeling time that requires manual participation is within 10 min, significantly reducing the modeling time compared to traditional methods where the labeling process may take several hours.

1. Introduction

The direct correlation between a material’s microstructure and properties necessitates precise and rapid microstructural characterization to elucidate the process–structure–property relationship [1,2]. This is particularly evident in steels, where microstructural variations induced by changes in composition and processing can significantly alter mechanical properties. Microscopy remains a fundamental tool in material characterization, with techniques like optical microscopy (OM) [3], scanning electron microscopy (SEM) [4], and electron backscatter diffraction (EBSD) [5] being standard. Despite its limitation in pixel information and microstructural resolution, OM is preferred for its cost-efficiency and ease of use. While traditionally conducted by experimentalists, image analysis is often constrained by human limitations in efficiency and quantification [6], challenging the scalability of characterization efforts. As the volume of required characterizations escalates, these constraints underscore the need for more automated and reliable analysis methods to keep pace with material science demands. In the past, semi-automated digital image processing methods were employed to segment metallographic images of materials [7,8]. Nevertheless, these methods relied solely on grayscale values and boundary features, making them less suitable for complex microstructural scenarios.

Deep learning (DL) methods, with convolutional neural networks (CNNs) as a representative example, have constituted a significant methodological breakthrough in artificial intelligence over the past few decades [9]. This is primarily attributed to their robust feature extraction and recognition capabilities when applied to image data [10], and they have found extensive applications in areas such as autonomous driving [11] and medical image analysis [12], among others. Utilizing CNNs to characterize material microstructure images allows for rapid and quantitative identification and statistical analysis [13]. Chowdhury et al. harnessed deep learning techniques to extract features from microstructure images and applied them for dendrite classification [14]. Kondo et al. employed CNNs to predict ionic conductivity in ceramics [15]. Warmuzek et al. utilized deep learning methods to classify several morphological categories [16]. These endeavors have demonstrated the capacity of CNNs to extract features from complex microstructures. More recently, many studies have leveraged encoder–decoder network architectures to achieve the semantic segmentation of material microstructure images [17]. The segmentation results depict the proportion and distribution of various structures in the images, which is vital for characterization. Ma et al. employed the DeepLab architecture to segment microscopic images of Al-La alloys [18]. Azimi et al. utilized a fully convolutional neural network (FCNN) to classify various steel microstructures [19]. Jang et al. similarly used an FCNN architecture for acicular ferrite segmentation [20]. Ostormujof and colleagues implemented the U-Net architecture for segmentation in dual-phase steel EBSD maps [21].

Since the CNNs used for semantic segmentation belong to supervised learning, labeling is a crucial step, and its accuracy directly impacts the model’s performance. In the modeling workflow mentioned in the previous works, the manual pixel labeling process is inevitable. This process requires experimentalists to use different colors to annotate different structures on training images and then extract these annotations as labels. Typically, CNN model training requires the dataset on the order of 10³ images and their corresponding labels. Although data augmentation techniques [22] can be employed, manual pixel labeling remains the workflow’s most labor-intensive and time-consuming aspect. Additionally, the microstructures of metallic materials are complex, with different structures intermingling and potentially lacking distinct boundaries [23], making manual labeling challenging.

Regarding labeling methods and dataset augmentation, Ma et al. developed a data augmentation method based on phase-field simulation to generate the synthetic grain boundary image data of pure steel [24]. Stuckner et al. utilized a pre-trained model, which was trained on a dataset with 100,000+ images, to reduce the data required for the subsequent training of semantic segmentation models [25]. Shen et al. used EBSD results as labels for SEM images, but this method has high equipment requirements and may not apply to OM images [26]. In order to solve the data amount requirements of deep learning models and the corresponding labeling problems, using synthetic image data to train deep learning models is a feasible method [27]. Guan et al. used a generative adversarial network (GAN) to generate synthetic medical images for DL model training and achieved similar precision compared to authentic images [28]. Richter et al. used synthetic game images to train deep learning models for real-world image segmentation [29]. Therefore, training deep learning models using synthetic data can reduce the labeling workload. However, only some works propose practical methods for generating synthetic image data for material microstructure segmentation.

In conclusion, quick and reliable metallographic analyses on a large number of samples or measurement fields are essential for testing and characterization needs in materials’ production and research. The CNN modeling process in the existing works mainly relies on manual pixel labeling, which is time-consuming and laborious. Therefore, there is an urgent need to develop more convenient and time-saving labeling methods.

Based on the above background, in order to optimize the labeling method and satisfy the dependence of model training on the amount of data, this work proposes a deep learning labeling method for material metallographic image segmentation and a workflow for generating synthetic microstructure images.

2. Methods

Material microstructures can be recognized due to their distinctive textures in metallographic images. For instance, martensite exhibits the lath-like textures of [30], pearlite exhibits the striated textures [31], and ferrite exhibits the white-bright textures, among others. Training a CNN model involves enabling the model to learn and extract features from various structural textures. Therefore, we believe that model training can also employ synthetic images containing authentic structural textures. In light of this, this study introduces a box-selection labeling approach for structural textures coupled with a workflow for generating synthetic microstructure images based on texture and label templates.

2.1. Cropping–Segmentation–Recomposition Strategy

CNN architectures require fixed input image sizes. Typically, a metallographic image has dimensions of 2560 pixels × 1920 pixels. However, current graphics processing unit devices have limited memory, making it challenging to directly train deep learning models with images of such large dimensions. Therefore, this study employs a cropping–segmentation–recomposition strategy, as illustrated in Figure 1. Firstly, the target image is cropped into smaller-sized images for segmentation. Subsequently, the segmentation results are reassembled into the original image size. The overlap-tile strategy was adopted to mitigate the information loss at the edges caused by cropping [32]. Only the central portion of the recognition results is retained to ensure accuracy. The choice of the cropping image size is crucial, as it should be large enough to capture the structural texture information in the image yet small enough to maintain training and recognition efficiency. After weighing, this study opts for a cropping image size of 160 pixels × 160 pixels while retaining the central 150 pixels × 150 pixels portion after recognition. As for the boundary and corner positions, this work also uses a symmetric operation [18,32]. The boundary and corner positions are expanded by 10 pixels to the outside of the image through symmetrical and rotational operations to ensure that the boundary of the original image is precisely preserved when retaining the central 150 pixels × 150 pixels portion.

2.2. Workflow of Producing Synthetic Image Data Set

Figure 2 illustrates a comparison between the conventional CNN segmentation model construction process and the method introduced in this study, with both approaches including stages for data preparation and model training. In the diagram, the red text highlights the manual labeling steps that necessitate human input. Traditional methods rely on manual pixel labeling, a time-intensive process where individuals color-code different structures within the original images to create labels. These labeled images are then utilized for training deep learning models.

This study, however, implements a more efficient box-selection method for labeling. Here, researchers quickly identify and encircle distinct textures of various microstructures with bounding boxes on the original images. This approach substantially cuts down on the labor and time traditionally required. Utilizing the textures enclosed by these boxes, the texture templates can be generated. These templates are then assembled to produce synthetic microstructure images, which serve as the basis for training deep learning models. Furthermore, the model’s segmentation results can be used to identify incorrectly segmented areas, which are then reselected with bounding boxes to provide additional texture information. Retraining the model with this supplementary texture data improves its ability to segment specific microstructural textures accurately. Subsequent sections will delve into the specifics of texture and label template creation, as well as the workflow adopted in this study.

2.2.1. Texture Template

The labeling method proposed in this work enables the DL model to learn the typical textures of various types of similar microstructure and segment textures when applying the model. The precision of the box-select labeling process is crucial. Experimentalists must ensure that the selection encompasses a comprehensive representation of the various typical textures within the same microstructural category.

Texture templates are derived from these selected representative structural textures. The heterogeneity of texture distribution across images necessitates variable bounding box sizes during the labeling process. These sizes may be smaller than the actual template dimensions, prompting an essential standardization of the diverse textures into uniform template sizes, as delineated in Figure 3. The standardization process involves flattening and aligning the typical structural textures and cropping them to conform to the predetermined template size, thus creating a series of texture templates. Each box-selected microstructural texture yields a distinct texture template. When selecting bounding box sizes during labeling, they should approximate or not exceed the specified template size. In this study, the texture template size matches the cropping image size of 160 pixels × 160 pixels.

2.2.2. Label Template

We simulated the distribution of the material microstructure in the images and manually drew 24 label templates, as shown in Figure 4. The black parts in the templates represent one type of microstructure, while the white part represents the other. Their distribution forms include two types of organizations adjacent to each other, one organization surrounded by another, and regular or irregular boundaries. The current label templates are suitable for cases where only metallographic images contain two types of microstructures, consistent with the cases used in this work. Suppose there are three or more types of microstructures. In that case, two kinds can be selected each time to generate synthetic images and ensure that each microstructure is selected or to redraw new label templates. It is worth noting that although drawing these label templates is also manual and time-consuming, they can be reused once made, so making the label template is not counted in the modeling time.

2.2.3. Generation of Synthetic Image Dataset

After obtaining the label and texture templates of different microstructures, synthetic images can be generated following the workflow in Figure 2:

• Randomly select one texture template from each microstructure texture template group, separately.
• Randomly select a label template. After that, different label parts in the label template are replaced with texture templates of different microstructures.
• One image can be derived into four images through common data augmentation operations such as rotation.

Therefore, four synthetic microstructure images and their labels can be obtained after each round of such operations (from randomly selecting each template to generating synthetic images). By setting the round number of operations to be performed, any number of synthetic images can be obtained to meet the data volume requirements for CNN model training.

2.2.4. Texture Supplement

The unique labeling method adopted in this study may result in the trained model failing to accurately identify textures absent from the initial set of templates. To address this, a supplementary training approach was incorporated into the workflow. In cases where the segmentation results show inaccuracies, the experimentalists can manually identify these poorly segmented areas to enrich the existing collection of texture templates. By increasing the randomly selected weights of these newly added textures, an augmented synthetic dataset can be generated specifically for additional training purposes. This strategy aims to enhance the model’s ability to generalize across various textures and improve its accuracy in segmenting specific microstructural features.

3. Cases and Analysis

Two cases of metallographic images containing two types of microstructures were used to verify the effectiveness of the labeling method and workflow. The U-net [30] model was selected as the DL model, whose architecture is shown in Figure 5. U-net is a classic encoder–decoder-type deep learning model that can achieve the pixel-to-pixel semantic segmentation function. Each stage of the encoder consists of convolutional and ReLU layers, and each stage of the decoder adds a concatenation convolutional layer. A dropout layer is added to the bottom bridge part to prevent overfitting. The model uses the max-pooling layer for down-sampling and the transposed convolution layer for up-sampling. The first two digits in Figure 5a represent the size of the image processed at this stage, and the third digit represents the number of convolution kernels. The model’s input is a 160 pixels × 160 pixels gray-scale metallographic image, consistent with the cropping–segmentation–recomposition strategy’s small size. The output size of the model is also 160 pixels × 160 pixels, where each pixel is given one of two preset categories.

In total, 80% of the images in the synthetic data set and their labels were randomly selected as the training set for U-net model training, and the remaining 20% of the images as a validation set for the model test. It is worth mentioning that the validation set still consists of synthetic images, which may not represent the model’s segmentation accuracy in authentic metallographic images. Therefore, an authentic metallographic image not included in the training set was used to verify the model’s segmentation accuracy for each case. The segmentation accuracy is defined as follows:

$$\mathrm{Accuracy}=\frac{{\mathrm{N}}_{\mathrm{right}}}{\mathrm{N}}$$

(1)

where N represents the total number of pixels in the image, and N_right represents the number of pixels correctly classified according to the ground-truth label. The segmentation accuracy for a specific category of microstructure is defined as:

$${\mathrm{Accuracy}}_{\mathsf{\alpha }}=\frac{{\mathrm{N}}_{\mathsf{\alpha }\_\mathrm{right}}}{{\mathrm{N}}_{\mathsf{\alpha }}}$$

(2)

where N_α represents the total number of α-type microstructure pixels, and N_{α_right} represents the number of α-type pixels correctly classified.

The untrained U-net model uses random initial parameters and weights. The solver uses the Adaptive Moment Estimation (Adam) algorithm and the mini-batch size is set to 8. The initial learning rate is 0.001 and the learning rate is adjusted to 1/10 of the previous value every 10 epochs during training.

3.1. Ferrite–Pearlite Case

The common ferrite–pearlite microstructure images were first used as a verification case. The composition of the steel sample is shown in

Table 1. Chemical composition of ferrite–pearlite specimen (wt%).

C	Si	Mn	Fe
0.195	0.257	0.4	Balance

.

First, we box-selected the two types of microstructure textures in two metallographic images and generated ten ferrite texture templates and nine pearlite texture templates, respectively. Although the gray value and texture of pearlite and ferrite are pretty different, the grain boundaries between ferrite grains are similar to the gray value of pearlite. In order to distinguish them, when making texture selections, we deliberately selected a few ferrite textures containing grain boundaries and generated 2000 synthetic images according to the workflow in Figure 2. The examples of the texture template and the synthetic images are shown in Figure 6. The U-net model was trained on a NVIDIA RTX3070 GPU device for 30 epochs.

Table 2. The time consumed by each process of ferrite–pearlite segmentation model modeling.

Process	Time
Box-selection Labeling	5 min
Generating synthetic image set	3 s
Model training	10 min
Total	15 min

shows the consumed time for labeling and model training.

After training, the model’s segmentation accuracy in the training and validation set reached 97.73% and 97.20%, respectively. An authentic metallographic image was used to test the model’s segmentation ability. Figure 7 shows the original image and its segmentation results. It is worth mentioning that the ground-truth label used the manual pixel labeling method, which took about 30 min to label a single image. The workflow of the traditional method shown in Figure 2 requires at least several images to be labeled, which may take at least a few hours. As a comparison, the whole modeling time of this case only takes about 15 min, significantly saving working time and human resources’ consumption.

The overall segmentation accuracy and the segmentation accuracy of each microstructure are calculated according to formulas (1) and (2), as shown in

Table 3. The time consumed by each process of ferrite–pearlite segmentation model modeling.

Microstructure Type	Accuracy (%) First Training	Accuracy (%) After Supplement Training
Ferrite	97.05	98.15
Pearlite	72.77	85.48
Total	93.11	95.92

.

It can be found that the overall segmentation accuracy of the model in real metallographic images reaches 95.92%, which proves that this workflow is effective. However, mis-segmentation still exist in the grain boundary region (ferrite) and the red frame (pearlite) region. Figure 8 shows the enlarged red frame mis-segmentation area.

It can be found that the pearlite recognition in this region is poor because the pearlite texture group built at the beginning does not contain this texture. Therefore, according to the texture supplementation process, the pearlite textures in this region was box-selected to form new texture templates, as shown in Figure 9a. More ferrite texture templates containing grain boundaries were also added, as shown in Figure 9b. Then, the supplementary texture appearance weight was increased ten times to form 800 new synthetic images. Finally, this new image set was used for ten epochs (5 min) of supplement training of the model. Figure 6d and Figure 8d show the new model’s segmentation results. The overall accuracy increased to 95.92% and the segmentation accuracy for pearlite increased to 85.48%. This proves that texture supplementation can improve the model’s segmentation ability for specific textures with sparse samples.

3.2. Bainite–Martensite Case

Another more complex image case of the bainite–martensite microstructure is also given to verify the effectiveness of the workflow.

Table 4. Chemical composition of bainite–martensite specimen (wt%).

C	Si	Mn	Cr	Ni	Cu	Mo	Fe
0.136	0.243	1.07	0.461	0.687	0.24	0.436	Balance

shows the composition of the steel samples used in this work. This group of images is gained from a series of continuous cooling transformation (CCT) diagram experiments. We obtained metallographic images containing only bainite or martensite microstructures based on different cooling rates. However, only a few images containing both types of microstructures are obtained near the critical cooling rate, and the amount of data is challenging to meet the deep learning model training requirement. The problem of insufficient data can be solved using the labeling method and synthetic image generation workflow in this work.

We box-selected the textures in two metallographic images containing only bainite and martensite structures separately and obtained ten bainite texture templates and twelve martensite texture templates. Based on these texture templates and label templates, 2400 synthetic images were generated, as shown in Figure 10. The model was trained on a NVIDIA RTX3070 GPU device for 30 epochs. The consumed time for labeling and model training is shown in

Table 5. The time consumed by each process of bainite–martensite segmentation model modeling.

Process	Time
Box-selection Labeling	5 min
Generating synthetic image set	3 s
Model training	12 min
Total	17 min

.

After training, the accuracy of the model in the training and validation set reached 96.73% and 96.20% respectively. An authentic metallographic image is used to test the model segmentation effect. Figure 11 shows the original image and its segment result.

Compared with the ground-truth label, the overall segmentation accuracy of the model is 95.40%, which proves that this workflow can be applied to the segmentation modeling of a complex microstructure. The segmentation accuracy for bainite is 98.10%, and the accuracy for martensite is 80.87%. The martensite segmentation accuracy is relatively low due to the difference in the total amount of the two types of microstructure and the ground-truth marking errors caused by unclear boundaries between these microstructures. The ferrite–pearlite and bainite–martensite cases prove that the labeling method proposed in this work can be used for fast and high-accuracy deep learning modeling for material microstructure segmentation.

4. Conclusions

• This work proposes a box-selection labeling method coupled with a synthetic image generation workflow for microstructure segmentation’s deep learning modeling process.
• The new approach used the box-selection labeling method to replace the traditional time-consuming and laborious manual pixel labeling process. Meanwhile, the synthetic microstructure image generation workflow can satisfy the dependence of model training on the amount of data.
• The texture supplement method can improve the model’s segmentation ability for specific textures with sparse samples.
• The modeling process can be completed within 20 min, and the labeling time requiring manual participation is within 10 min. The time and human resources consumed could be significantly saved compared to a conventional workflow.
• Two metallographic image cases of steel materials containing different microstructures were used to verify the effectiveness of the marking method and workflow. The results show that the overall segmentation accuracy reached 95.92% and 95.40%, respectively.

5. Limitations and Future Works

In the present work, the proposed labeling method is based on the distinct grayscale values or textures exhibited by different microstructures. However, it is not applicable to microstructures with identical textures or those that are too small to be effectively discriminated. For instance, this method is not suitable for dual-phase steels (where both austenite and ferrite structures display similarly bright textures), grain boundaries (network-like distribution), or secondary phases (merely a few micrometers in size). Future developments should aim to devise more rapid and convenient labeling techniques, liberating experimentalists and researchers from the burdensome and inefficient repetition labeling inherent in current practices.