Casting Microstructure Inspection Using Computer Vision: Dendrite Spacing in Aluminum Alloys

Abstract

This paper investigates the determination of secondary dendrite arm spacing (SDAS) using convolutional neural networks (CNNs). The aim was to build a Deep Learning (DL) model for SDAS prediction that has industrially acceptable prediction accuracy. The model was trained on images of polished samples of high-pressure die-cast alloy EN AC 46000 AlSi9Cu3(Fe), the gravity die cast alloy EN AC 51400 AlMg5(Si) and the alloy cast as ingots EN AC 42000 AlSi7Mg. Color images were converted to grayscale to reduce the number of training parameters. It is shown that a relatively simple CNN structure can predict various SDAS values with very high accuracy, with a $R^2$ value of 91.5%. Additionally, the performance of the model is tested with materials not used during training; gravity die-cast EN AC 42200 AlSi7Mg0.6 alloy and EN AC 43400 AlSi10Mg(Fe) and EN AC 47100 Si12Cu1(Fe) high-pressure die-cast alloys. In this task, CNN performed slightly worse, but still within industrially acceptable standards. Consequently, CNN models can be used to determine SDAS values with industrially acceptable predictive accuracy.

1. Introduction

It is well known that the size of dendrites and the secondary dendrite arm spacing (SDAS) strongly depend on the solidification rate of a given material [1,2]. In addition, the chemical composition of the alloys has an additional influence on this structural characteristic [3]. Moreover, some authors have shown a relationship between mechanical properties and SDAS [1,4,5,6,7,8]. Properties of fracture mechanics will also depend on chemical composition, casting defects such as porosity and oxide films [8], and the size and shape of Si or Fe-rich brittle phases [9]. Most authors show a relationship between SDAS and ultimate tensile strengths (UTS) and elongation (E), while many authors indicate that SDAS has no significant effect on yield strength (YS). Moreover, another study indicates that the hardness of the material depends on SDAS, but cannot be described well enough using only this relationship [10]. Consequently, it is reasonable to assume that some material properties can be determined directly from the value of SDAS. Thus, it could be useful to know the SDAS value of the material. In this regard, an automatic method for determining SDAS could be a significant advantage.

The scope of artificial intelligence (AI) is more significant in disciplines such as computer science or electrical engineering than in materials science. However, in the last three decades, many applications can be found in materials science as well. In general, neural networks, as a core algorithm of AI, have been applied in materials science as early as 1998 [11]. Singh et al. estimated the YS and UTS as a function of each of the 108 variables for the steel rolling process. An artificial neural network (ANN) with only two hidden units was used. In another early paper, Hancheng et al. [12] developed a model to predict tensile strength based on compositions and microstructure by using adaptive neuro-fuzzy inference method. Agrawal et al. [13] determined the fatigue strength from steel processing parameters with $R^2$ score of over 97%. Liu et al. [14] determined the Young’s modulus, YS, and magnetostrictive strain using machine learning techniques (ML). The ML approach reduces the computation time by an average of 81.62% compared to standard optimization methods. Yang et al. [15] used ANN with one layer and 8 neurons to predict UTS and E from the heat treatment parameters. Santos et al. [16] predicted UTS of an automotive iron casting using ANN and K-Nearest Neighbor method, which outperformed several other methods, with the highest accuracy reaching 85%. Liao et al. [17] constructed ML algorithms to predict macroshrinkage of aluminum alloys based on their experimental dataset. Additionally, the interested reader is referred to reference [18], which shows several applications of DL in the field of materials science. Furthermore, in [18] and references therein, determination of material properties using DL could be found as well. Herriott and Spear [19] investigate the ability of ML and DL models to predict microstructure-sensitive mechanical properties in additive manufacturing of metals (MAM). ML and DL methods accelerated the prediction of microstructure and mechanical response for MAM.

Given that in recent decades our ability to generate data has far surpassed our ability to make sense of it in virtually all scientific domains [13], the development of DL methods could be of particular benefit in materials science. One process that involves a lot of input data is quality control. Quality control is a fundamental part of many manufacturing processes, especially casting or welding. Unfortunately, manual quality control procedures are often time consuming and error prone [20]. Quality control, SDAS in particular, could also be performed by Light Optical Microscopy (LOM) of polished samples, but SDAS evaluation still relies on manual measurements. Although there exists research tackling the problem of SDAS prediction using ANNs [21], SDAS is not predicted directly from the microstructure image. Instead, the authors predicted SDAS based on processing parameters: pouring temperature, insulation on the riser and chill specific heat, while the dataset was based on numerical simulation results. To the best of the authors’ knowledge, there is currently no literature that determines SDAS directly from the microstructure image. Following the literature review, this research hypothesizes that SDAS could be determined directly from the microstructure image data using DL methods.

2. Related Work

The algorithms available for implementing ML can be broadly categorized into the following types: shallow learning (e.g., vector machine (SVM), decision tree (DT) and ANN) and DL (e.g., CNN, recurrent neural network (RNN), deep belief network (DBN) and deep coding network) [22]. Shallow learning algorithms cannot achieve the same accuracy on different tasks as DL, although they could reduce the computational cost. As shown in [19,23] CNN performs better than standard (2D) ML models such as Ridge, XGBoost regression and the like. A transfer learning-based approach to use convolutional deep networks in [24] is generally superior to all other reconstruction approaches (decision tree, Gaussian random field, two-point correlation, physical descriptor) for most numerically evaluated material systems. CNN has also been used for casting defects recognition in [25]. These authors used 640,000 images to train CNN and also Generative Adversarial Network (GAN) was used to generate even more data. In the present study, image data were used to predict SDAS using CNNs. It is reiterated that the task of determining SDAS is often manual and subjective.

Data used in materials science can be obtained from the following sources: Material properties from experiments and simulations, chemical reaction data, image data and published data [22]. DeCost et al. [26] applied a deep CNN segmentation model to enable novel automated microstructure segmentation applications for complex microstructures. Ferguson et al. [20] proposed a casting defect detection system in X-ray images based on the Mask Region-based CNN architecture. The proposed defect detection system outperforms the state-of-the-art performance for defect detection on GDXray Castings dataset. Chowdhury et al. [27] created an image-driven machine learning approach to classify micrographs. In references therein, classification of precipitate shapes in a two-phase microstructure, the classification of cast iron microstructures and the analysis of precipitate shapes in nickel-based superalloys could also be found. Azimi et al. [23] use a segmentation-based approach based on Fully Convolutional Neural Networks (FCNNs), which are an extension of CNNs, accompanied by a max-voting scheme to classify microstructures. Exl et al. [28] used an ML-approach to identify the importance of microstructure properties in causing magnetization reversal in ideally structured large-grained Nd2Fe14B permanent magnets. The embedded Stoner–Wohlfarth method is used as a reduced-order model to determine local switching field maps that guide the data-driven learning procedure. They used datasets consisting of 700 and 800 experiments. Yucel et al. [29] showed relationships between optical micrographs and mechanical properties (YS, UTS and E) of cold-rolled high-strength low-alloy steels (HSLA) measured in standardized tensile tests. However, the models developed in the latter work are only applicable for a constant initial condition before annealing treatment, i.e., constant composition, constant hot rolling and constant cold rolling parameters. Li et al. [24] used a transfer learning approach for microstructure reconstruction and structure-property prediction. Pokuri et al. [30] show a data-driven approach for mapping microstructure to photovoltaic performance using CNNs. The VGG-16 CNN-based architecture achieved a test accuracy of up to 96.61%. DeCost et al. [31] trained an SVM to classify microstructures into one of seven groups with accuracy greater than 80% with 5-fold cross-validation.

3. Materials and Models

3.1. Aluminum Alloy Samples

In the present study, samples for training were cut and polished from three different materials cast by three different methods: high-pressure die-cast EN AC 46000 AlSi9Cu3(Fe), gravity die-cast EN AC 51400 AlMg5(Si) and EN AC 42000 AlSi7Mg alloy cast as ingots. The chemical composition of all the alloys used conformed to DIN EN 1706 [32]. The die-cast alloys were supplied in the form of commercial ingots. The ingots were melted in the gas furnace, then degassed with argon and transferred to the holding furnace at a temperature of 700 ± 10 °C. The melt was then automatically ladled into the shot sleeve and injected into the die cavity in the case of the AlSi9Cu3(Fe) alloy. Cast parts were produced using a cold chamber high pressure die casting machine (HPDC) with an intensification pressure of 800 bar. In the case of AlMg5(Si) alloy, the melt was ladled directly into the metal die and solidified under gravity conditions. Moreover, in the case of AlSi7Mg alloy, the melt was poured directly into the ingot die while exact process parameters and temperatures are not known. It should be noted that the AlMg(Si) and AlSi9Cu3(Fe) samples were cut from medium sized components between 5 and 10 kg.

Additional samples were obtained to test the prediction accuracy. This dataset consists of three materials with chemical composition according to the DIN EN 1706 [32] standard: EN AC 42200 AlSi7Mg0.6, EN AC 43400 AlSi10Mg(Fe) and EN AC 47100 AlSi12Cu1(Fe). AlSi7Mg0.6 was cast by gravity die casting technique, while AlSi10Mg(Fe) and AlSi12Cu1(Fe) alloys were cast by HPDC process. The exact process parameters and casting history for these materials are not known. Polished samples for this group of materials were cut from medium sized cast components between 5 and 10 kg.

All specimens were cut using a classic band saw and polished using standard techniques. Images were taken using an Olympus BX51 optical microscope equipped with an XC30 digital camera.

3.2. Dataset and Image Preprocessing

The dataset is created from a single polished cross section per material. The manually measured values of SDAS for AlSi9Cu3(Fe), AlSi7Mg and AlMg5(Si) alloys were $6·{10}^{-3}$, $29·{10}^{-3}$ and $32·{10}^{-3}$ mm, respectively. Manual SDAS measurements were performed such that the distance between the centers of two secondary dendrite arms was measured perpendicular to the primary arm, Figure 1. In other words, “Method E” from reference [33], was adopted here for manual SDAS measurement. The automated technique proposed here is based on the same SDAS measurement principle. These SDAS values were obtained at 5× magnification. To obtain more SDAS values for training, additional SDAS values were derived using different magnifications on the microscope. Thus, the equation for the derived SDAS values could be obtained with the following expression:

$$\mathrm{SDAS}=S·F,$$

(1)

where SDAS is the physical SDAS value of the alloy, S is the scaled SDAS value used with the ML algorithm, and F is the factor of magnification. Note that the S values were used with the ML algorithm instead of the physical SDAS values. Only for magnification of 5× when F = 1, S = SDAS. The magnification factor for all magnifications used in the present research is given in Table 1. In addition, the values for the number of pixel per micrometer at different magnifications and the factor F are given. The procedure is illustrated using the example of the material AlSi7Mg in Figure 2. For this example, S values of $29·{10}^{-3}$ and $55·{10}^{-3}$ were obtained for training with the same polished cross-section sample with relatively constant SDAS values. The S values used for training are shown in Table 2 and Figure 3. It should be noted that S was determined using average measurement results on a polished cross section and not using Equation (1), i.e., 10 measurements were taken for each magnification and the average result was selected. The deviations between the measurements were less than 10% for each sample. It should be emphasized that a polished cross-section with a relatively constant value of SDAS was chosen, but due to the randomness of dendrite growth, some minor variations in S values are possible. Additionally, for the purpose of training, polished cross sections were chosen that contained defects. The defects could be roughly classified into five groups as shown in Figure 4: (a), (b)—air and shrinkage porosity defects; (c), (d)—scratches; (e)—blurred image; (c), (f), (g), (i)—different brightness and contrast of the image; (g), (h)—externally consolidated crystals (ECS). The presence of defects is beneficial to build the model’s resistance to industry working conditions, as such defects are often seen in samples. About 10% of the dataset were images showing materials with some kind of defects. We should reiterate that images with this type of defect should not be used for SDAS predictions and are for training purposes only, as explained earlier. Finally, note that the Figure 3 and Figure 4 contain images as supplied to the neural network. This is the reason for a lower standard of quality than usual in material science. However, the original higher quality images are available as supplementary materials to this paper (Figures S1 and S2).

Figure 1. SDAS definition: the distance between two secondary dendrites.

Table 1. Magnification factor F and appropriate pixel per micrometer value.

F	Magnification	Pixel/${10}^{-3}$ mm
1	5×	1.36
0.5	10×	2.72
0.25	20×	5.44
0.125	40×	10.88
0.1	50×	13.60

Figure 2. The procedure of deriving different S values using different magnifications on the microscope: (a) 5× magnification image; (b) 10× magnification image. The scale bar corresponds to S value.

Table 2. Images count per different material and different magnification.

Alloy	Magnification	S	F	S-SDAS/F	Images Count
AlSi9Cu3(Fe)	5×	6 $\times {10}^{-3}$ mm	1	0	1076
AlSi9Cu3(Fe)	10×	13 $\times {10}^{-3}$ mm	0.5	1	6820
AlSi9Cu3(Fe)	20×	26 $\times {10}^{-3}$ mm	0.25	2	815
AlSi9Cu3(Fe)	40×	52 $\times {10}^{-3}$ mm	0.125	4	1236
AlSi9Cu3(Fe)	50×	65 $\times {10}^{-3}$ mm	0.1	5	1128
AlSi7Mg	5×	29 $\times {10}^{-3}$ mm	1	0	1216
AlSi7Mg	10×	55 $\times {10}^{-3}$ mm	0.5	3	1176
AlMg5(Si)	5×	32 $\times {10}^{-3}$ mm	1	0	1032
AlMg5(Si)	10×	64 $\times {10}^{-3}$ mm	0.5	0	1174

Figure 3. $200\times 200$ pixel images used for training, their derived S values and alloy.

Figure 4. Images of different types of defects used for the training: (a,b)—air and shrinkage porosity defects; (c,d)—scratches; (e)—blurred image; (c,f,g,i)—different brightness and contrast of the image; (g,h)—ECS (some images are purportedly of lower quality).

The image dataset for training consists of $200\times 200$ pixel images created from the original $2080\times 1544$ pixel images, as shown in Figure 5. The original image was first split into 70 images with a pixel resolution of $208\times 211$, which were then resized to $200\times 200$ pixel. The S values and image count used for training, validation and testing are listed in Table 2. The training set, validation set and test set were split in the ratio 60:20:20. The data instances were evenly distributed in terms of S values.

Figure 5. Schematic representation of dataset generation: the original image was first split into 70 images with a pixel resolution of $208\times 211$, which were then resized to $200\times 200$ pixel.

In an earlier approach [34], the images were preprocessed by the morphological transformation, Gaussian blur and Otsu’s threshold [35] to reduce the number of training parameters. The color scale of the image was reduced from an eight-bit three-channel depth (i.e., full RGB) to a one-bit color depth (e.g., black or white). Unfortunately, the approach requires manual tuning of the hyperparameters for each SDAS value, making it impractical. For the present study, grayscale images were used to achieve fully automatic detection of SDAS using DL to avoid any manual operation. Nevertheless, the training parameters were reduced by a factor of three by using only one color channel instead of three.

3.3. Overview of the CNN Model

Predictive model at hand is based on the CNN architecture. It is assumed that other ML approaches would exhibit inferior behavior and therefore they were not considered. In the present case, the CNN model is based on a basic feedforward (sequential) network. The model consists of three 2D convolutional layer blocks. Each convolutional filter is followed by a Rectified Linear Unit (ReLU) activation, batch normalization and a max-pooling layer. Filter sizes used were $(5,5)\times 32$, $(3,3)\times 32$ and $(2,2)\times 32$ for the first, second and third convolutional layers, respectively. Zero-padding was applied evenly across both dimensions to compensate for edges. The convolutions were performed with stride 1. Each max-pooling filter was of size $(2,2)$, and pooling was performed with stride 2. This resulted in the following intermediate activation maps: $200\times 200\times 1$ (input), $100\times 100\times 32$ (following the first convolutional block), $50\times 50\times 32$ (following the second convolutional block) and $25\times 25\times 32$ (following the third convolutional block). Activation of the final convolutional block was then flattened to create a meta-layer of 20,000 neurons, which was then followed by a fully-connected layer consisting of 64 neurons, followed by the ReLU activation function and batch normalization. Finally, this layer was then fully connected to the output layer, which contained a single linear neuron—because of the assumed regression operation. Thus, for a 2D input image of size $200\times 200\times 1$, a single real value—S value is obtained. The Adam optimizer was used, using a learning rate of $0.0005$. The schematic CNN architecture is shown in Figure 6. The total number of parameters was 1,294,977. It was trained with the Keras library on an Intel^® Core™ I7-4970 CPU running at 3.60 GHz using 8 parallel processing units. Training was completed in 5 h—for 30 epochs total. A batch size of 32 was used for the training.

Figure 6. The CNN architecture used for estimating SDAS directly from microstructure image patches.

4. Results and Discussion

The CNN model described in the previous section showed very good performance, having the $R^2$ score of $91.5\%$ on the test set. We performed an additional experiment to test the accuracy and practical usability of the model for possible industrial applications. The input image of $2080\times 1544$ pixel is taken as a reference and split into 70 images of $200\times 200$ pixel using the same approach as shown in Figure 5. The average result of 70 predictions is then captured while the prediction deviation and prediction error were also captured. We performed two mutually independent evaluation tests: one using materials that were used during the training, and another using materials that were not used during the training. It should be noted that individual images of both group of materials that were used for this test were not involved in the training, i.e., disjoint test subsets were used for both. The results are shown in Table 3 and Table 4, while the images used for the prediction task are shown in Figure 7 and Figure 8. S values were obtained using the same procedure as explained in the previous section.

Table 3. Results of prediction for materials used during training.

Material	S	Pred. S	Pred. Dev.	Pred. Err.	Magnification
	µm	µm	µm	%
AlSi9Cu3(Fe)	6	4.8	1.2	20	5×
AlSi9Cu3(Fe)	13	13.6	0.6	4.6	10×
AlSi9Cu3(Fe)	27	27.7	0.7	2.6	20×
AlSi9Cu3(Fe)	52	52.9	0.9	1.7	40×
AlSi9Cu3(Fe)	65	64.9	0.1	0.15	50×
AlMg5(Si)	32	29.8	2.2	6.9	5×
AlMg5(Si)	64	62.6	1.4	2.2	10×
AlSi7Mg	29	23.0	6.0	20.7	5×
AlSi7Mg	55	55.9	0.9	1.6	10×
Average prediction deviation on 10× magnification: 0.76 µm
Average prediction error on 10× magnification: 2.15%

Table 4. Results of prediction for materials not used during training.

Material	S	Pred. S	Pred. Dev.	Pred. Err.	Magnification
	µm	µm	µm	%
AlSi7Mg0.6	30	28.1	1.9	6.3	5×
AlSi7Mg0.6	60	58.1	1.9	3.1	10×
AlSi10Mg(Fe)	8	11.1	3.1	38.8	5×
AlSi10Mg(Fe)	15	23.9	8.9	59.3	10×
AlSi10Mg(Fe)	31	35.8	4.8	15.5	20×
AlSi10Mg(Fe)	62	52.7	9.3	15.0	40×
AlSi12Cu1(Fe)	5	13.9	8.9	178	5×
AlSi12Cu1(Fe)	19	26.1	7.1	37.4	20×
AlSi12Cu1(Fe)	40	41.4	1.4	3.5	40×
Average prediction deviation on max. mag.: 4.60 µm
Average prediction error on max. mag.: 7.2%

Figure 7. Images of materials used during the training: The images were split into $200\times 200$ pixel images following the procedure in Figure 5 and used for the prediction task.

Figure 8. Images of materials that were not used during the training: The images were split into $200\times 200$ pixel images following the procedure in Figure 5 and used for the prediction task.

The predicted results are very accurate for the trained materials. Average prediction deviation was evaluated for 10× magnification, giving only $0.97·{10}^{-3}$ mm. Moreover, the maximum prediction deviation is $6·{10}^{-3}$ mm, while every other prediction deviation is less than $2.2·{10}^{-3}$ mm. In Table 3, it can be observed that a higher prediction deviation as well as a higher prediction error correlates with a smaller magnification. This could be directly attributed to the resolution of the images. At higher magnification, a dendrite is described with more pixels, so it is expected that the predicted results are more accurate. For example, the highest magnification of 50× results in an error of only 0.1 µm. Hopefully, SDAS smaller than $5·{10}^{-3}$ mm is rarely seen in industrial applications. However, even assuming the highest deviation, such SDAS prediction model performances are quite acceptable for industry. A model with such performance could even be used to determine the SDAS distribution on a single microstructure image and/or polished cross-section sample.

The prediction error for the predicted results was also recorded. The average prediction error for the trained materials at 10× magnification is very low $-2.15\%$ while the highest is $20.7\%$. Again, the highest prediction errors were associated with the lowest resolution for all alloys considered. Increasing the number of pixels that depicts SDAS through higher magnification effectively eliminates the accuracy problem. At the highest magnifications, the errors were in the range of $0.15$–$2.2\%$, as can be seen in Table 3. It is therefore recommended to use a magnification of at least 10×.

In addition, prediction accuracy was also tested with materials that were not used during training (Table 4). The physical SDAS values of AlSi10Mg(Fe), AlSi12Cu1(Fe) and AlSi7Mg0.6 were $8·{10}^{-3}$ mm, $5·{10}^{-3}$ mm and $30·{10}^{-3}$ mm, respectively. The results shown in Table 4 indicate that the predictions are less accurate than for the trained materials. The prediction error is also slightly higher, while the average prediction deviation is $4.60·{10}^{-3}$ mm for highest magnifications. For the AlSi10Mg(Fe) alloy, the highest error of $15\%$ was obtained for 40× magnification, while errors close to $3\%$ were obtained for the other alloys. Note that a very high error (178%) was obtained for AlSi12Cu1(Fe) at low magnification, indicating that higher magnifications should again be preferred for the application of the CNN trained on other materials. This allows the error to be reduced to 3.5% for the 40× magnification, which is in the same range as when the CNN is applied to trained materials, cf. Table 3. For general analysis (e.g., to determine whether the component was cast in sand or in permanent mold, or for general determination of solidification rate), the model could be used successfully in the foundry industry, ensuring that magnification greater than 10× is used. As an example, even the deviation of the different measurement methods reported in [33] falls within the performance range of the model. The highest prediction deviation of $9.3·{10}^{-3}$ mm was obtained for the alloy AlSi10Mg(Fe).

5. Conclusions

The present study confirms the hypothesis that SDAS can be determined directly from microstructure image data by the DL methodology. The model shows excellent performance on the materials used in the training, provided that the highest possible magnification is used. On the other hand, the model performs slightly worse for the materials not used in training. However, both variants could be used in any type of industry depending on the model accuracy. A model for materials used during training could even be used to determine the distribution of SDAS on a single microstructure image and/or polished cross-section sample. Furthermore, the model could still be applied for materials not used during training, although some caution should be exercised. Thus, from the discussion shown, it appears that the technique based on DL can predict SDAS with a high degree of certainty.

Since manual tuning of hyperparameter values is avoided in the present study, such an approach could be fully automated. Furthermore, this approach now has a potential application as part of Industry 4.0 for the automotive or other sector. Additionally, the methods of DL could also have a potential application for automated prediction of mechanical properties from an image of the microstructure. However, for this type of task, the link between SDAS and material properties should be part of the analysis.