A Novel Image Segmentation Approach for Microstructure Modelling

Abstract

Microstructure models are used to investigate bulk properties of a material given images of its microstructure. Through their use the effect of microstructural features can be investigated independently. Processes can then be optimised to give the desired selection of microstructural features. Currently automatic methods of segmenting SEM images either miss cracks leading to large overestimates of properties or use unjustifiable methods to select a threshold point which class cracks as porosity leading to over estimates of porosity. In this work, a novel automatic image segmentation method is presented which produces maps for each phase in the microstructure and an additional phase of cracks. The selection of threshold points is based on the assumption that the brightness values for each phase should be normally distributed. The image segmentation method has been compared to other available methods and shown to be as or more repeatable with changes of brightness and contrast of the input image than relevant alternatives. The resulting modelling route is able to predict density and specific heat to within experimental error, while the expected under predictions for thermal conductivity are observed.

1. Introduction

Thermally sprayed materials have been the subject of numerous works focusing on methods for modelling based on images of their microstructures [1,2,3,4,5]. In general these studies aim to predict the bulk properties of the material to validate the modelling route before using the model to investigate underlying mechanics and directions for future materials or processing parameters. In addition some studies have attempted to computationally generate equivalent images of microstructures so individual parts of the structure can be optimised [1,6,7].

There are several steps in this process which introduce error into the final model [2]. The imaging method and settings, image segmentation and processing method, mesh generation, model formulation, boundary conditions and domain size must all be carefully considered for a representative model. Unfortunately some of these steps, notably the image segmentation and processing contain hidden parameters which can be tuned by the user to fit the desired result. The resulting modelling route is therefore not necessarily generally applicable.

Several of the above parameters have been previously investigated and best practice established. Imaging methods were considered by Deshpande et al. [8] who concluded back scattered SEM should be used with low electron energies for 2D models. This method has been used in the majority of studies since. Mesh generation has been considered by Bolot et al. [9] and Coffman et al. [10]. In general two methods are accepted, either pixels can be converted directly into elements, resulting in a very large mesh, or a representative mesh generated through programs such as OOF2, which again introduce tunable parameters and results in very long mesh generation times.

1.1. Image Segmentation

Accurate methods for segmenting SEM images into maps describing the location of each phase in the microstructure are of key importance in generating accurate models [11]. Standard threshold methods, such as described by Otsu [12], can be used. These are designed to separate between normally distributed peaks on the histogram however if many cracks are present in the material these do not belong to either the metal peak or the porosity peak instead they have an intermediate brightness and are not normally distributed [13]. If standard threshold methods are used the resulting model greatly over predicts mechanical and thermal properties [11], however accurate predictions of density are possible [8,14]. In order to compensate for this problem semi automatic methods can be used or alternative automatic methods, some works have also used a manually selected threshold point.

With semi automatic selection the threshold value is chosen to enforce a certain ratio between the phases, this value is normally determined experimentally [13,15]. In the alternative automatic method suggested by Bolot et al. [9] the threshold level is determined as the intercept between the X axis of the histogram with a line fitted to the straight portion of the Gaussian like peak representing the matrix phase. This method produces higher threshold values and thus higher porosity contents in two phase materials than the Otsu method [11]. Additionally Deshpande et al. [8] and Dorvaux et al. [16] used a threshold point at which the number of isolated bright pixels in the binary image abruptly increases. However what constitutes an abrupt increase is not specified and it is unclear weather this method is performed manually or automatically.

The sensitivity and repeatability of the threshold method used are of key importance, this was investigated by Dorvaux et al. [16]. They showed that for their images a change of 20 units (on a 0–255 scale) from the value determined by the Otsu method resulted in a 5% change in the resulting thermal conductivity. They concluded that the results were not sensitive to small changes in the threshold value around the value determined by Otsu. No works have been identified which consider the sensitivity or repeatability of non standard techniques, however this result is unlikely to be mirrored.

The problem with selecting a threshold level was highlighted by Zivelonghi et al. [13]. They showed that accurate property estimates could be obtained by using a semi automatic threshold value and manually finding cracks. Changing the conductivity of the crack phase from that of the matrix material to air produced a reduction in the resulting thermal conductivity by a factor of three. The cracks present in the microstructure have previously been shown to be responsible for the majority of the loss of thermal conductivity from the bulk value by Dorvaux et al. [16]. Hence there is a clear need for an automatic threshold method which is able to separate out cracks in the microstructure as a separate phase. Several methods to identify cracks or defects based on images have been developed for inspection purposes [17,18], however, these methods do not consider SEM images and are typically focused on identifying single cracks in simple uniform bodies rather than extracting a crack network from a cross section.

Based on the limitations identified above, the aim of this work is to develop a justifiable, repeatable, automatic, method which builds on the work of Zivelonghi et al. [13] to segment the image into phases including a phase of cracks. The method will be compared to other image segmentation options commonly used in the field.

2. Methodology

The necessary steps in the modelling route outlined above are described below. In each case current best practice is used.

2.1. Materials

Given the number of tunable parameters it is clear that modelling a single property for a single material cannot confirm that a modelling route is generally reliable. For this reason the proposed modelling route will be applied to multiple spray batches of two separate thermally sprayed materials. The properties of density, specific heat and thermal conductivity will be estimated.

The two modelled materials are composite abradable materials, these consist of a metal matrix with a dislocator and some porosity. They are typically used to maintain seals at the blade tips in gas turbine compressors and are the subject of several previous microstructure modelling studies [1,2,3,4,5]. Specifically these materials are a aluminium silicon polyester and an aluminium silicon hexagonal boron nitride (hBN) composite. These are available as powders from oerlikon metco (Pfäffikon, Freienbach, Switzerland) as 601 and 320 respectively, for the remainder of this work they will be referred to as 601 and 320.

These materials can be sprayed at different hardnesses by altering spray parameters. This allows more variations of the materials to be investigated, trends to be observed and the modelling route further validated. A total of five spray batches are investigated. These are identified by the material number described above and the superficial hardness (HR15Y) number which is commonly used to classify these coatings. As such 601-55 refers to material 601 (AlSi polyester) with a superficial hardness of 55.

2.2. Image Acquisition

In line with best practice, as discussed in the introduction, images were acquired by backscattered SEM at 5 keV, a reduced vacuum was used to control charge on the surface of non conductive areas. Sets of fifteen images were taken at several different magnifications to allow for RVE estimation. Images taken at the lowest magnification were consecutive (the left edge of the first image was the right edge of the second etc.) to ensure non biased image selection. For higher magnifications the magnification was increased without moving the sample: such that, for example, the third image of each set is centred on the same area. The magnifications used and the resulting model sizes are given in

Table 1. Magnifications of images taken and the corresponding edge length of the 900 by 900 pixel square used for modelling.

Magnification (×)	Model Size ($\mathsf{\mu }$m)
80	1223
120	815
200	489
300	326
400	245
500	196
600	163
800	122

.

In addition to this, two further sets of images were taken for three of the samples tested. For these images the brightness and contrast of the images were varied. The samples were not moved between images and the magnification used was constant. Examples of the extremes of each set are shown in Figure 1A,B for brightness and Figure 2A,B for contrast, these images will be used to evaluate the repeatability of the image segmentation procedure compared to other methods.

2.3. Proposed Image Segmentation Procedure

Each phase in the micrograph should be represented by normally distributed brightness values in the final image. This assumption has been verified by acquiring images with the settings as described above but with a magnification of 10,000×. These images were taken so a single phase was present in the frame. A histogram of one of these images with a normal distribution fitted to it is shown in Figure 3A while Figure 3B shows a quartile-quartile plot of the data and the fitted normal distribution.

Compliance to normal distributions is observed for metal phases in both 601 and 320, and polyester in 601. hBN in 320 has not been shown to follow a normal distribution as the tail on the dark side is too large as shown in Figure 3C,D. Matějíček et al. [14] showed that the hBN phase in these coatings contains some porosity, it is thus assumed that the hBN in these areas is normally distributed and the additional material in the darker tail is porosity. This assumption allows a general procedure to be developed, as described below. This assumption should be verified for any new material the process is used for.

Figure 4 shows a histogram from a full image of an example microstructure (601-55). As shown the histogram consists of two peaks which approximate normal distributions. Between the peaks the level is higher than would be expected if the histogram were simply a combination of normal distributions. As each phase should be represented by a normal distribution optimum threshold points can be found by fitting normal distributions to each of these peaks.

The equation describing a normal distribution is given in Equation (1). In which S is a scale parameter. $\mu$ is the mean of the distribution and, as the distribution is symmetrical, the centre of the peak. $\sigma$ is the standard deviation of the distribution. This curve is fitted to the histogram of the image by setting the above parameters.

$$Y=S\times e^{\frac{{(x-\mu )}^2}{2{\sigma }^2}}$$

(1)

The centre of each peak can be found by splitting the histogram at a point given by Otsu’s method and taking a weighted average of the top 11 points in each region (excluding any over exposed pixels). With this as the mean, the maximum value of the line is scaled to the the peak by setting the scale parameter S in Equation (1) to the maximum value from the histogram for that peak. The deviation parameter $\sigma$ is set by minimising the least squares error between the line described by Equation (1) and the points from $\mu -15$ to $\mu +10$ again excluding overexposed pixels.

Figure 4 shows the histogram with scaled Gaussian curves for the dislocator and metal phase fitted as described above. The blue line in the figure shows the number of pixels not described by the fitted curves. As shown for the central section and the darkest area of the histogram this is more than 50% of the pixels at some brightness levels.

Pixels at brightness levels that are not well explained by either normal distribution (>$50\%$) are considered as porosity if they are darker than the peak for the dislocator phase. If they are brighter than this they form the set of pixels: $C{Z}_{GTO}$, an example of this set is shown in Figure 5. The resulting pixel set contains several features which are not cracks. Smudge type areas, transitions from the metal phase to the dislocator phase and some isolated pixels are included in this set. The following image processing steps remove these areas.

In order to remove smudge type areas a local threshold is used. A new image $I_b$ is created from the original with pixels belonging to the pore and dislocator phases replaced by random values from the distribution that describes the metal phase. The deviation from the local (15 by 15 pixels) mean is calculated for each pixel in this image. Pixels which deviate from the local mean by $-5\%$ or less form the set of pixels: $C{Z}_{LTO}$, an example of this set is shown in Figure 5. This includes some random pixels from the dislocator and pore phases but these are removed in a later step.

Isolated areas smaller than three pixels are then removed from the pixel sets $C{Z}_{GTO}$ and $C{Z}_{LTO}$ resulting in two new pixel sets: $C{Z}_{GT}$ and $C{Z}_{LT}$ respectively.

To remove pixels surrounding the dislocator and pore phases, these phases are dilated by a single pixel in each direction and the result removed from $C{Z}_{GT}$. This results in none of the cracks extending to the dislocator phase. The contact zone or cracks phase ($C{Z}_a$) is then found as the intersection between the sets: $C{Z}_A=C{Z}_{GT}\cap C{Z}_{LT}$. This process is summarised in Figure 5.

To reconnect the contact zone phase with the dislocator and pore phases, the union of the pore, dislocator and cracks phases is dilated by a single pixel in each direction and thinned to a minimally connected stroke. These pixels are added to the cracks phase if they were originally members of either $C{Z}_{GTO}$ or $C{Z}_{LTO}$. Pixels originally belonging to the dislocator phase are then removed from this set. This is described by Equation (2).

$$C{Z}_B=(C{Z}_A\cup ({(C{Z}_A\cup Pores\cup Dislocator)}_{Dilateandthin}\cap (C{Z}_{GTO}\cup C{Z}_{LTO})))-Dislocator$$

(2)

Pixels are then added to $C{Z}_B$ to force 4-connectivity (shared pixel edge) in previously 8-connected (shared pixel corner) bodies, while this step is technically un-physical it is necessary to produce physical behaviour for diagonal cracks. Lastly, $C{Z}_B$ is skeletalised and pixels which were originally part of the dislocator or pore phase are removed resulting in the contact zone phase ($CZ$). Pixels removed from the crack phase are allocated to the metal phase if they are brighter than the threshold determined by Otsu’s method, otherwise they are allocated to the dislocator phase. The process of reconnecting the cracks to other phases and the subsequent processing is summarised in Figure 6.

A resulting histogram of each phase is shown in Figure 7. An example original image and segmented image are shown side by side in Figure 8. The full matlab code for this process is given in the additional material.

2.4. Model Formulation

The processed phase maps were converted to a finite element mesh by the direct pixel to element method, giving one element per pixel with nodes at the corners. Boundary conditions of $T=0$ and $T=1$ ${}^{°}$C are applied to the top and bottom boundaries respectively, thus giving a temperature gradient through the material, the sides of the material are considered insulated. The static thermal problem was then solved by a purpose written matlab program. Using a uniform square mesh with purpose written code provides substantial improvements over commercial codes, not least as the model does not have to be exported after image segmentation. The finite element code is provided in the additional material of this work. Typical time for image processing, mesh generation and solution was around 2 min for a 900 by 900 pixel area.

2.4.1. Material Models

The material models used for each phase of the microstructure are summarised in

Table 2. Material properties used in the finite element models.

Material	Conductivity W/m/K	Specific Heat Capacity Cp J/kg/K	Density kg/m${}^3$	Reference
AlSi(12%)	130	910	2700	[9]
hBN	33	808	2100	[19]
Polyester	0.15	1350	900	[20]
Pores	0.0257	1005	1.205	[21]
Cracks	-	910	2700	[9] *
Dimensionality	$M^1{L}^1{T}^{-3}{\mathrm{\Theta }}^{-1}$	$L^2{T}^{-2}{\mathrm{\Theta }}^{-1}$	$M{L}^{-3}$	-

* Specific heat capacity and density of the metal phase is used for the crack phase as discussed in main text.

. These are taken from values measured from large scale samples and are used without modification. Pixels belonging to the crack phase are assumed to be metallic with a crack much smaller than the pixel size. Thus the density and specific heat of the crack phase set the same as the metal phase. The effect of varying the conductivity of this phase will be investigated.

2.4.2. Other Threshold Methods

The image to model process has been repeated using the Otsu [12] and Bolot [9] methods for image segmentation. As the materials used consist of three phases and the alternative methods are for use with two phase materials for the comparative sections of this work the novel method will be used to segment into a phase of metal and a single phase of cracks, porosity and dislocator which is given the properties of porosity. Alternative methods will also split the image into two phases which will be assigned the properties for the metal matrix and the porosity as given in Table 2.

2.5. RVE Estimation

The method of RVE estimation described by Kanit et al. [22] will be used. This is simply the modelled areas and number of models which produces a standard error for the properties of interest of less than 5%. Although other methods exist these are less conservative. Hence, This approach was chosen because it provided the most robust assessment of the segmentation method proposed.

2.6. Experimental Methods

The density of the samples has been measured by the water displacement method using an Archimedes balance apparatus. Five measurements were taken from separate samples ($10\times 10\times 2.5$ mm${}^3$) for each material.

For each material investigated three diffusivity measurements were taken from separate samples. Measurements were taken at 40 ${}^{°}$C as this is the closest to room temperature possible. Measurements were taken on a flashline 3000 laser flash diffusivity apparatus (Anter Corporation, Pittsburgh, PA, USA). This has been calibrated to output both diffusivity and heat capacity of the sample with $95\%$ confidence bounds at $\pm 2.88\%$. Combined with the density measurements this allows the thermal conductivity to be calculated.

3. Results

3.1. RVE Validation

The Results from the RVE estimation are shown in Figure 9A,B for 601 and 320 respectively. As shown the standard error is below the 5% error threshold for 15 images taken at 120× for both 601 and 320. This means that a set of fifteen images is adequate to estimate the bulk properties with 5% random error at the given magnification. As such, where property estimates are made in this work, they will be the average from 15 models at a magnification of 120×.

3.2. Comparison to Experiment

3.2.1. Specific Heat and Density

The modelled specific heat and density are outputs of the image segmentation procedure as they are simply the average of the phase properties weighted by their composition. These results are shown in

Table 3. Calculated composition and material properties of the coatings based on image analysis of 15 micrographs from each image at 120X magnification with standard error ($n=15$).

Coating-HR15Y	Metal (%)	CZ (%)	Dislocator (%)	Porosity (%)	Density (kg/m${}^3$)	${\mathit{C}}_{\mathit{p}}$ (J/K/kg)
601-82	$53.4\pm 0.4$	$11.1\pm 0.2$	$35.2\pm 0.4$	$0.17\pm 0.02$	$1761\pm 8$	$1076\pm 2$
601-69	$39.7\pm 0.4$	$9.8\pm 0.2$	$50.0\pm 0.5$	$0.52\pm 0.07$	$1521\pm 7$	$1140\pm 2$
601-55	$31.2\pm 0.2$	$8.3\pm 0.2$	$60.2\pm 0.2$	$0.34\pm 0.02$	$1383\pm 5$	$1183\pm 1$
320-72	$63.1\pm 0.3$	$22.3\pm 0.1$	$13.5\pm 0.3$	$1.15\pm 0.07$	$1986\pm 5$	$919\pm 0.4$
320-55	$53.5\pm 0.3$	$21.3\pm 0.2$	$23.8\pm 0.3$	$1.47\pm 0.27$	$1944\pm 9$	$907\pm 1$

for each coating and each hardness. Modelled densities are shown in the context of the experimental results in Figure 10. The calculated specific heats are shown in Figure 11.

As shown the image segmentation procedure is able to produce estimates for the density and specific heats of the samples to within experimental error for 601. For 320 the density is over estimated, this may be due to difficulty in separating porosity from hBN or due to retained binder (latex) around the hBN.

3.2.2. Thermal Conductivity Results

As only a single reference provides an estimate for the conductivity of the contact zone phase the effect of varying its conductivity is investigated here. Figure 12A,B show the change in overall conductivity with changes in conductivity of the contact zone. As seen by Zivelonghi et al. [13] insulating behaviour is seen below ${\lambda }_{cz}/{\lambda }_{metal}=0.001$ with little additional effect with lower conductivities.

As with Zivelonghi et al. [13] the closest match to the measured values is found for ${\lambda }_{cz}/{\lambda }_{metal}=0.01$ however as an underestimate is expected for the 2D case this value should not be used. Since the exact value is unknown, the essentially insulating case of ${\lambda }_{cz}/{\lambda }_{metal}=0.001$ will be used for future modelling. This is compared to the experimental value in Figure 13 below.

For the AlSi polyester the model under predicts the thermal properties as expected. This under prediction is larger as a percentage of the actual value for softer batches with poor connectivity of the metal phase. For the AlSi hBN abradable a large over prediction is seen. Matějíček et al. and others have noted that residual binder surrounds the hBN particles [14,23,24]. This isolates the hBN from the heat flow in the metal matrix. If the conductivity of the hBN phase is reduced to that of the binder used the calculated conductivity gives the expected under prediction as shown in Figure 13B labelled no hBN.

3.3. Comparison to Alternative Techniques

For this section of the work, the image segmentation technique is used in a two phase mode. The contact zone, dislocator and porosity are grouped into a single phase which is given the properties of porosity. Results in this section should only be used to compare between techniques, not to provide absolute estimates.

Figure 14 shows the mean conductivity results from the ‘brightness’ set of images (described in Section 2.2 paragragh 2) for each threshold method tested. As these are binary models of three phase materials comparison to experimental data is not relevant. For each material the Otsu method gives the highest prediction indicating as others have concluded that this method is unable to identify cracks leading to over predictions of material properties [11].

The repeatability of each method has also been investigated. Figure 15 shows box plots of the conductivity results for each threshold method and each coating tested. These results have been normalised by subtracting the mean result of each set. Figure 15A shows that the proposed segmentation technique is repeatable despite changes in brightness of the input image. The proposed method out performs the relevant alternative method proposed by Bolot et al. [3] in this respect. Adequate repeatability is found with changes in contrast (Figure 15B) although the range of results is slightly larger than those generated by the Bolot method. Although the range of results generated by the Otsu method is smaller in both cases, this method is inappropriate for property estimation due to over predictions discussed above.

The systematic error produced with changes in brightness or contrast can be investigated by linear regression of the results.

Table 4. Significance of non zero slope from linear regression of conductivity results from different threshold methods with changes in brightness and contrast.

Coating	Variable	LT-GT	Bolot	Otsu
601 55	Brightness	NS	****	*
601 82	Brightness	NS	*	***
320 55	Brightness	NS	*	***
601 55	Contrast	NS	**	NS
601 82	Contrast	NS	***	****
320 55	Contrast	*	*	**

* NS stands for not significant and *, **, ***, **** indicate significance at the $p<0.05,0.01,0.001$ and $0.0001$ levels respectively.

summarises these results, ideally there would be no significant trend in the results with respect to changes in brightness or contrast of the input image. As shown the suggested method has no significant bias in five out of six of the conditions tested, in this respect it outperforms both the Otsu and Bolot methods of image segmentation. However it should be noted that changes in the result from the Otsu method are small and not of practicable importance.

4. Discussion

An automatic method for image segmentation has been developed which is capable of automatically extracting a phase of cracks from the microstructure as described by Zivelonghi et al. [13]. The method has been compared to other current methods and is shown to be as or more repeatable than alternatives [3] which threshold in the same region with changes in brightness or contrast of the input image. In addition the method has physical underpinnings which justify the selection of threshold points for this application.

This segmentation method produced estimates of density and specific heat that were within experimental error of measured values. This extends the work on porosity of Deshpande et al. [8] and Matějíček et al. [14] to other properties which are simply a volumetric mean of the components of the composite.

Under predictions were seen for modelled thermal conductivity results as is to be expected of two dimensional models due to the loss of connectivity in the third dimension. As with Zivelonghi et al. [13] the overall conductivity was strongly dependent on the contact zone or crack conductivity. The proper value to be used for this variable remains unknown.

Separately these results also explain some phenomena seen in high speed wear testing of these materials. Variation in wear behaviour is seen across the width of the rub track of the same scale as variation between the spray batches shown here [24]. Models made from $200\times 200$ $\mathsf{\mu }$m${}^2$ images showed standard deviation in thermal conductivity results of between 46% and 66% of the mean value (Figure 9). Thus roughly 30% of such areas will have properties equal to the bulk properties of one of the adjacent batches (50% change in value, one standard deviation). This rate decreases as the sampled area increases, 2% of $1.2\times 1.2$ mm${}^2$ sections are expected to vary from the bulk mean by 50% or more. As such local variation in the abradables is in the same scale as variation between batches and can be expected to produce the same changes in wear mechanism.

5. Conclusions

An automatic image segmentation method has been developed which allows automatic separation of a phase of cracks from a microstructure image. Estimates of the density and the specific heat capacity of the material could be made to within experimental error and expected under predictions for thermal conductivity were observed.

The segmentation method was shown to have less systematic bias with brightness and contrast than other available methods. Random error was also lower than seen for competing methods for changes in brightness of input image and was similar to competing methods for changes in contrast of the input image.