Deep Learning and Bayesian Hyperparameter Optimization: A Data-Driven Approach for Diamond Grit Segmentation toward Grinding Wheel Characterization

Abstract

Diamond grinding wheels (DGWs) have a central role in cutting-edge industries such as aeronautics or defense and spatial applications. Characterizations of DGWs are essential to optimize the design and machining performance of such cutting tools. Thus, the critical issue of DGW characterization lies in the detection of diamond grits. However, the traditional diamond detection methods rely on manual operations on DGW images. These methods are time-consuming, error-prone and inaccurate. In addition, the manual detection of diamond grits remains challenging even for a subject expert. To overcome these shortcomings, we introduce a deep learning approach for automatic diamond grit segmentation. Due to our small dataset of 153 images, the proposed approach leverages transfer learning techniques with pre-trained ResNet34 as an encoder of U-Net CNN architecture. Moreover, with more than 8600 hyperparameter combinations in our model, manually finding the best configuration is impossible. That is why we use a Bayesian optimization algorithm using Hyperband early stopping mechanisms to automatically explore the search space and find the best hyperparameter values. Moreover, considering our small dataset, we obtain overall satisfactory performance with over 53% IoU and 69% F1-score. Finally, this work provides a first step toward diamond grinding wheel characterization by using a data-driven approach for automatic semantic segmentation of diamond grits.

1. Introduction

Diamond grinding wheels (DGWs) have a central role in cutting-edge industries such as aeronautics, defense and spatial applications. DGWs are mainly produced by powder metallurgy (PM) technologies such as the sintering process [1]. As illustrated in Figure 1, a DGW is composed of two parts: a matrix and diamond abrasive grits. The matrix, generally a metallic alloy, ensures the retention of diamond abrasive grits and acts as a binder for the cohesion of the wheel. Characterizations of DGWs are essential in order to optimize the design and machining performance of such cutting tools. One way is to quantify the abrasive power of DGWs. This quantity could be seen as a complex relationship between matrix wear and loosening diamond grits. Non-exhaustively, we could distinguish several approaches for the evaluation of DGW abrasive power: 3D topographic analysis of active surfaces [2,3], micro-geometric approaches [4,5] or in situ acoustic emission monitoring [6].

As the abrasive power evaluation systems are difficult to set up and use in an industrial environment, more promising approaches are computer vision-based methods. The aim of such methods is to leverage the knowledge of subject experts in order to characterize DGWs from images. In these approaches, one of the major challenges is the detection of diamond grits and their interfaces. Generally, diamond detection methods rely on manual operations on DGW images. However, these methods are time-consuming, error-prone and inaccurate. In addition, due to the diversity of DGW image acquisition parameters, it is challenging to generalize traditional image processing algorithms for an accurate semantic segmentation of diamond grits.

In recent years, in the data-driven era, with the rise of deep learning algorithms, semantic segmentation has become an extensively studied research area in the field of computer vision [7,8]. The aim of semantic segmentation algorithms [9] is to assign a separate class label to each pixel of an image for object detection purposes. Applications of such algorithms can be found in autonomous vehicles [10], where the pixel labels could be trees, cars, humans or roads, and in medical image analysis [11,12], where they could be cells, tumors or even brain activity analysis [13,14]. For DGW image segmentation, it is about diamond abrasive grits.

The semantic understanding of a scene by a computer is a challenging problem mainly solved today thanks to the development of convolutional neural networks (CNNs) by LeCun et al. [15] and the computational resources of modern computers. CNNs use successive convolution and pooling operations on an input image. The convolution product between the input image and a kernel extracts features from the input [16]. Then, the result, the feature map, is operated by a pooling layer to capture the essential features which describe the object of interest. This operation produces spatial reduction and a loss of object position information. Moreover, a fully connected network with non-linear activation functions is connected at the end to make predictions, and the whole network is trained by a backpropagation algorithm [17]. CNN constitutes the “fundamental blocks” of more advanced CNN-based architectures for computer vision applications [18,19].

It is well known that training a CNN from scratch is energy-intensive and requires huge labeled datasets [20,21]. That is why, to overcome this problem, it is established in the computer vision community to use transfer learning techniques [22,23,24]. Transfer learning consists of using a pre-trained CNN model [25] (often called “backbones”), trained over a large standardized dataset model by shifting its weight over a new training dataset of interest (i.e., DGW images). For example, in computer vision (especially segmentation tasks), ImageNet [26] is a widely used dataset for a pre-trained model of over 14 million labeled images. Based on this observation, several new CNN architectures have been developed over the years, such as AlexNet [27], VGG [28], ResNet [29], GoogLeNet [30], U-Net [31], Inception [32], DenseNet [33] and EfficientNet [34].

In this work, we implement a U-Net architecture for the semantic segmentation of diamond grits, using ResNet34 backbones trained on ImageNet as an encoder. However, although transfer learning is efficient, with more than 8600 hyperparameter (HP) combinations, it is impossible to manually find the optimal configuration of HPs. Some examples of hyperparameters include the model architecture and training parameters such as epoch number, batch size or learning rates. For that reason, finding the best combination of hyperparameters remains challenging even for experts in the field. In this respect, we use an Automated Machine Learning (AutoML) [35] methodology by leveraging Bayesian Hyperparameter Optimization (BO) [36,37,38,39,40] and Hyperband [41] algorithms. Finally, our work is part of an ongoing effort to make deep learning models more accurate, reliable and easier to use for researchers, engineers and programmers.

The key contributions of this article include:

• A practical application and implementation of state-of-the-art (SOTA) deep learning computer vision algorithms for semantic segmentation of objects in images in the industrial field of DGW manufacturing.
• An assessment of segmentation performances of the corresponding models over state-of-the-art metrics.
• To the best of our knowledge, this work is the first published attempt of using a U-Net deep learning model for automatic segmentation of diamond grits from DGW images.
• The foundations of upcoming data-driven methods for DGW characterizations.
• A unified methodology for automatic configuration/calibration of deep learning models by leveraging Bayesian optimization and Hyperband algorithms, an AutoML framework for making deep learning approaches easier to use.

The remainder of this paper is organized as follows. Section 2 introduces the implemented U-Net deep learning architecture and the Bayesian and Hyperband algorithms, including the overall hyperparameter optimization strategy. Section 3 presents the data acquisition, software and hardware specifications, with an emphasis on manual hyperparameter complexity reduction. Section 4 provides the experimental results in two parts: Bayesian optimization experiments and diamond grit segmentation results. Finally, Section 5 summarizes the article, draws conclusions about this work and outlines future works.

2. Methods

The aim of this section is to progressively introduce our AutoML framework. The U-Net deep learning architecture and associated segmentation metrics are introduced. Then, the Bayesian optimization and Hyperband algorithms are presented in detail. Finally, a presentation of the overall Bayesian hyperparameter optimization strategy is given.

2.1. Model Architecture and Metrics

In the previous section, CNN basic architectures were introduced. These kinds of networks are well suited to capture the context of an image but with a loss of spatial information due to the down-sampling effect. This drawback, due to the alternation of convolution and max pooling layers, makes CNN basic architecture not well suited for semantic segmentation tasks. For that reason, we use the U-Net encoder–decoder model first introduced by Ronneberger et al. [31], as shown in Figure 2. The U-Net architecture is composed of two paths: the encoder and decoder. The encoder is a contracting path with 4 successive convolution blocks associated with max pooling steps. The convolution blocks extract raw features from DGW images (“what is a diamond grits?”), and max pooling operations grab the most dominant ones. This implies a spatial reduction in the (XY) image axis by increasing the depth of the feature channels. A direct consequence is a loss of information about the position of the object (diamond) in the image. To address this issue, Ronneberger et al. introduced skip connections, an image concatenation mechanism that allows high-resolution spatial features to be cropped from the encoder and concatenated with the corresponding up-convolution of the expansion path. Indeed, the decoder is composed of an expanding path of 4 transposed convolution blocks and respective skip connections in order to learn spatial information (i.e., “where is the diamond?”). Furthermore, a fully connected layer is connected at the end of the decoder for classification purposes. In addition, we use the pre-trained weight of ResNet34 [29] as a feature extraction for the contracting path. This transfer learning technique allows us to save a large amount of training time, computational resources and leverage the U-Net architecture to perform semantic segmentation on a small DGW dataset of 153 images.

Evaluating the performance of semantic segmentation tasks is essential in order to assess the prediction accuracy of the developed model. One way to access such performance prediction is to use the confusion matrix as a starting point for the definition of segmentation metrics [42,43]. Such a matrix is a tabular summary of correct and incorrect predictions.

From the confusion matrix, it is possible to extract the respective coefficients—True Positive (TP), True Negative (TN), False Positive (FP) and False Negative (FN)—and use them to define two fundamental metrics used in our work: the Intersection Over Union (IoU) and the F1-score. These metrics are defined as follows:

$$\mathrm{IoU}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}$$

(1)

$${\mathrm{F}}_1=\frac{2\mathrm{TP}}{2\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}$$

(2)

Although the IoU metrics are popular to assess the performance of the semantic segmentation model, they tend to penalize over and under segmentation prediction. Therefore, prediction performance tends to be underestimated with this index. That is why it is convenient to also use the F1-score defined in Equation (2).

2.2. Bayesian Optimization Methodology

Deep learning and more generally machine learning models are driven by a set of input parameters called hyperparameters. In the case of deep learning, this can be, e.g., network architecture, such as the number of convolution blocks, activation functions and training parameters such as epochs number or learning rate. Fine-tuning hyperparameters is essential to leverage the potential of the developed deep learning model. However, manually tuning hyperparameters is time-consuming and nearly impossible. For example, in the simple version of our model, there are more than 8600 hyperparameter combinations. Deep learning hyperparameter optimization (HPO) is an active research area [44,45,46,47,48,49]. In such problems, the goal is to maximize/minimize an expensive objective function whose closed form and its gradients are unknown. In this paper, we use a Bayesian optimization (BO)-based approach. BO is a black box (i.e., model-based) optimization algorithm searching for a global optimum of an unknown objective function from which observations can be made. The BO algorithm is composed of two components: a surrogate model for objective function approximation and an activation function for the integration of the regression history. In addition, it is worth noting that one of the main advantages of BO is its ability to learn from past experiments. Thus, the global optimization problem could be formulated as follows:

$${\mathrm{x}}^{*}=\arg \underset{\mathrm{x}\in \mathrm{X}}{\max }\mathrm{f}\left(\mathrm{x}\right)$$

(3)

where ${\mathrm{x}}^{*}$ designs the hyperparameter input in the search space X, i.e., $\mathrm{f}\left(\mathrm{x}\right)$ is optimal. However, in order to use the BO for hyperparameter optimization, we assume the following constraints:

• We have a few input control parameters x (less than 20).
• The objective function is relatively “smooth” (i.e., not so discontinuous in the discrete meaning).

As a black box optimization approach, BO uses a surrogate model for the regression task (i.e., fit the unknown objective function). In this work, we used a statistical Gaussian Process (GP) as a surrogate model. The key ideas behind BO are illustrated in Figure 3.

From Figure 3, we can describe the underlying ideas behind the BO algorithm as follows:

(a)

Based on the initial observation points (i.e., sample points measured) from the objective function $\mathrm{f}\left(\mathrm{x}\right)$, a first regression attempt is made using an initial version of a Gaussian Process model. In addition, a confidence interval is built due to the probabilistic nature of the GP. Then, an acquisition function $\mathsf{\alpha }\left(\mathrm{x}\right)$, derived from the GP model, is used to estimate the next computing step.

(b)

The GP model has been updated from past observation points. Thus, the regression improves and the uncertainty area decreases toward the objective function. The next observation point ${\mathrm{x}}_{\mathrm{n}}$ is computed by taking the global maximum of the associated acquisition function.

(c)

Finally, the fitting quality is iteratively improved. When the regression becomes acceptable, the best hyperparameter configuration ${\mathrm{x}}^{*}$ is saved and the optimization problem can be solved by finding a global optimum of the GP model.

Several surrogate models have been developed in the field of BO [40,50]. The most common one is the Gaussian Process because of its acceptable performance and its simplicity of implementation. In addition, using GP as a regression model to fit the unknown objective function is appealing because it considers the uncertainty associated with each prediction.

Given a set of observations $D_{1:t}=\left\{\left(x_1,y_1\right),\dots ,\left(x_t,y_t\right)\right\}$, a Gaussian Process is a stochastic model defined by Equation (4) as the multivariate normal distribution N:

$$\mathrm{P}\left(y|D_{1:t},\mathrm{x}\right)=N\left({\mathsf{\mu }}_{\mathrm{t}},{\mathsf{\sigma }}_{\mathrm{t}}^2\right)$$

(4)

where model predictions are represented by the mean term ${\mathsf{\mu }}_{\mathrm{t}}$ and the uncertainty by the variance ${\mathsf{\sigma }}_{\mathrm{t}}^2$, defined in Equations (5) and (6), respectively.

$${\mathsf{\mu }}_{\mathrm{t}}=k^T{K}^{-1}{D}_{1:t}$$

(5)

$${\sigma }_t^2=\mathrm{k}\left(\mathrm{x},\mathrm{x}\right)-k^T{K}^{-1}k$$

(6)

$$K=\left[\begin{array}{ccc}\mathrm{k}\left(x_1,x_1\right)& \cdots & \mathrm{k}\left(x_1,x_t\right)\\ ⋮& \ddots & ⋮\\ \mathrm{k}\left(x_t,x_1\right)& \cdots & \mathrm{k}\left(x_t,x_t\right)\end{array}\right]+{\mathsf{\sigma }}_r^{2}I$$

(7)

However, since the observation of real systems could be noisy, we introduced a sampling noise ${\mathsf{\sigma }}_{\mathrm{r}}^2$ in Equation (7). In Equations (5)–(7), $K$ represents the covariance matrix. It describes how the observations are related to each other. In other words, the covariance matrix measures the correlation strength between observations by using a kernel function $\mathrm{k}\left({\mathsf{\chi }}_1,{\mathsf{\chi }}_1\right)$ as coefficients. The kernel function has a strong impact on the prediction quality of the GP surrogate model. In fact, it drives the output responses of the GP and the shape of the objective function that we can fit. That is why choosing it carefully according to the optimization problem is of major importance. In this paper, since deep learning metrics such as loss functions or validation IoU are continuous without periodicity, we used the Matérn kernel function that is a generalization of radial basis kernel (RBF) [51], defined by Equation (8):

$$\mathrm{k}\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{x}}_{\mathrm{j}}\right)=\frac{1}{\mathsf{\Gamma }\left(\mathsf{\nu }\right)2^{\mathsf{\nu }-1}}{\left[\frac{\sqrt{2\mathsf{\nu }}}{\mathrm{l}}\mathrm{d}\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{x}}_{\mathrm{j}}\right)\right]}^{\mathsf{\nu }}{\mathrm{K}}_{\mathrm{v}}\left[\frac{\sqrt{2\mathsf{\nu }}}{\mathrm{l}}\mathrm{d}\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{x}}_{\mathrm{j}}\right)\right]$$

(8)

where $\mathsf{\Gamma }\left(\mathsf{\nu }\right)$ is the gamma function, $\mathrm{d}\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{x}}_{\mathrm{j}}\right)$ the Euclidian distance between two observation points and ${\mathrm{K}}_{\mathrm{v}}$ is the modified Bessel function. The parameter $\mathsf{\nu }$ controls the smoothness of the output function. In fact, a smaller $\mathsf{\nu }$ leads to a sharper approximated function.

Thus, the first part of the BO algorithm used a Gaussian Process as a surrogate model of the objective function. The remaining part of the algorithm concerns the choice of an activation function. Such a function is used to decide which observation point ${\mathrm{x}}_{\mathrm{t}}$ should be inferred next. In this work, we used the common Expected Improvement (EI) acquisition function defined by Equation (9):

$$\mathsf{\alpha }\left(\mathrm{x}\right)=EI\left(\mathrm{x},{\mathrm{D}}_{\mathrm{t}}\right)=\underset{-\infty }{\overset{+\infty }{{\displaystyle \int }}}\max \left(y^{*}-y,0\right)\mathrm{P}\left(y{|\mathrm{D}}_{1:\mathrm{t}},\mathrm{x}\right)dy\mathrm{with} {\mathrm{y}}^{*}=\underset{1<\mathrm{i}<\mathrm{n}}{\min }\mathrm{f}\left({\mathrm{x}}_{\mathrm{i}}\right)$$

(9)

Based on the updated GP model $\mathrm{P}\left(y{|\mathrm{D}}_{1:\mathrm{t}},\mathrm{x}\right)$ and the observation database ${\mathrm{D}}_{\mathrm{t}}$, the activation function considers the region of high uncertainty with a large confidence interval and those with the minimum observed true objective function score $y^{*}$ in order to compute the next sampling point ${\mathrm{x}}_{\mathrm{t}}$. Finally, the entire BO algorithm is summarized in Algorithm 1.

Algorithm 1: Bayesian optimization.

input:$\left(\mathrm{T},\mathrm{f}\left(\mathrm{x}\right),\mathsf{\alpha }\left(\mathrm{x}\right)\right)$ 1   initialization:${\mathrm{D}}_0\leftarrow \{{\mathrm{x}}_0,{\mathrm{y}}_0\}$ 2   random initialize the Gaussian Process 3   for$\mathrm{t}\leftarrow 1to\mathrm{T}do$ 4  ${\mathrm{x}}^{\prime }\leftarrow \arg \underset{\mathrm{x}\in \mathrm{X}}{\max }\mathsf{\alpha }\left(\mathrm{x},{\mathrm{D}}_{\mathrm{t}}\right)$ 5  ${\mathrm{y}}^{\prime }\leftarrow \mathrm{f}\left({\mathrm{x}}^{\prime }\right)$ 6   ${\mathrm{D}}_{\mathrm{t}}\leftarrow {\mathrm{D}}_{\mathrm{t}-1}\cup \{{\mathrm{x}}^{\prime },{\mathrm{y}}^{\prime }\}$ 7   Update GP model by computing $\mathrm{P}\left({{\mathrm{y}}^{\prime }|\mathrm{D}}_{\mathrm{t}},{\mathrm{x}}^{\prime }\right)$ 8   end for return best hyperparameters ${\mathrm{D}}^{*}$ such as ${\mathrm{x}}^{*}=\arg \underset{\mathrm{x}\in \mathrm{X}}{\max }\mathrm{f}\left(\mathrm{x}\right)$

2.3. Hyperband Early Stopping Algorithm

Although the BO algorithm allows us to automatically find the optimal hyperparameters in order to maximize IoU and F1-score segmentation metrics, it does not prevent the full test of unpromising experiments. For that reason and to save computational time resources, we included an early stopping mechanism in our BO loop for preventing unpromising training experiments. Hyperband is a bandit optimization algorithm developed by Li et al. [41]. It is based on the Successive Halving (SH) algorithm from Jamieson et al. [52]. The SH algorithm is resource-based, which means that it allocates a budget $\mathrm{B}$, for example, training time to a set of hyperparameter configurations $\mathrm{n}$. It evaluates the performance of all configurations by uniformly allocating the $\frac{B}{n}$ average resource. Then, it throws away the worst half of the performances and repeats the procedure until one configuration remains. However, a fundamental issue is how to choose the tradeoff between considering a large budget (e.g., long training time) with a small number of configurations, or the reverse. Hyperband attempts to solve this tradeoff by fixing the budget and exploring several configurations using a grid search method (i.e., design of experiment of all possible configurations).

The Hyperband algorithm, as described in Algorithm 2, requires two inputs: R, the maximum number of allocated resources to one configuration, and $\mathsf{\eta }$, the proportion of throwing out configurations generated by SH algorithms at each iteration. A bracket designed a run of the SH algorithm. The input parameters control the number of brackets and allocating budgets.

Algorithm 2: Hyperband early stopping.

input:$\left(\mathrm{R},\mathsf{\eta }\right)$ 1   initialization:${\mathrm{s}}_{\max }=\lfloor {\log }_{\mathsf{\eta }}\mathrm{R}\rfloor$, B =(${\mathrm{s}}_{\max }+1$)R 2   for$\mathrm{s}\in \left\{{\mathrm{s}}_{\max },{\mathrm{s}}_{\max }-1,\dots ,0\right\}$do 3   $\mathrm{n}=\lceil \frac{\mathrm{B}}{\mathrm{R}}\frac{{\mathsf{\eta }}^{\mathrm{S}}}{\mathrm{S}+1}\rceil ,\mathrm{r}={\mathrm{R}\mathsf{\eta }}^{-\mathrm{S}}$ 4   // Start Successive Halving inner loop 5   $\mathsf{\Omega }=\text{get\_hyperparameter\_configuration}\left(\mathrm{n}\right)$ 6   for$\mathrm{i}\in \left\{0,\mathrm{s}\right\} do$ 7     ${\mathrm{n}}_{\mathrm{i}}=\lceil n{\mathsf{\eta }}^{-1}\rceil$, ${\mathrm{r}}_{\mathrm{i}}=n{\mathsf{\eta }}^{\mathrm{i}}$ 8     $L\left(\mathsf{\Omega }\right)=\left\{\mathrm{run}\text{\_}\mathrm{then}\text{\_}\mathrm{return}\text{\_}\mathrm{val}\text{\_}\mathrm{loss}\left(t,r_i\right)| t\in \mathsf{\Omega }\right\}$ 9     $\mathsf{\Omega }=\mathrm{top}\text{\_}\mathrm{k}(\mathsf{\Omega },L\left(\mathsf{\Omega }\right),\lfloor \frac{{\mathrm{n}}_{\mathrm{i}}}{\eta }\rfloor )$ 10   end for 11   end for return configuration with the best validation loss seen so far

In lines 1–2, the algorithm starts with the more aggressive bracket with the maximum allocated resources for maximizing exploration (i.e., give some minimal time to tracking experiment and prevent premature experiment early stopping). The function “$\mathrm{get}\text{\_}\mathrm{hyperparameter}\text{\_}\mathrm{configuration}\left(\mathrm{n}\right)$” performs a hypercube sampling of the input hyperparameter and obtains an initial guess configuration. Then, each bracket (i.e., SH iterations, lines 6–9) decreases the number of remaining configurations $\mathrm{n}$ by the dropout $\mathsf{\eta }$ until the last bracket (s = 0). For each bracket, the function “$\mathrm{run}\text{\_}\mathrm{then}\text{\_}\mathrm{return}\text{\_}\mathrm{val}\text{\_}\mathrm{loss}\left(t,r_i\right)$” (line 8) takes an input hyperparameter configuration $t$ with the corresponding resources budget $r_i$, performs a training of the network and returns the associated validation loss. Finally, at line 9, the function $“\mathrm{top}\text{\_}\mathrm{k}”$ returns the top k training configurations.

To summarize, Hyperband is an adaptative algorithm that tracks iterations as a budget resource and associated training/validation losses. In this way, by adaptatively optimizing the average budget ratio, it could automatically stop unpromising experiments early.

2.4. Bayesian Hyperparameter Optimization Strategy (AutoML Framework)

In this paper, we suggested a combined hyperparameter strategy using Bayesian optimization and Hyperband algorithms, following an AutoML approach, as illustrated in Figure 4. Our implemented strategy follows three main steps:

• Configuration and model training: This step is about automatically changing the hyperparameters of a given U-Net architecture, which we call a U-Net configuration. Then, we perform a training of the selected configuration and monitor related training metrics such as the loss function that we record and store for further use.
• Evaluate U-Net configuration performances: In this step, an assessment of prediction performances of the current U-Net configuration is carried out through the computation of segmentation metrics such as IoU.
• Bayesian Optimization Loop (BOL): Once a U-Net configuration is trained and evaluated, the hyperparameter optimization loop begins. In fact, when a U-Net configuration is trained, a surrogate Gaussian Process model is fitted in real time by observing the influence of input variation (U-Net hyperparameters) on the outputs, the training loss functions and the corresponding IoU score. In addition, a “watchguard” mechanism is implemented through the Hyperband algorithm in order to automatically address unpromising training experiments to save computing resources. Thus, the optimization loop performs several iterations on the overall process (steps 1–3) by varying the U-Net configurations until the target IoU score is reached (IoU $\ge 0.5)$.

3. Implementation Details

3.1. Data Acquisition, Hardware and Software Specifications

DGW image acquisition is realized using a numerical microscope, Keyence VHX-5000. Our training image stack is built on 153 “.tiff “images of size 256 × 256 × 3. We manually labeled each DGW image, as illustrated in Figure 5, using the open source software QuPath and Fiji [53,54]. For performing Bayesian optimization experiments and model training, the Google Colab Pro+ environment with an Nvidia Tesla P100 GPU 54 GB RAM was used. We built our deep leaning model with TensorFlow v2 [55] through the high-level API framework Keras [56]. In addition, we used the Weight and Bias (W&B) [57] python API, with its implementation of Bayesian optimization (from scikit-learn [58]) and the Hyperband algorithm. Finally, our experiments were monitored with Keras callbacks and the W&B MLOps [59,60] platform.

3.2. Hyperparameter Search Space Reduction

Although our hyperparameter optimization strategy is fully automated, we must reduce the hyperparameter search space for easier convergence toward the optimization solution and to speed up computations and save resources. To overcome these shortcomings, the hyperparameter search space was reduced, thanks to several assumptions that we discussed later.

Table 1. Hyperparameter search space values (raw configuration A).

Hyperparameters	Range
Optimizer	SGD, Ftrl, Nadam, Adam, Adamax, Adagrad, Adadelta, RMSprop
Activation function	ReLU, SeLU, ELU, sigmoid, softplus, softmax, softsign, tanh, exponential
$\mathit{\alpha }$	[1 × 10−1; 1 × 10−2; 1 × 10−3; 1 × 10−4; 1 × 10−5]
Batch size	[2; 4; 8; 16; 32; 64]
Epochs	[30; 40; 50; 60]

and

Table 2. Hyperparameter search space combination numbers.

Configuration	$\mathit{\Lambda }$	ξ (%)	Optimizer	Activation Function	$\mathit{\alpha }$	Batch Size	Epochs
A	8640	0	8	9	5	6	4
B	1296	85	3	9	2	6	4
C	144	98.3	3	1	2	6	4

summarize the hyperparameter search space values and the combination numbers per category, respectively.

The total number of hyperparameters combinations $\mathit{\Pi }$ is computed by the general combinatorics formula described by Equation (10):

$$\mathit{\Lambda }={\displaystyle \prod }_{i=1}^r{\mathrm{n}}_{\mathrm{i}}$$

(10)

where $\mathit{\Lambda }$ is the search space total configurations number, r represents the hyperparameter variables, ${\mathrm{n}}_{\mathrm{i}}$ represents their associated value numbers and ξ is the total average reduction factor.

In deep learning, optimizer algorithms are key systems which strongly impact the training and prediction performance of the model. Such an optimization system modifies the weights of the network through the adaptation of learning rate and the minimization of a loss function by a gradient descent algorithm [61]. This optimization procedure confers to the deep learning model the ability to learn from these errors. In addition, optimizer choice is a huge research area [62,63,64,65,66,67,68] and a complex question strongly correlated with the type of data to analyze. We can distinguish two main optimizer families:

• Stochastic gradient descent (SGD) with momentum (i.e., inertia added for reducing variance and increasing the convergence rate).
• Adaptative stochastic gradient descent optimizer, which automatically tunes learning rate parameter $\mathit{\alpha }$ during training.

Although the SGD family could be efficient, fine-tuning the learning rate, momentum and implementation of a learning rate scheduler is mandatory. Taking all of that into account, adaptative optimizers such as Adam, Adamax and RMSprop were selected. Therefore, we could reduce the range of learning parameter $\mathit{\alpha }$ between 1 × 10⁻³ and 1 × 10⁻⁴. These assumptions led to reducing the hyperparameter search space combinations by 85% from 8640 in configuration A to 1296 in B. Although this is a huge search reduction improvement, it is not sufficient and should be improved. Thus, our last degree of freedom is to reduce the activation function search space. Activation functions play a key role in deep learning networks by applying nonlinear transformations to their input weights. By doing so, they trigger/activate the learning process of the network. They are used as hidden and output layers and strongly influence the prediction quality. They are generally chosen in functions of the deep learning model architecture and application field [69,70]. In this work, we use sigmoid for the output prediction layer. The sigmoid function was used in the early days of deep learning, and it remains well suited for binary classification tasks as in our problem, the semantic segmentation of two classes: diamond grits and the background. Based on these additional assumptions, the hyperparameter search space was reduced by 13.3%, leading to a total reduction factor of 98% with 144 remaining configurations, becoming more affordable for our optimization strategy.

4. Results and Discussion

In this section, the Bayesian optimization experiments and diamond grit segmentation results are presented in detail. First, a sensibility analysis of hyperparameter optimization experiments is carried out in regard to segmentation metrics such as IoU and F1-score. Finally, a practical application of the best U-Net configurations is performed for the segmentation of diamond grits on three DGW types.

4.1. Bayesian Optimization Experiments Results

Our search space reduction effort based on the assumptions described previously drastically reduced the search space to 144 configurations. However, although the search space becomes computationally more affordable, the aim of BO algorithms is not to explore all the hyperparameter configurations. In fact, unlike brute force algorithms such as grid search, we are trying to find the best configurations in a limited number of experiments, as shown in

Table 3. Hyperparameter reduced search space.

Optimizer	Epochs	Batch Size	$\mathit{\alpha }$	s	R	$\mathsf{\eta }$
Adamax, Adam, RMSprop	[30, 40, 50, 60]	[2; 4; 8; 16; 32; 64]	[1 × 10−3; 1 × 10−4]	2	27	3

. We performed 64 BO experiments for a total uninterrupted computing time of 22 h, as illustrated in Figure 6. For the Hyperband early stopping mechanism, we used a maximum allocated resource value R = 27, such as one epoch is equal to one allocated resource. This parameter is set according to our previous experience, where an increase in IoU was observed in the range of the 25th epochs. In addition, we set s = 2, which corresponds to two brackets of the Successive Halving (SH) algorithm. Thus, the SH algorithm will discard one-third ($\mathsf{\eta }$ = 3) of unpromising experiments in the following bands $\left[\frac{27}{3}, \frac{27}{9}\right]$ (i.e., ninth and third epochs).

Thanks to our hyperparameter optimization strategy, 24.6% of configurations with a validation IoU above 0.50 and a validation F1-score over 0.66 were identified. Furthermore, the Hyperband early stopping mechanisms identify 18% of unpromising training experiments by tracking, in a resource-based manner, the validation IoU in regard to runtime for the best previous experiments. In that way, unpromising trials are generally identified early, in the range of 2 min to 4 min after the experiments start.

The best hyperparameter configuration led to a validation IoU of 0.52 and a validation F1-score of 0.68 for a training time of 22 min 38 s. Although the BO algorithm quickly finds the best hyperparameter configurations, understanding the effects of training parameters and optimizer choices is essential from an engineering perspective. That is why we investigated several training configurations. The best/worst experimental results for each hyperparameter configuration explored by the algorithm are shown in

Table 4. Summary of hyperparameter BO experiments.

ID	Total Exp	Total Failed	Total $\mathbf{IoU}\ge 0.5$	Optimizer	Epochs	Batch Size	$\mathit{\alpha }$	Runtime	Early Stops	Val IoU	Val F1	Loss	Val Loss
A1	5	1	20%	Adamax	60	8	1 × 10−3	11 min 8 s	False	0.11	0.20	0.61	1.37
A2				Adamax	60	4	1 × 10−3	22 min 16 s	False	0.50	0.66	0.44	0.73
A3				Adamax	60	4	1 × 10−3	3 min 10 s	True	0.03	0.06	0.76	1.31
A4				Adamax	40	4	1 × 10−4	14 min 59 s	False	0.18	0.30	0.77	1.10
A5				Adamax	40	2	1 × 10−4	30 min 22 s	False	0.48	0.64	0.56	0.75
B1	34	3	26%	Adam	60	8	1 × 10−4	11 min 14 s	False	0.07	0.14	0.87	1.24
B2				Adam	60	4	1 × 10−3	22 min 20 s	False	0.48	0.64	0.48	0.73
B3				Adam	60	4	1 × 10−3	2 min 23 s	True	0.00002	0.00005	0.73	190.68
B4				Adam	60	2	1 × 10−4	44 min 29 s	False	0.50	0.66	0.37	0.77
B5				Adam	60	2	1 × 10−3	3 min 38 s	True	0.01	0.02	0.66	30.7
B6				Adam	50	8	1 × 10−4	9 min 28 s	False	0.02	0.04	0.75	1.35
B7				Adam	40	2	1 × 10−4	30 min 13 s	False	0.52	0.68	0.42	0.74
B8				Adam	30	2	1 × 10−4	22 min 35 s	False	0.50	0.66	0.47	0.75
B9				Adam	30	2	1 × 10−4	22 min 51 s	False	0.45	0.61	0.55	0.77
C1	25	6	24%	RMSprop	60	32	1 × 10−3	2 min 47 s	False	0.021	0.04	0.81	1.36
C2				RMSprop	60	4	1 × 10−3	22 min 17 s	False	0.49	0.66	0.47	0.74
C3				RMSprop	60	4	1 × 10−3	2 min 29 s	True	0.04	0.09	0.76	1.41
C4				RMSprop	60	2	1 × 10−3	44 min 27 s	False	0.51	0.67	0.39	0.76
C5				RMSprop	60	2	1 × 10−3	3 min 48 s	True	0.02	0.05	0.71	1.34
C6				RMSprop	50	2	1 × 10−4	37 min 10 s	False	0.49	0.65	0.38	0.80
C7				RMSprop	50	2	1 × 10−3	5 min 56 s	True	0.004	0.009	0.65	1.66
C8				RMSprop	30	4	1 × 10−3	11 min 20 s	False	0.0004	0.0008	0.55	1.796
C9				RMSprop	30	2	1 × 10−3	22 min 38 s	False	0.52	0.68	0.47	0.73
C10				RMSprop	30	2	1 × 10−4	22 min 37 s	False	0.50	0.66	0.48	0.77
C11				RMSprop	30	2	1 × 10−3	4 min 11 s	True	0.001	0.003	0.66	1.9

.

As the selection of hyperparameter configurations is automatically computed by the BO algorithm, some experiments are missing a rigorous quantitative comparison. Thus, our study should be seen as a first qualitative attempt of hyperparameter configuration analysis of DGW image segmentations.

The Adamax optimizer configuration represents 7.8% of the experiments which have been the least explored one by the algorithm: it counts a total of five trials and one failed by Hyperband early stopping. This small experimental group led to acceptable segmentation performances. In fact, 66% of this group led to a validation IoU between 0.48 and 0.50. In addition, the batch size tends to have a linear effect on the runtime. Indeed, a decrease of 50% of the batch size led to an average half augmentation of validation IoU and F1-score. Furthermore, decreasing the learning rate coefficient alpha with epoch number seems to confer acceptable performances.

Opposite to Adamax, the Adam optimizer category has been the most explored by the algorithm, representing around 53.1% of the experiments. It counts around 26% of hyperparameter configurations with a validation IoU above 0.5 and 8.8% of failed experiments. However, with this optimizer, increasing the epoch number and batch size seems to reduce the segmentation performance and training stability. In fact, an augmentation of the epoch number tended to generate some overfitting phenomena observed with huge validation losses for experiments B3 and B5, as shown in Table 4. In addition, the best segmentation performances seem to be reached with an overall learning rate coefficient alpha of 1 × 10⁻⁴.

The RMSprop optimizer configuration represents 39% of the BO trials. It counts around 24% of hyperparameter configurations with a validation IoU above 0.5. However, it records 24% of discarded experiments due to the largest exploration of the hyperparameter range. We clearly observed a drop in segmentation performance with batch size in this group. The batch size and epoch number seem to follow a linear trend with the runtime and confirm the same observed effect on the two other groups. Thus, the RMSprop optimizer tends to be more stable than Adam in regard to overfitting when the training time increases. Finally, the RMSprop optimizer achieved the overall best performance, with an IoU over 0.52 and an F1-score of 0.68 with a quick training time (experiment C9, Table 4).

Thanks to this qualitative study, we could formulate the following rules about the observed effects of hyperparameter range and optimizer choice for the semantic segmentation of DGW images:

• The batch size and epoch number have a linear effect on the runtime. For the three studied optimizers, increasing the batch size and epoch number by 50% led to an augmentation of 50% of the training time.
• Increasing the batch size over a value of 4 with a high epoch number led to a significative reduction in segmentation performance.
• The Adam optimizer tends to be more sensitive to overfitting with large epoch numbers.
• Decreasing the learning rate coefficient alpha in conjunction with the epoch number exhibits the best overall performances for Adamax and Adam optimizers.
• With RMSprop optimizer, no evident patterns are seen for fine-tuning the learning rate alpha.
• Adam and RMSprop optimizers exhibit better segmentation performances with minimal training time compared to Adamax.
• The overall best configuration is reached by RMSprop optimizer with a reduced training time of 25% compared to Adam for equivalent segmentation performances.

Finally, the best hypermeter configurations found by our BO optimization strategy are summarized in

Table 5. Best hyperparameter configurations found by our BO optimization strategy.

Optimizer	Epochs	Batch Size	$\mathit{\alpha }$	Runtime	Val IoU	Val F1-Score
Adamax	60	4	1 × 10−3	22 min 16 s	0.50	0.66
Adam	40	2	1 × 10−4	30 min 13 s	0.52	0.68
RMSprop	30	2	1 × 10−3	22 min 38 s	0.52	0.68

.

Finally, it is of importance to highlight some limitations of using pre-trained ResNet34 backbone weights on the encoder part of the network. In fact, although transfer learning techniques allow us to train U-Net deep learning architecture on a small dataset, it prevents us from fully using our AutoML framework. Indeed, input images must be 256 × 256 × 3 due to transfer learning usage, which leads to a lack of flexibility in regard to input images size. In addition, we are not able to fully leverage the power of Bayesian optimization algorithms for neural architecture search (number of CNN layers, size of convolution kernel and so on) due to the inherent constraints of using the ResNet34 weights on the encoder part.

4.2. Diamond Grits Segmentation Results

Thanks to our optimization strategy and the previous experimental study, we found the best hyperparameter configurations, shown in Table 5. Based on these, we trained three diamond grit segmentation models, as shown in Figure 7, Figure 8, Figure 9 and Figure 10. We trained our models on a heterogeneous stack of 153 DGW images. We deliberately chose to test our model on three types of DGW images:

• Type I: Standard metallic DGW images taken at high magnification.
• Type II: Metallic DGW images taken at low magnification.
• Type III: Electroplated DGW images.

In addition, it is worth noting that deep learning experiment reproducibility is a complex problem [71,72,73,74]. Indeed, previous segmentation results should not be seen as a rule of thumb but as automatic methodology and guidelines toward the best segmentation performance for our dataset. Furthermore, it is important to notice that we have not labeled all the diamond grits on the ground truth. Each model was tested on different types of DGW images for performance estimation purposes, and the results are summarized in Figure 10.

Although the reproducibility of deep learning experiments is a complex problem, the results conform to the best hyperparameter configurations found previously with our BO experiments. In fact, they are even better, with a validation IoU in the range of [0.51–0.53], and of [0.67–0.69] for the corresponding F1-score. On type I DGW images, the models performed relatively well, with an overall validation IOU of 0.52 and F1-score of 0.68, as shown in Figure 8. The RMSprop optimizer exhibits the best performance, detecting a labeled diamond grit not seen by the others. In addition, smoother segmentation with better edge detection was achieved by the RMSprop optimizer. On the type II images shown in Figure 9, some subtle differences began to appear in the segmentation quality, although the IoU/F1 score denotes the RMSprop configuration as the best. In fact, Adam detects more diamond grits than the two others, whereas RMSprop tends to better preserve the diamond grits’ shape. Although our models reached some overall acceptable segmentation scores, the type III images shown in Figure 10 highlight some limitations of only using IoU/F1-score for partially annotated ground truth. In fact, the RMSprop configuration is the worst one in regard to diamond grit detection. The best configuration in this case is Adam, which segments practically all the diamond grits. This highlights an important point about the selected segmentation metrics for evaluating model performances. For type III images, we found that IoU/F1-score are not well suited for evaluating the model performance, as the labels/real diamond numbers are strongly unbalanced. In such extreme cases of partially annotated labels, the IoU/F1-score should be used in conjunction with other metrics such as Dice, Hausdorff distance, partially annotated algorithms [75,76,77] and visual evaluation.

Regarding the loss evolution, RMSprop exhibits some overfitting behavior until the 20th epoch, as shown in Figure 10b. Adamax also has some predisposition to overfitting in the range of [20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50] epochs. Thus, the Adam optimizer tends to quickly converge toward a loss minimum and is more stable than the two others. In fact, Adam and Adamax converge 50% quicker than RMSprop by following a growing trend before 20 epochs, as shown in Figure 10c. On the other hand, RMSprop’s validation IoU tends to decrease from the 55th epoch.

On the computing resources side, although Adamax has a bigger training epoch number, it is the cheapest to train in regard to computing time and GPU power, as shown in Figure 10d. It takes around 22 min 30 s to be trained for an average GPU power consumption of 35%. Adam has a bit longer training time than the two others, with around 30 min. In the end, RMSprop is the most GPU-demanding, with GPU peak usage in the range of [80–60%]. This could be explained by its tendency for model overfitting in the early training stages.

Although the best U-Net configurations led to overall satisfactory diamond grit segmentation performances, some limitations of the presented methods cannot be ignored. Using a small training dataset amounts to losing the ability of the network to generalize, which leads to segmentation issues for DGW images that are scarcely represented, such as type III.

5. Conclusions

The configuration and calibration of deep learning computer vision remains a challenging task, even for subject experts. In this respect, the Bayesian optimization results demonstrate the ability to automatically find the best configuration of such models by leveraging an AutoML approach. Indeed, using the Bayesian optimization loop in conjunction with the Hyperband early stopping mechanism allows an effective scanning of complex search spaces in a cost-saving way. By using the proposed hyperparameter optimization methodology, it becomes possible to leverage U-Net within the industrial environment of DGW manufacturing.

Thus, to the best of our knowledge, this article is the first published work of using U-Net deep learning architecture for semantic segmentation of diamond grits of DGW materials. Considering the small size of the training dataset, acceptable segmentation performances were reached, which largely outperformed time-consuming and error-prone manual methods.

The availability of the methods concerning the lack of flexibility of the fixed image input format and U-Net architecture due to the ResNet34 backbone would be improved in future works. A potential improvement lies in removing the pre-trained weights on the encoder part by using a Generative Adversarial Network (GAN) [78] to generate simulated DGW surface images. In that respect, it should be possible to substantially increase the training dataset size and apply the presented AutoML methodology on the neural architecture for better segmentation results.

Finally, this work is the first step of a broader research effort of making deep learning computer vision algorithms easier to use for scientists, engineers and programmers.