Deep Learning Model for Prediction of Diffusion in Defect Substances

Abstract

Actual diffusion activity function is an important metric utilized to describe the diffusion activities of a vacancy defect substance. In this paper, we propose a deep learning three-dimensional convolutional CNN model (D³-CNN). A 3D convolution has its kernel slides in three dimensions as opposed to two dimensions with 2D convolutions. 3D convolution is more suitable for three-dimensional data. We also propose an amplification learning technique to predict the actual diffusion activity of a vacancy defect substance, which is impacted by the geometrical parameters of the defect substance and the vacancy distribution function. In this model, the geometric parameters of a three-dimensional constructed vacancy defect substance are generated. The 3D dataset is obtained by the atoms diffusion defect (ADD) simulation model. The geometric parameters of the 3D vacancy defect substance are computed by the proposed amplification technique. The 3D geometric parameters and the diffusion activity values are applied to a deep learning model for training. The actual diffusion activity values of a substance with a vacancy size ranging from size 0.52 mm to 0.61 mm are used for training, and the actual diffusion activity values of substance vacancy of size between 0.41 and 1.01 are classified by the three-dimensional network. The model can realize high speed and accuracy for the actual diffusion activity value. The mean relative absolute errors between the D³-CNN and the ADD models are 0.028–7.85% with a vacancy size of 0.41 to 0.81. For a usual sample with a vacancy of size equal to 0.6, the CPU computation load required by our model is 14.2 × 10⁻² h, while the time required is 15.16 h for the ADD model. These results indicate that our proposed deep learning model has a strong learning capability and can function as an influential model to classify the diffusion activity of compound vacancy defect substances.

1. Introduction

Atom diffusion, in vacancy defect substances, is a pervasive diffusion phenomenon that is observed in interconnects and insulation in IC production [1,2,3], atom separation [4,5], and energy extraction [6,7]. Atom diffusion is triggered by the thermal diffusion of atoms under a temperature incline gradient; it is a mass transfer process [8,9]. The atom diffusion process in a vacancy defect substance is denoted using the diffusion activity of a vacancy defect substance, which is impacted by its geometrical dimensions [10,11,12]. The prediction of the actual diffusion activity is critical for optimizing the model performance of atom diffusion in vacancy defect systems.

Substantial efforts have been dedicated to further investigation of the atom diffusion process in vacancy defect substances. Conventional models for predicting the actual diffusion activity of vacancy defect substances incorporate laboratory metrics [13], statistical simulations (e.g., the atoms diffusion defect (ADD) simulation model [14]), atomic dynamics techniques [15], and empirical mathematical formulas [16]. For instance, the authors in [17] presented an empirical mathematical formula to predict the atom diffusion cap.

The ability of vacancy defect systems. Based on the ADD model, the authors in [18] investigated the atom diffusion process in vacancy defect saturated substances. They established that the mass formula is more accurate at classifying the diffusion activity of vacancy defect substances. The authors in [19] investigated the impact of shape properties on atom diffusion in rubbery substances. Their study proved that the ADD formula can predict the atom vacancy defect substance even at higher water levels. Hence, the authors in [20] estimated the values of atoms diffused in vacancy defect constituents and presented an equation for the diffusion activity of vacancy defect substances. Published models are very slow with high computational load, particularly for vacancy defect substrates with large input sizes.

In recent research, deep learning models were utilized for predicting atom diffusion processes in vacancy defect substances [21]. Different models strictly obeyed physical analysis to accomplish mapping, particularly for approximate atom relations [18,19,20,21]. The atom flow in vacancy defect substance can be captured by intelligent models to generate a representation for the Bayes formula from spatial resources.

Deep learning models are usually trained using spatial resources. The published research on diffusion prediction of vacancy defect substances is mostly in two-dimensional structures. For instance, the authors in [22] predicted the operative heat conductivity of vacancy defect substances using geometrical 2D images employing intelligent learning models. They established models that can attain operative heat conductivity of vacancy defect substances. Hence, the authors in [3] studied the heat properties of 3D vacancy defect substrate using deep learning models, as the training data was extracted from cross-view images of 3D vacancy defect structures. The authors in [12] studied the atom diffusion proficiency of built 2D vacancy defect structures utilizing machine learning. They found that the intelligent model has a better performance in classifying the atom diffusion of composite vacancy defect substances. The results revised above explain that deep learning models can be appropriately employed to study diffusion phenomena in vacancy defect substances.

Nevertheless, two-dimensional convolutional neural networks (CNN) can utilize a two-dimensional scheme to classify a two-dimensional portion [13]. A two-dimensional neural network only uses one portion, and it cannot control the framework from neighboring points. It should be noted that the connectivity in two-dimensional vacancy substances is significantly different from the three-dimensional presentation. Therefore, it is crucial to employ an intelligent model to classify the substance attributes of three-dimensional constructions. To study this problem, three-dimensional CNNs that utilize three dimensions to classify the 3D space patch are needed. Three-dimensional CNNs can predict barrier diffusion [14], actual conductivity of vacancy substances [15], and velocity distributions in such three-dimensional substrates [16]. All cases are validated experimentally. Therefore, it is time inefficient with a high computational load when a large number of cases has to be evaluated.

In our research, a supervised deep learning CNN (D³-CNN) model that achieves direct mapping from constructed three-dimensional vacancy defected structures to obtain diffusion activity values is proposed. An amplification D³-CNN is presented. The proposed D³-CNN extracts hidden properties from the three-dimensional defect substrate and defines the required information utilized in its predictions. The diffusion activity functions of the three-dimensional defect structures with a vacancy size ranging from 0.42 mm to 0.82 mm are predicted.

This paper is structured as follows: Section 2 presents the investigation of the physical model of vacancy defect substances. Section 3 presents the training process of the D³-CNN model. Experimental results are presented in Section 4. The conclusions are presented in Section 5.

2. Physical Models

2.1. Description of the Physical Model

A typical vacancy defect substance is used to investigate atom diffusion phenomena. This structure is also used to train deep learning models. The diffusion process is impacted by the vacancy size, the volume of the bulk atoms, and the substrate type.

2.2. Reconstruction of 3D Substrate

We must reconstruct the 3D substrate samples for the three-dimensional D-CNN (D³-CNN) model training and test the model performance. Three-dimensional structures are first produced from 2D samples in different views; then, the actual diffusion activity functions are computed from energy barrier thickness values. The computation of the 3D structure is depicted in Figure 1. The dimension of the selected vacancy is computed as follows:

$$S_T=S_x\times S_y\times S_z=200u\times 200u\times 200u$$

(1)

where $S_x, S_y,S_z$ are the dimensions in the x, y, and z directions, and u is the vacancy unit. The vacancy unit is the lattice unit cell that contains one-eighth of an atom at each of its eight corners, so it contains one atom in total.

The diffusion is denoted by diff_in at the entry point and diff_out at the exit point.

The 3D structures are built by a technique through which multiple-view 2D structures are randomly located. It is expected that the 2D structures overlap and their distances are normally distributed. The 2D structures are randomly positioned in a cubic space computing the volume of the 2D structures until it is at the setting value (V). There are two elements in the construction (the vacancy space and the 2D structures). The vacancy defects (D) are the segment of the residual that omits undetected 2D structures. The variables for the 3D structure include the vacancy defect (D), threshold (t), the mean radius of the 2D structures (d_mean), and the diameter standard deviation (σ).

The reconstructions have a d_mean of 32u (vacancy unit), σ is equal to 3.1u, and t equals 0.29; all are fixed, while the vacancy positions are variable with values of 0.22, 0.29, 0.38, 0.49, 0.61, and 0.72, enlarged to the extent of 52 items of vacancy sites from 0.38 to 1.01, and used as the input dataset [21].

2.3. Mathematical Presentation

The information of the constructed 3D structures is computed using an attentiveness slope. This is controlled by the atom distribution law [19] that is depicted in the following equation:

$$\frac{\partial diff}{\partial t}-\nabla \left(B_d \nabla diff\right)=0$$

where $B_d$ is the bulk structure diffusion activity value. The concentration in the vacancy space is denoted by diff_out at $time \tau =0$. $\frac{\partial diff}{\partial t}$ is the diffusion gradient. The four sides (front, top, back, and bottom sides) are non-vacancies, where the boundary conditions for the corresponding domains are computed as follows:

$$diff=Dif{f}_{in} , x=0 and time>0$$

(2)

$$diff=Dif{f}_{out} , x=S_x and \tau >0$$

$$\frac{\partial diff}{\partial y}=0, y=0 or S_y, and \tau >0$$

$$\frac{\partial diff}{\partial z}=0, z=0 or S_z, and \tau >0$$

The atoms diffusion defect (ADD) simulation model computes the actual diffusion activity value in the training phase of the deep learning model. The ADD model is accurate in predicting atom diffusion properties in vacancy defect structures [15].

The diffusion equations are solved through the time vacancy model utilizing the accurate ADD technique, and the actual diffusion activity value of the vacancy substrate is attained. An explanation of the ADD is depicted here. The formula to compute the ADD is computed as follows:

$$a_i \left(L+ve{l}_i \Delta \tau , \tau +\Delta \tau \right)-a_i\left(L,\tau \right)=\frac{1}{r}[a_i\left(L,\tau \right)-a_i^{eq}\left(L,\tau \right)]$$

(3)

where $a_i$ is the atom distribution function (DF), $L$ is the location, $vel$ is the velocity vector, $\Delta \tau$ is the time step ($\Delta \tau =\Delta x/$diff), $a_i^{eq} is the$ equilibrium atom distribution function, and $\tau$ is the relaxation time.

$\mathsf{\phi }$ is a formula of atom diffusion activity value, which is computed as follows:

$$\mathsf{\phi }=\frac{3B_d}{dif{f}^2\Delta \tau }+0.5$$

(4)

To abolish mathematical error in the implemented simulation, the relaxation time is set to “1”, which is included in the stable range [0.5,2]. Non-equilibrium feedback patterns are utilized at the in-point and out-point for static concentrations. These feedback schemes are performed on the four side surfaces because of their excellent precision in boundary composite geometry [19]. The ADD simulates unstable atom diffusion, which is needed to attain the steady state macroscopic constraint of a convergence parameter, computed as follows:

$$\sqrt{{{\displaystyle \sum }}_{i,j,k}{\left(dif{f}_{i,j,k}^{\tau +7000}-dif{f}_{i,j,k}^{\tau }\right)}^2)/ {{\displaystyle \sum }}_{i,j,k}{\left(dif{f}_{i,j,k}^{\tau +1000 }\right)}^2<1.0\times {10}^{-7}{}_{ }^{ }{ }_{ }^{ }}$$

(5)

where $dif{f}_{i,j,k}^{\tau +7000} and dif{f}_{i,j,k }^{\tau }$ are the concentration in the vacancy space at time $\tau$ to $\tau +7000$. The diffusion, the concentration (diff), and the atoms’ mass flux $M_f$ at each node can be computed as follows:

$$diff={\displaystyle \sum }_i{a}_i$$

(6)

where $a_i$ is the atom distribution function.

$$M_f=\left(M_{fx}, M_{fy}, M_{fz}\right)={\displaystyle \sum }_{i}ve{l}_i{\left(a_i\right)}^{\frac{\tau -0.5}{\tau }}{}_{ }$$

(7)

After obtaining the concentration (diff) and atoms flux ($M_f$) at each point, the actual diffusion activity value of the defect substrate ($B_{Diffus}\left(eff\right)$) is normalized by dividing by $B_{Diffus}$ across the diffusion route.

The actual diffusion activity value will be further utilized as input information for the D³-CNN training.

3. Training Process of the D³-CNN Model

The proposed D³-CNN model comprises an input layer that uses blocks of data inputs and passes the input to the convolutional one block at a time. The convolutional computes the key features of each block. The max pooling then computes the maximum from the feature map portion enclosed by the pooling filter to decrease the computational load and pool the important features. The dropout avoids overfitting by dropping some of the output of the pooling layer randomly. The output and the fully connected layer decide the final prediction answer. In our research, the input data items pass to the 3D convolution layers, which extract the features and construct the feature maps. Max pooling is a pooling layer that computes the maximum for patches of a feature map and utilizes it to produce a downsampled (pooled) feature map

The pooled feature maps are passed to the ReLU activation function to incorporate nonlinearity. The FC layers will condense the information and transfer it to the predicted $M_f$ and $diff.$

The Proposed Amplification D³-CNN Learning Technique

The training phase of any deep learning model will require a large input dataset. This will lead to a lengthy, unfeasible simulation time to extract all input data items by the ADD simulation model. To face this challenge, a single structure input will go through the data amplification technique. In our technique, data and features of large 3D vacancy defects can be split into data of smaller vacancies using the sliding window spatial algorithm (SWP). An 8u sliding block is utilized to amplify the data items. During the window sliding, symbolic constructions are selected to stop the SWP from selecting similar blocks. The actual diffusion activity functions of the smaller structures are computed with the atoms’ mass flux diffusion. The basis for utilizing this sliding amplification model is to split the 24 original structures of vacancies (0.33 and 0.51). Additionally, the 1024u × 1024u × 1024u vacancy units are split into 512 sub-structures with sizes 128u × 128u × 128u. The vacancy sizes of the generated substructures vary from 0.45 to 0.61. The vacancy properties of the generated substructures (128u × 128u × 128u) are different from the original structure (1024u × 1024u × 1024u) because generating smaller substructures from the bigger structures yields randomness (the original structures have vacancies with disorderly configuration, and the generated substructure consists of a random configuration of the original ones). The process of splitting the original big structures into smaller substructures will yield different atomic mass flux distribution. Their actual diffusion activity functions are computed by their corresponding flux values. This model can escape the production of abundant actual structures in the chemistry lab, which is an extremely slow process. The generated 12,288 substructures and their computed actual diffusion activity functions are used in the training phase as shown in Figure 2.

4. The Proposed D³-CNN Model

The 3D substrate is a grouping of several 2D vacancy substrate images taken from different views. The spatial associations are ignored by the conventional layer. A three-dimensional deep network can reduce this problem and execute a function with a 3D instead of a 2D square structure.

The configuration data and the actual diffusion activity values are used as input and are computed as follows:

$$Z_i=\left\{x,y,z, f\left(x,y,z\right)\right\}where i=1 to N$$

(8)

where $x,y,z$ are the Cartesian coordinates, and each ranges from 1 to 128u. N is equal to 2048.

$$f\left(x,y,z\right)=\{\begin{array}{c}1\\ 0\end{array} where 1 means solid and 0 means vacancy$$

(9)

The actual diffusion activity value is computed by the ADD. Dimensions of 0.52 and 0.61 are used as training data. Hereafter, a small training set with 9000 samples is utilized as the input to the input layer. Other data (3000), with vacancy dimensions ranging from 0.41 to 0.81, are used in prediction in the testing phase.

When the training subset and the testing subset are organized, they are used as inputs into the D³-CNN architecture. The training subset should be optimized using hyper-parameters such as size. For many tries, the batch size is set to 32. Therefore, the input cube for the input sample will be represented as a matrix of dimensions (32 × 128u × 128u × 128u) and is computed as follows:

$$Z^i=\left\{Z_{\left(n-1\right)\times 32+1}+Z_{\left(n-1\right)\times 32+2}+Z_{\left(n-1\right)\times 32+3}+\dots \dots +Z_{\left(n-1\right)\times 32+32}\right\}$$

(10)

where n is selected in the iterations, and i represents the D³-CNN input layer.

The process is presented in Figure 3.

5. Experimental Results

5.1. Selection of the Hyper-Parameters

The hyper-parameters are the number of network layers $N_L$, kernel size $K_s$ number of nodes in each network layer $N_n$, and the activation functions $f_{act}$. These parameters are obtained before the training starts. Using practical solutions, the hyper features that can support the technique’s accuracy are usually unknown. The hyper-parameters are selected by decreasing the mean absolute error ($MAE$) for the 32 input structures and are computed as follows:

$$MAE=\frac{1}{32}{\displaystyle \sum }_{k=1}^{32}\left|\left(B_{Diffus}{\left(eff\right)}_{ }^{D_{ }^3-CNN}-B_{Diffus}{\left(eff\right)}_{ }^{}\right)\right|{ }^{ }$$

(11)

When the $MAE$ function joins, the hyper-features are considered as satisfactorily learned. Then, the ADD classification $B_{Diffus}\left(eff\right)$ will be computed to measure the accuracy of choosing the hyper-parameters.

Table 1. Mean relative error of the D³-CNN models including hyper-parameters.

Number of D3-CNN Convolutional Layers	Dropout Layers
	1	2	3
	Mean Error
4	24.1%	–	–
6	21.5%	6.8%	–
8	22.4%	10.9%	16.1%

depicts the mean error of the predicted result and the ground truths. The mean error $RE$ is computed as follows:

$$RE=\frac{\left|\left(B_{Diffus}{\left(eff\right)}_{ }^{D_{ }^3-CNN}-B_{Diffus}{\left(eff\right)}_{ }^{}\right)\right|}{B_{Diffus}{\left(eff\right)}_{ }^{}}$$

(12)

The training, as depicted in Table 1, indicates that:

• Increasing the dropout improves the accuracy
• Arrangement of six D³-CNN and the dropouts realizes high accuracy.
• The proposed method accomplishes a low 9.8% mean error from the testing dataset.

The training loss function will rapidly converge and scarcely decrease at 1200 epochs. The testing loss function varies across the process and decreases to 0.15 × 10⁻² after 2700 epochs. The testing loss functions will converge, demonstrating that the proposed model can attain stable and efficient results as depicted in Figure 4.

Figure 5 depicts the mean absolute error of the D³ classification and the result computed by the ADD in 50 cases, which are partitioned into clusters. Every cluster describes the probability of matching results. The probability denotes the percent of the data lodging different error values. Cumulative probability depicts the additive percent of the data lodging different error values.

5.2. The Verification of the D³-CNN Model

The architecture of the D³-CNN model is depicted in

Table 2. D³–CNN Layers.

Layer Number	Layer	Filter Size	Activation
1	Input	128 × 128 × 128	–
2	Convolutional	26/7 × 7 × 3	–
3	Pooling	3 × 3 × 3 (max)	ReLU function
4	Convolutional	50/7 × 7 × 3	–
5	Pooling	3 × 3 × 3 (average)	ReLU function
6	Dropout layer	0.6	–
8	Normalization	70	ReLU function
9	Convolutional	90/5 × 5 × 3	–
10	Dropout	0.4	–
12	Output		–

.

Validation of the D³-CNN investigates the actual diffusion activity functions in the prediction of the D³-CNN model and the ADD that depicts a comparison of the actual diffusion activity functions for the testing samples with vacancies of sizes 0.42 to 0.81 as predicted by the model, the ADD, the experimental results in [22], and the model presented in [17]. The actual diffusion activity value of vacancy defect substances grows linearly with the increase in the number of defects. This depicts that a high defect size contributes to diffusion in vacancy defect substances. The training dataset has structures with vacancies of sizes of 0.45 and 0.61, and the D³-CNN attains an error of 0.031% to 7.75% for substances with vacancies of sizes ranging from 0.41 to 0.78 when compared to ADD. Nevertheless, the results obtained by the authors in [13,14,15,16] have weighty deviations when compared with ADD. This is due to the design of the vacancy defect substance with a specific geometry. To obtain higher prediction rates, these models need to train new structures. However, the D³-CNN does not face these challenges as the D³-CNN is trained using geometric features of vacancy defect substances (i.e., 12,280 smaller structures mined using an amplification process) and has a robust predictive power for unknown testing data as depicted in Figure 6.

Figure 6 compares the ground truth of the labeled samples against the prediction values from our model. As can be seen from Figure 6, our proposed model has prediction results similar to the ground truth. The ADD algorithm performs less well than the D³-CNN model.

The time cost is an important performance metric for deep learning models [5]. To prove the efficiency of our proposed model, we compared the time required by the ADD model and the training phase and testing phases of our model. The experimental results for the defect structure with a vacancy size of 0.5 are depicted in

Table 3. Time cost of the D³-CNN model and ADD.

Phase	GPU (NVIDIA)
D3-CNN training	14.2 × 10−2 h
ADD training	15.16 h
D3-CNN prediction time per input sample (average from 200 runs)	1.45 min
ADD prediction time per input sample (average from 200 runs)	3.15 min

. It illustrates that the D³-CNN model needs 14.2 × 10⁻² h for the training phase, while the ADD needs 15.16 h. This implies that the D³-CNN model is faster by two orders of magnitude than the ADD. In the prediction time, the D³-CNN model will require half the time required by the ADD model.

6. Conclusions

In this paper, we proposed a platform to accurately classify the actual diffusion activity value by employing a deep learning model. The 3D structures of the vacancy defect substance are generated through a stochastic method; the actual diffusion activity value of the vacancy defect substance is attained by a vacancy score technique utilizing the ADD simulation model. The data produced by the processes are fed to the D³ network for input and testing phases. Vacancies of sizes between 0.50 and 0.60 are utilized in the learning phase. The D³ model confirmed a strong training capacity and attained less mean absolute error of values 0.028% to 8.97% in the testing phase. The ADD algorithm needed a computational CPU load of 15.16 training hours while the D³-CNN model needed only 14.2 × 10⁻² training hours. This deep learning model is an influential model that could be utilized to classify the diffusion activity of composite vacancy defect substances. The model achieved high speed and better accuracy for the prediction of the diffusion value. Additionally, the mean relative absolute error of the D³-CNN was computed as 0.028% on average from the ground truth. These results indicate that our proposed deep learning model has faster strong learning and prediction capability.

For future work, other extensions to this work can be considered. A new model would be the application of the methodology to predict other substance features, such as porousness.