Study on Prediction of Zinc Grade by Transformer Model with De-Stationary Mechanism

Abstract

At present, in the mineral flotation process, flotation data are easily influenced by various factors, resulting in non-stationary time series data, which lead to overfitting of prediction models, ultimately severely affecting the accuracy of grade prediction. Thus, this study proposes a de-stationary attention mechanism based on the transformer model (DST) to learn non-stationary information in raw mineral data sequences. First, normalization processing is performed on matched flotation data and mineral grade values, to make the data sequences stationary, thereby enhancing model prediction capabilities. Then, the proposed de-stationary attention mechanism is employed to learn the temporal dependencies of mineral flotation data in the transformed vanilla transformer model, i.e., non-stationary information in the mineral data sequences. Lastly, de-normalization processing is conducted to maintain the mineral prediction results within the same scale as the original data. Compared with existing models such as RNN, LSTM, transformer, Enc-Dec (RNN), and STS-D, the DST model reduced the RMSE by 20.8%, 20.8%, 62.8%, 20.5%, and 49.1%, respectively.

1. Introduction

Mineral grade is a crucial performance indicator for froth flotation [1,2], and accurate real-time prediction of mineral grade assists flotation operators in better recognizing the current flotation conditions, allowing for subsequent measures for extracting crucial minerals. Consequently, investigating grade monitoring during the froth flotation process is imperative.

In traditional flotation devices, operators often estimate grades by observing froth visual characteristics closely related to mineral grades. However, this monitoring method is highly subjective and the accuracy is easily influenced by the operator’s level of experience.To enhance the precision and stability of mineral grade monitoring, X-ray fluorescence (XRF) analyzers have been developed and put into use in modern flotation plants. However, XRF analyzers are expensive and difficult to maintain. To reduce costs, flotation plants often use one XRF analyzer to monitor multiple mineral grades simultaneously, which increases measurement intervals [3], limiting their application in fully automated flotation control.

In recent years, with the advancement of digital image processing technologies, an increasing number of visual features have been extracted for the purpose of constructing mineral grade monitoring methods [4,5]. The features of a froth image can be divided into static features, dynamic features, and statistical features. The static features include bubble size and froth color, which are the most obvious features in froth flotation conditions; the dynamic features include froth stability and load bearing rate, reflecting the overall change trend in the froth; and the statistical features mainly refer to the texture of bubbles, which reflects the fineness of mineral granules attached to the surface of the froth. For instance, Popli et al. [6] employed a support vector machine (SVM) to investigate the correlation between multiple visual features extracted from froth images and mineral grades, aiming to predict grade outcomes, whereas Bendaouia et al. [1] predicted the grade of a mineral concentrate by extracting features such as bubble size, color, and texture.

In essence, froth image data are stored in a time-sequential manner, rich with temporal information and representing the dynamic change trends between data. To mine useful information from these time-series data, various deep learning networks such as recurrent neural networks (RNN), long-short term memory (LSTM), and their variant models [7,8,9], as well as other time-series networks, have been applied for the performance monitoring of flotation processes. Zhang et al. [10] introduced a froth flotation grade monitoring method based on long short-term memory (LSTM) by using froth video sequences. Based on RNN, Yuan et al. [3] achieved grade monitoring by processing the nonlinear relationships between relevant data in the flotation process and mineral grades. Zhang et al. [11] proposed using an encoder–decoder structure combined with RNN models to predict the grade of zinc tailings. However, dependency relationships over longer time spans are difficult to capture using RNN and LSTM models. A taste prediction method based on the transformer model was proposed by Peng et al. [12]. The vanilla transformer used for time series prediction benefits from the long-term dependency capturing capability of the attention mechanism, allowing for effective capturing of temporal dependencies and nonlinear relationships in time series data [13]. It occupies a dominant position in research on neural network-based long-term forecasting [14,15].

However, the vanilla transformer model applied for grade monitoring assumes that froth data are stationary, neglecting the non-stationarity factors in mineral froth data. Non-stationary data refer to data in a time series that exhibit instability or unpredictability. The defining characteristic of non-stationary data is that their statistical properties, such as mean, variance, etc., change over time, indicating that the distribution over time is not constant [16]. In the zinc ore flotation process, the characteristics of the feed material are affected by time, resulting in non-stationary flotation data [17,18]. It is difficult to describe these data using a single model or rule. Recently, some studies have shown that the transformer model is relatively weak in handling non-stationary time series, which can easily lead to severe overfitting phenomena [19].

To mitigate the decline in predictive performance caused by non-stationary data, classical statistical methods such as ARMA and ARIMA [20] can achieve stationarity of time series through difference operations. However, these traditional models cannot compete with deep-learning-based models in capturing dynamic changes in data and solving long-sequence time series forecasting problems [21]. With deep learning models, some researchers have attempted to alleviate the non-stationarity in the original time series by normalizing the global data distribution of related time series into a specific distribution with mean 0 and variance 1, commonly referred to as Z-score standardization [15,22]. Specifically, the core purpose of this normalization method is to transform non-stationary data into stationary data in order to improve prediction accuracy. However, this approach is actually not reasonable, because it affects the ability of the prediction model to capture the time dependency of time series data, which can easily lead to overfitting of the prediction results [23].

To address the issue of non-stationarity in deep learning models due to differences in the statistical properties of input windows, Shen proposed a two-level transformer framework, which exhibited excellent stability [21]. On the other hand, Liu effectively mitigated the overfitting phenomenon caused by non-stationary data such as weather and electricity consumption predictions by optimizing the attention mechanism in the transformer model [24]. Inspired by this, this paper proposes a de-synchronized attention mechanism based on the transformer model after normalizing the data, which approximates the non-stationary information in the original floating data sequence by learning the mean and variance of the original data floating sequence, rather than the normalized statistical information. The main contributions of this paper are as follows:

(1)

To enhance the predictive capability of the model, normalization is performed on the input data, and a subsequent denormalization is applied to the output results, ensuring that the predicted outcomes remain consistent with the original data’s scale range.

(2)

To prevent the predictive model from overfitting, a de-stationary attention mechanism is constructed to account for the non-stationary information within the original data, replacing the attention mechanism in the vanilla transformer.

(3)

Considering the difficulty of learning non-stationary information in the original mineral data, a multi-layer perceptron is used to approximate non-stationary information from raw data and statistical quantities.

(4)

To solve the problem of the mismatch between froth videos and XRF detection time points for zinc ore grade values, we automatically match the froth image closest to the XRF detection time point, ensuring the consistency of experimental data.

The remaining sections of this paper are organized as follows: Section 2 presents an example of zinc tailings grade prediction and the input structure for matching mineral grades. The details of normalization, the de-stationary attention mechanism, and denormalization are described in Section 3. THe experimental results and discussions are provided in Section 4, while the conclusions are presented in Section 5.

2. Relevant Theories

2.1. Example of Zinc Mine Tailings Grade Prediction

Figure 1 shows a lead–zinc flotation process in a Chinese factory. Due to the co-occurrence of lead and zinc in the same ore, the process consists of three stages: lead flotation, zinc flotation, and lead–zinc mixed concentrate flotation. Lead flotation tailings serve as the feed for zinc flotation. The zinc flotation circuit includes three-stage roughing machines, three-stage scavenging machines, and three-stage cleaning machines. Lead flotation tailings enter the first rough, where flotation reagents are added. The froth then flows into three-stage cleaners for further flotation, producing zinc concentrate. The slurry is redirected for re-flotation, resulting in a lead–zinc mixed concentrate.

To monitor mineral grade values in real time during flotation, XRF analyzers (Thermo Fisher Scientific (Waltham, MA, USA). at the green triangle location in Figure 1) and three cameras (at First rough, Scavenger III, and Cleaner III sections) were installed. Due to the high cost of the XRF analyzer, only one unit was used to monitor all critical minerals, adopting a sampling multiplexing method. The dataset used here came from video footage captured in the Cleaner III flotation cell.

2.2. Matching of Froth Video and Mineral Grade Values

The camera took a froth video every 1–2 min, lasting about 3 min each. Due to the high cost of the XRF analyzer, monitoring of zinc ore grade values could only be carried out using a sampling multiplexing method. The sampling time interval of the XRF analyzer was not completely fixed, with an approximate sampling interval of 17–18 min, which is much longer than the sampling interval for froth videos. As shown in Figure 2, the timestamp of the froth video feature vector did not match the timestamp of the marked zinc ore grade value, requiring timestamp matching.

Due to the minimal change in the froth state over a short period, this study employed the first frame of a froth video as a representative of the entire froth video status. By utilizing techniques such as the optimized watershed algorithm, multi-color space fusion, and neighboring gray-level dependence matrix [25], we extracted 18 visual features of froth, including the average size of bubbles, mean red values, and energy, to characterize the froth image. Thus, the froth video could be represented using a set of froth visual features:

$$X=\{F_1,F_2,\cdots ,F_f\}$$

(1)

where f is the total number of froth visual features, $f=18$. The variables $F_1$∼$F_4$ represent the average bubble size, standard deviation, steepness, and skewness of the bubbles; $F_5$∼$F_9$ denote contrast, correlation, entropy, uniformity, and energy; $F_{10}$∼$F_{12}$ indicate the mean values of red, green, and blue; $F_{13}$∼$F_{14}$ represent the mean hue and grayscale values; $F_{15}$∼$F_{17}$ stand for the relative mean values of red, green, and blue; and $F_{18}$ denotes the carrying rate. The definition and calculation method of the relevant features can be found in Appendix A. Huang also encapsulated the explicit significance and computational formulas for characteristics [25].

This paper represents the total number of froth videos and XRF-annotated grade values as $N_v$ and $N_l$, respectively. Let $\Delta T_v$ and $\Delta T_l$ denote the sampling time intervals for froth videos and XRF analyzers. The sampling rate ratio between froth videos and annotated grade values is then represented as N.

$$N=(\Delta T_v)/(\Delta T_l)$$

(2)

Due to the requirement for the marked value to match the number of filmed froth videos (denoted as N), the relationship between the number of input froth videos (froth visual feature vectors) and the number of XRF analysis marked labels can be expressed as

$$N_v{=N}_l\times N$$

(3)

Considering the inability of the initial froth videos $X_i(i=1,2,\cdots ,N_v)$ to achieve complete correspondence with the zinc ore grade values $Y_j(j=1,2,\cdots ,N_l)$ marked by the XRF analyzer, we opted to match the froth video with the shortest time interval relative to the marked zinc ore grade values using the XRF analyzer. It should be emphasized that the timestamp of the matched flotation video should precede the time point at which the zinc ore grade values were marked by the XRF analyzer [11].

$$\left\{\begin{array}{cc}{X}_m^{\prime }=argmin(d(t|X_i,t|Y_j)),\hfill & \\ s.t.t|X_i\le t|Y_j.\hfill \end{array}\right.$$

(4)

where $X_m^{\prime }(m=j=1,2,\cdots ,N_l)$ is the closest video to the matched floating video $Y_j$. The function $d(·)$ represents the Euclidean distance function, while $t|X_i$ denotes the timestamp of the froth video $X_i$, and $t|Y_j$ represents the timestamp of the XRF analyzer marking the grade value $Y_j$. The effective input for the matching froth video sequence should be

$$X^{\prime }=[X_1^{\prime },X_2^{\prime },\cdots ,X_{N_l}^{\prime }]\in {\mathbb{R}}^{N_l\times f}$$

(5)

3. Methodology

The overall flow chart of this paper is presented in Figure 3. First, we normalized the matched flotation data videos, to enhance the predictive capability of the model. Meanwhile, we stored the mean and variance of the original data, for subsequent learning of relevant non-stationary information within these values. Subsequently, we selected the vanilla transformer model to process the normalized froth video sequences. Specifically, in the encoding layer, we replaced the attention mechanism of the vanilla transformer model with a de-stationary attention mechanism module that considered the original non-stationary information. Approximating non-stationary information from the original data and statistical quantities through multi-layer perceptrons helped to avoid over-fitting in the predictive model. Lastly, to maintain consistency between the predicted mineral grades and the original data scale range, we de-normalized the predicted values to obtain the final prediction sequence. The combination of normalization and de-normalization enhanced the model’s predictive capability, while the de-stationary attention mechanism could capture the time dependence of the data in the flotation process, reducing prediction errors caused by over-stationarity.

3.1. Normalization

To eliminate the scale differences between sequences and enhance the stability of the distribution of visual features in the sequential froth videos from different input points, a sliding window approach was employed to normalize each dimension of the time series data $X_m^{\prime }$ [26]. This ensured that the data within adjacent windows possessed identical mean and variance. Following the normalization process, the resulting normalized input sequence was denoted as $X^{*}=[X_1^{*},X_2^{*},\cdots ,X_{N_l}^{*}]$.

The specific calculation formulas for the mean value ${\mu }_{X^{\prime }}$, variance ${\sigma }_{X^{\prime }}^2$, original input data $X_m^{\prime }$, and normalized data $X_m^{*}$ of the original matched input data sequence X are respectively shown in Equations (6) to (8):

$${\mu }_{X^{\prime }}=\frac{1}{N_l}\sum _{m=1}^{N_l}{X}_m^{\prime }.$$

(6)

$${\sigma }_{X^{\prime }}^2=\frac{1}{N_l}\sum _{m=1}^{N_l}{(X_m^{\prime }-{\mu }_{X^{\prime }})}^2$$

(7)

$$X_m^{*}=\frac{1}{{\sigma }_{X^{\prime }}}⨀(X_m^{\prime }-{\mu }_{X^{\prime }})$$

(8)

where ⨀ represents the Hadamard product. Such standardized operations could reduce the distributional differences between the floating video time series inputs, thereby making the input distribution of the model more stable.

However, on the other hand, over-stationary processing results in the loss of intrinsic non-stationary information in time series, rendering a model unable to capture the time-dependent nature of mineral grade values in the flotation process, and thus affecting the accuracy of grade prediction outcomes.

To demonstrate the non-stationarity of the flotation data, we chose to display the time series data of the bubble size in the first frame of the froth video detected by the XRF analyzer, with a total of 800 data points and a time interval of 17–18 min between each data point, as shown in Figure 4. The red solid line and black dashed line in the figure represent the average bubble size and variance during the time period, respectively. From Figure 4, it can be observed that the data curve does not have a clear trend, and the mean and variance show different variations. Consequently, this study proposed modifying the attention mechanism of the vanilla transformer model into a de-stationary attention mechanism, aiming to approximate the attention learning in the original non-stationary sequences.

3.2. De-Stationary Attention Mechanism

In the baseline transformer model, the attention mechanism was calculated as follows [13]:

$$Attn(Q,K,V)=Softmax((Q{K}^{\top })/\sqrt{d_k})$$

(9)

where $Q={[q_1,q_2,\cdots ,q_{N_l}]}^{\top },K={[k_1,k_2,\cdots ,k_{N_l}]}^{\top },V={[v_1,v_2,\cdots ,v_{N_l}]}^{\top }$, respectively, represent the query, key, and value in the attention mechanism.

Assuming that the model’s embedding layer and forward propagation layer maintain linear characteristics in the temporal dimension, the Q, K, and V values in the attention mechanism can be calculated through a linear function f [24]. For the normalized input sequence $X^{*}=(X^{\prime }-1{\mu }_{X^{\prime }}^{\top })/{\sigma }_{X^{\prime }}$ of the floating video, where $1\in {\mathbb{R}}^{N_l\times 1}$ is a vector consisting of all ones. The formula for calculating $Q^{*}$ in the attention mechanism is as follows:

$$Q^{*}=\left[\begin{array}{c}{q}_1^{*}\\ q_2^{*}\\ ⋮\\ q_{N_l}^{*}\end{array}\right]=\left[\begin{array}{c}f(X_1^{*})\\ f(X_2^{*})\\ ⋮\\ f(X_{N_l}^{*})\end{array}\right]=\left[\begin{array}{c}f(\frac{X_1^{\prime }-1{\mu }_{X_1^{\prime }}^{\top }}{{\sigma }_{X_1^{\prime }}})\\ f(\frac{X_2^{\prime }-1{\mu }_{X_2^{\prime }}^{\top }}{{\sigma }_{X_2^{\prime }}})\\ ⋮\\ f(\frac{X_{N_l}^{\top }-1{\mu }_{{N_l}^{\prime }}^{\top }}{{\sigma }_{{N_l}^{\prime }}})\end{array}\right]=\left[\begin{array}{c}\frac{f(X_1^{\prime })-1f({\mu }_{X_1^{\prime }}^{\top })}{{\sigma }_{X_1^{\prime }}})\\ \frac{f(X_2^{\prime })-1f({\mu }_{X_2^{\prime }}^{\top })}{{\sigma }_{X_2^{\prime }}})\\ ⋮\\ \frac{f(X_{N_l}^{\prime })-1f({\mu }_{X_{N_l}^{\prime }}^{\top })}{{\sigma }_{X_{N_l}^{\prime }}})\end{array}\right]=\frac{Q-1{\mu }_Q^{\top }}{{\sigma }_{X^{\prime }}}$$

(10)

It is noteworthy that all input feature variables underwent normalization processing, thus ensuring that each variable in the floating video possessed the same variance, which is a constant, denoted as ${\sigma }_{X_1^{\prime }}={\sigma }_{X_2^{\prime }}=\cdots ={\sigma }_{X_{N_l}^{\prime }}={\sigma }_{X^{\prime }}$. ${\mu }_Q\in {\mathbb{R}}^{d_k\times 1}$ represents the average value of Q in the temporal dimension, and similar conclusions can be drawn for $K^{*}$ and $V^{*}$. The equation can be derived as follows:

$$Q^{*}{K}^{*\top }=(Q{K}^{\top }-1({\mu }_Q^{\top }{K}^{\top })-(Q{\mu }_K)1^{\top }+1({\mu }_Q^{\top }{\mu }_K)1^{\top })/{\sigma }_{X^{\prime }}^2$$

(11)

Therefore, the current attention mechanism is calculated as follows:

$$Softmax(\frac{Q{K}^{\top }}{\sqrt{d_k}})=Softmax(\frac{{\sigma }_{X^{\prime }}^2{Q}^{*}{K}^{*\top }+1({\mu }_Q^{\top }{K}^{\top })+(Q{\mu }_K)1^{\top }-1({\mu }_Q^{\top }{\mu }_K)1^{\top }}{\sqrt{d_k}})$$

(12)

According to the shift-invariance property of the Softmax operator [13], Equation (12) can be simplified to Equation (13).

$$Softmax(\frac{Q{K}^{\top }}{\sqrt{d_k}})=Softmax(\frac{{\sigma }_{X^{\prime }}^2{Q}^{*}{K}^{*\top }+1({\mu }_Q^{\top }{K}^{\top })}{\sqrt{d_k}})$$

(13)

From the above formula, it can be inferred that the attention calculation in the current transformer model considers not only the $Q^{*}$ and $K^{*}$ obtained from the stabilized froth video sequence $X^{*}$, but also ${\sigma }_{X^{\prime }}$, ${\mu }_Q$, K from the original froth sequence. This approach significantly mitigated the tendency of the baseline transformer model to predict overly stable mineral grade values.

The key to finding non-stationary factors in the original data is to approximate $\alpha ={\sigma }_{X^{\prime }}^2$ and $\beta =K{\mu }_Q$. However, due to the presence of a nonlinear activation layer in fully connected feedforward neural networks, strict linear properties are difficult to apply in the vanilla transformer model. To weaken the linear assumptions between model layers, this paper employed multi-layer perceptrons (MLP) to directly learn adaptive de-stationary factors from the unsteady correlated statistical data $X^{\prime }$, Q, and K in the froth video sequences. By learning de-stationary factors, the model could better capture essential features in the data, thereby improving the performance. However, we could only learn limited non-stationary information from Q and K, thus necessitating the learning of non-stationary factors from the original froth video sequence $X^{\prime }$. The formula for de-stationary attention is as follows:

$$log\alpha =MLP({\sigma }_{X^{\prime }},X^{\prime })$$

(14)

$$\beta =MLP({\mu }_{X^{\prime }},X^{\prime })$$

(15)

$$Attn(Q^{*},K^{*},V^{*},\alpha ,\beta )=Softmax(\frac{\alpha Q^{*}{K}^{*\top }+1{\beta }^{\top }}{\sqrt{d_k}})V^{*}$$

(16)

The de-stationary of the attention mechanism design could facilitate learning non-stationary factors from the original floating video data $X^{\prime }$, ${\mu }_{X^{\prime }}$, and ${\sigma }_{X^{\prime }}$.

Following the introduction of de-stationary attention mechanism in the encoding stage of the baseline transformer model, we conducted predictions for ore grade.

To enhance predictive accuracy during the model training phase, we normalized the mineral flotation data.

3.3. De-Normalization

The mineral grade prediction results obtained during the model testing phase are denoted as $Y_g^{*}(g=1,2,\cdots ,N_l)$, and the sequence of prediction results is labeled as $Y^{*}=[Y_1^{*},Y_2^{*},\cdots Y_{N_l}^{*}]$.

To maintain the predicted results within the same scale and range as the original data, the final flotation outcome $Y_g^{\prime }$ was obtained by performing inverse normalization [27] on the grade prediction result $Y_g^{*}(g=1,2,\cdots ,N_l)$, based on ${\sigma }_{X^{\prime }}$ and ${\mu }_{X^{\prime }}$.

$$Y_g^{\prime }={\sigma }_{X^{\prime }}⨀(Y_g^{*}+{\mu }_{X^{\prime }}).$$

(17)

The final predicted sequence is denoted as $Y^{\prime }=[Y_1^{\prime },Y_2^{\prime },\cdots Y_{N_l}^{\prime }]$. Through the conversion processes of normalization and de-normalization, the model achieved improved predictive outcomes, while becoming more aligned with the actual grade values.

4. Experiment and Result Analysis

4.1. Dataset

The experimental data presented in this study originated from a lead–zinc flotation plant in Guangdong, China. The dataset was collected between 9 September 2020 and 16 October 2020. The total number of samples amounted to 3004, with a resolution of 692 × 518 pixels for the froth images. The dataset used in this study can be accessed online at http://dx.doi.org/10.21227/q1q5-d663 (accessed on 24 March 2023).

During the experimental process, the dataset was randomly divided into three portions: the training set accounted for 60% and was utilized for model training; the validation set accounted for 20% and was employed for parameter selection; and the testing set accounted for 20% for assessing the performance of the mineral grade prediction model. The performance of the testing set was used as the vanilla standard to evaluate the predictive ability of the model.

This paper employed root mean square error (RMSE), mean absolute error (MAE), and determination coefficient (R²) as evaluation metrics for the model, with their respective calculation formulas presented below:

$$RMSE=\sqrt{\frac{1}{N_l}\sum _{i=1}^{N_l}{(Y_i-Y_i^{\prime })}^2}$$

(18)

$$MAE=\frac{1}{N_l}\sum _{i=1}^{N_l}|Y_i-Y_i^{\prime }|$$

(19)

$$R^2=1-\frac{{\sum }_{i=1}^{N_l}{(Y_i-Y_i^{\prime })}^2}{{\sum }_{i=1}^{N_l}{(Y_i-\overline{Y_i^{\prime }})}^2}$$

(20)

In this context, $Y_i$ and $Y_i^{*}$ represent the actual ore grade measured by the XRF analyzer and the predicted tailing grade value, respectively. $\overline{Y_i^{*}}$ denotes the average ore grade value predicted by the model, and $N_l$ refers to the total number of samples.

4.2. Hyperparameter Selection

The choice of hyperparameters has a significant impact on the predictive performance of a model. In the mineral grade prediction model proposed in this study, the batch size $bs$, learning rate $l_r$, sampling time step T, and number of hidden units N were crucial hyperparameters that required careful selection.

To identify the optimal hyperparameters, a grid search strategy [28] was employed on the validation set for selection. The grid search strategy is an efficient optimization algorithm that divides the search space into a grid. It quickly eliminates impossible solutions, narrows down the search range, and improves efficiency. Meanwhile, tt also has a good parallel processing capability, making it faster and effectively utilizing computer resources. The batch size was chosen in group $\left\{1,2,4,8,16\right\}$; the learning rate was selected in group $\left\{0.1,0.01,0.001,0.0001\right\}$; the sampling time step was decided in group $\left\{1,3,5,7,10\right\}$; and the number of hidden units was determined in group $\left\{1,2,4,8,16\right\}$. Optimal prediction performance is achieved when the loss is minimized, with the loss on the validation set serving as the criterion. The prediction model was trained on the training dataset and validated on the validation dataset. The validation process was randomly executed 50 times, and the average of the validation losses is presented in the Figure 5. When $bs=2$, $l_r=0.001$, $T=10$ and $N=10$, the validation loss was minimal.

To validate the effectiveness of the method proposed in this paper, we compared the proposed model, the DST model, with other state-of-the-art models, including the RNN model [7], the LSTM model [8], the transformer model [13], the Enc-Dec (RNN) model [11], and the STS-D model [9]. For a fair comparison, we selected optimal hyperparameters for the proposed model and these networks on the validation set through a grid search. The optimal hyperparameters are recorded in

Table 1. Optimal hyperparameters for all models.

Method	bs	${\mathit{l}}_{\mathit{r}}$	T	N
DST	2	0.001	10	10
RNN	2	0.001	7	20
LSTM	2	0.001	7	20
Transformer	2	0.001	1	10
Enc-Dec(RNN)	2	0.001	5	10
STS-D	8	0.0001	10	10

.

As shown in Table 1, except for the STS-D model, the other models achieved the minimum validation loss at a batch size of 2, while the STS-D model achieved the minimum at a batch size of 8. Except for the STS-D model, which had the minimum validation loss at a learning rate of 0.0001, the other models achieved the lowest validation loss at a learning rate of 0.001. When the learning rate is too low, models tend to fall into local optima. The optimal sampling step size for most models ranged between 5 and 10, with the hidden unit count typically being 20.

After conducting 50 random test dataset runs, each comparative model took an average value as the experimental result.

Table 2. Prediction results for the different models. Data format: mean ± (standard deviation).

Method	RMSE	MAE	${\mathit{R}}^2$
DST	0.1582 ± (0.0043)	0.3955 ± (0.0025)	0.8978 ± (0.0203)
RNN	0.1998 ± (0.0001)	0.4436 ± (0.0002)	0.8862 ± (0.0001)
LSTM	0.1998 ± (0.0001)	0.4436 ± (0.0001)	0.8862 ± (0.0001)
Transformer	0.4216 ± (0.1112)	0.8650 ± (0.1981)	0.8309 ± (0.0203)
Enc-Dec(RNN)	0.1990 ± (0.0001)	0.4457 ± (0.0002)	0.8903 ± (0.0001)
STS-D	0.3110 ± (0.1050)	0.6711 ± (0.2051)	0.8673 ± (0.0205)

displays a comparison of the evaluation index data among the various models, with the DST model exhibiting the best performance for all evaluation metrics. Compared to the RNN, LSTM, transformer, Enc-Dec (RNN), and STS-D models, the DST model’s RMSE was reduced by 20.8%, 20.8%, 62.8%, 20.5%, and 49.1%, respectively. Considering the non-stationary information of the original mineral data, the DST model’s MAE values were respectively 10.8%, 10.8%, 54.3%, 11.3%, and 41.2% lower than those of the other models. Furthermore, the DST model achieved the highest $R^2$ value, indicating a better fitting efficacy and predicted values closer to the actual mineral grade. Compared to the other five models, the DST model’s values increased by 0.0116, 0.0116, 0.0669, 0.0075, and 0.0305, respectively.

The above evidence indicates that the mineral grade prediction model proposed in this study, which considered non-stationary information in the original mineral data, exhibited better accuracy compared to other models that did not consider this information. This is primarily because the proposed grade prediction model not only enhances the predictability of the mineral grade sequence by utilizing a combination of normalization and de-normalization modules but also takes into account the non-stationary information present in the original mineral data, thereby reducing the occurrence of overfitting due to excessive smoothing.

To further demonstrate the effectiveness of the proposed model, the predicted values of all models were compared with the true values on the test data. As illustrated in Figure 6, the blue line represents the observed grade values measured by the XRF analyzer, while the red line denotes the predicted values from the model. As can be seen more intuitively from Figure 6, the fitting effect of the method proposed in this paper was superior to the other comparison methods. Specifically, compared with the RNN model, LSTM model, and Enc-Dec (RNN) model, the method proposed by us had a higher accuracy and stability of fitting effect; compared with the STS-D model, our method had better dynamic adaptability in predicting non-stationary time series; and in comparison with the transformer model, the method proposed by us exhibited stronger robustness in dealing with non-stationary data.

5. Conclusions

This study enhanced the predictive capability of the model by normalizing matched flotation data videos, and introducing a de-stationary attention mechanism module that considered original non-stationary information, replacing the attention mechanism of the vanilla transformer model. Furthermore, the study saved the mean and variance of the original data to learn relevant non-stationary information and approximated non-stationary information through multi-layer perceptrons to prevent overfitting of the predictive model. After obtaining the predicted values of mineral grades, the final predicted sequence was obtained through inverse normalization. In the real-time dataset experiment, the proposed mineral grade prediction method demonstrated the best fitting effect. This confirmed the effectiveness of the combination of normalization and de-normalization, as well as the destabilizing attention mechanism, which enabled the model to capture the time-dependency of data in the flotation process, thereby reducing prediction errors caused by over-stationarity. In addition, since the dataset used in this method only covered the froth image feature data of zinc ore in a specific period, in future research, we will consider the characteristics of the feed in the flotation process, to make the proposed method more universal and widely applicable.