The Robust Supervised Learning Framework: Harmonious Integration of Twin Extreme Learning Machine, Squared Fractional Loss, Capped L_2,p-norm Metric, and Fisher Regularization

Abstract

As a novel learning algorithm for feedforward neural networks, the twin extreme learning machine (TELM) boasts advantages such as simple structure, few parameters, low complexity, and excellent generalization performance. However, it employs the squared $L_2$-norm metric and an unbounded hinge loss function, which tends to overstate the influence of outliers and subsequently diminishes the robustness of the model. To address this issue, scholars have proposed the bounded capped $L_{2,p}$-norm metric, which can be flexibly adjusted by varying the p value to adapt to different data and reduce the impact of noise. Therefore, we substitute the metric in the TELM with the capped $L_{2,p}$-norm metric in this paper. Furthermore, we propose a bounded, smooth, symmetric, and noise-insensitive squared fractional loss (SF-loss) function to replace the hinge loss function in the TELM. Additionally, the TELM neglects statistical information in the data; thus, we incorporate the Fisher regularization term into our model to fully exploit the statistical characteristics of the data. Drawing upon these merits, a squared fractional loss-based robust supervised twin extreme learning machine (SF-RSTELM) model is proposed by integrating the capped $L_{2,p}$-norm metric, SF-loss, and Fisher regularization term. The model shows significant effectiveness in decreasing the impacts of noise and outliers. However, the proposed model’s non-convexity poses a formidable challenge in the realm of optimization. We use an efficient iterative algorithm to solve it based on the concave-convex procedure (CCCP) algorithm and demonstrate the convergence of the proposed algorithm. Finally, to verify the algorithm’s effectiveness, we conduct experiments on artificial datasets, UCI datasets, image datasets, and NDC large datasets. The experimental results show that our model is able to achieve higher ACC and $F_1$ scores across most datasets, with improvements ranging from 0.28% to 4.5% compared to other state-of-the-art algorithms.

1. Introduction

In the field of machine learning, researchers have been dedicated to enhancing the efficiency and accuracy of models. Sakheta et al. [1] improved the prediction of the biomass gasification model through six machine learning algorithms. The research demonstrated that the XGBoost algorithm has significant advantages in improving the accuracy of gasification product prediction. Maydanchi et al. [2] systematically compared various machine learning methods and found that tree-based ensemble methods, such as XGBoost, gradient boosting, and random forest, excelled in diabetes prediction. Kim et al. [3] successfully classified three similar enterococci by combining MALDI-TOF mass spectrometry techniques and multiple supervised learning algorithms (e.g., KNN, SVM, random forest). Although these methods have made significant progress in different domains, there remains room for improvement in enhancing computational efficiency and response times. The extreme learning machine (ELM) [4,5] offers a promising solution to these challenges with its efficient training process and superior generalization capabilities. It was first proposed by Huang et al. [6] and quickly gained widespread application in multiple fields, including image classification [7], fault detection [8,9], disease diagnosis [10], computer vision [11], face recognition [12], and signal processing [13]. These application cases fully validate the practicability and effectiveness of ELM as an efficient neural network training method.

In binary classification tasks, traditional ELM only learns a single hyperplane to distinguish between classes. Recently, two nonparallel hyperplanes classification algorithms have attracted significant attention and research interest [14,15]. These algorithms involve the training of multiple hyperplanes, where each hyperplane is designed to minimize its distance to one of the two classes while maximizing its distance from the other class. For example, the twin support vector machine (TSVM) is notable for its efficiency in learning two nonparallel separating hyperplanes more quickly than the traditional support vector machine (SVM) by solving two reduced-sized quadratic programming problems (QPPs). The various variants of TSVM [16,17,18,19] have been extensively studied and have been successfully applied in classification tasks.

Inspired by TSVM, Wan et al. [20] proposed the twin extreme learning machine (TELM). It is noteworthy that the TELM and TSVM use the hinge loss function, which is unbounded and tends to exaggerate the impact of noise and outliers on the model. Consequently, the research community has expressed increasing interest in exploring alternative loss functions. Wang et al. [21] proposed a new robust capped $L_1$-norm twin support vector machine (CTWSVM), which maintains the benefits of TWSVM and enhances the robustness of the model. Wang and Yu et al. [22] proposed a new robust loss function, the capped Linex loss function, which was applied to the TSVM to enhance the classification capabilities of the model. Kumari A et al. [23] introduced the capped pinball loss function into the universum twin support vector machine (UTWSVM), and proposed a universum twin support vector machine (Tpin-UTWSVM) based on capped pinball loss function, which improved the model’s generalization performance. Ma et al. [24] proposed a robust adaptive capped $L_{\theta \epsilon }$ loss, altering the loss function value by adjusting the adaptive parameter $\theta$ during the training process. Applying this loss function to TSVM, an adaptive robust learning framework was proposed, namely the adaptive robust twin support vector machine (ARTSVM). All the above models use bounded capped loss functions, which constrain the impact of noise within certain limits and make the classifiers less sensitive to noise.

In order to further reduce the impact of noise, many scholars have begun to look for new metrics to substitute for the squared $L_2$-norm metric used in the TELM. Ma et al. [25] proposed a fast robust twin extreme learning machine (FRTELM) based on capped $L_1$-norm metric and loss function in the classic TELM learning framework, which enhances the robustness of the TELM in handling classification problems. Yang et al. [26] added the idea of projection on the basis of the twin extreme learning machine, and combining this with the capped $L_1$-norm metric and loss function, they proposed a new capped $L_1$-norm projection twin extreme learning machine ($C{L}_1$-PTELM). It lessens the influence of outliers and demonstrates more robustness than the TELM. Ma and Yang et al. [27] proposed a new robust TELM framework (RTELM) using the capped $L_1$-norm metrics and capped $L_{\theta \epsilon }$ loss function. RTELM addresses the limitations of $L_2$-norm metric and hinge loss, particularly in scenarios with outliers. It retains the strengths of the TELM and further enhances the robustness of classification. These algorithms show that the capped $L_1$-norm metric is resistant to outliers. In fact, the capped $L_1$-norm metric is considered an effective approximation of the $L_0$-norm by a non-negative parameter, and it is superior in robustness to the $L_1$-norm metric [27]. In addition, related scholars have begun to focus on the capped $L_{2,p}$-norm metric and have applied it to their models. This metric is bounded and can be flexibly tuned by adjusting the p-value to adapt to diverse datasets and reduce the effect of noise. Yuan et al. [28] created a novel framework to improve robustness by substituting the squared $L_2$-norm metric with the robust capped $L_{2,p}$-norm metric in a least squares twin support vector machine (LSTSVM), which is called capped $L_{2,p}$-norm LSTSVM ($C{L}_{2,p}$-LSTSVM). Wang et al. [29] proposed a capped $L_{2,p}$-norm metric based on the robust twin support vector machine with Welsch loss function (WCTBSVM). The generalization performance and robustness of the TSVM are further improved. Jiang et al. [30] proposed a novel robust twin extreme learning machine learning framework (CWTELM) by combining the capped $L_{2,p}$-norm metric and Welsch loss function with the TELM. CWTELM improves robustness while preserving the advantages of TELM, thereby enhancing classification performance.

Besides altering metrics and loss functions, regularization techniques play a vital role in improving the generalization capabilities of models. The Fisher regularization term is a notable technique that minimizes within-class variance and excels in improving class separability and robustness. Ma and Wen et al. [31] proposed a Fisher regularization ELM (Fisher-ELM) to reach a minimal within-class scatter. Fisher-ELM utilizes the statistical properties of the data, which exhibits excellent generalization ability. Although Fisher-ELM incorporates statistical knowledge into its framework, it tends to ignore the potential effects of noise or outliers. To reduce the negative effects of these factors, Xue and Zhao et al. [32] first proposed a novel asymmetric Welsch loss function and integrated it into Fisher-ELM, then proposed a robust Fisher regularization extreme learning machine with asymmetric Welsch-induced loss function (AWFisher-ELM). This model better copes with the adverse effects of noise and outliers, enhancing the robustness of the model. Xue et al. [33] added Fisher regularization to the TELM and proposed Fisher regularization TELM (FTELM), which both keeps the strengths of the TELM and minimizes the intra-class differences of samples. In order to further improve the noise immunity of the FTELM method, a new capped $L_1$-norm Fisher regularization TELM (C$L_1$-FTELM) is proposed by combining the capped $L_1$-norm metric and loss function to enhance the robustness of the model.

In this paper, we first propose a bounded, smooth, and symmetrical squared fractional loss (SF-loss). Based on the proposed SF-loss, we also integrate the TELM, capped $L_1$-norm metric, and Fisher regularization and propose a robust supervised TELM learning framework (SF-RSTELM). SF-RSTELM can effectively utilize the statistical properties of the data, which the TELM lacks. In addition, it can effectively reduce the impact of noise and outliers by employing the bounded capped $L_{2,p}$-norm metric and SF-loss function. In contrast, the TELM uses the unbounded squared $L_2$-norm metric and hinge loss, which are susceptible to the influence of noise and outliers.

The main work of this paper is summarized as follows:

(1)

A new robust loss function called squared fractional loss (SF-loss) is presented. It has some important properties such as being bounded, smooth, symmetric, and noise-insensitive. Moreover, the robustness of the SF-loss is analyzed according to the perspective of M estimation theory [34], and its Fisher consistency is proved according to the Bayesian rule [35].

(2)

An innovative method named “The Robust Supervised Learning Framework: Harmonious Integration of Twin Extreme Learning Machine, Squared Fractional Loss, Capped $L_{2,p}$-norm Metric, and Fisher Regularization” is proposed. This framework cleverly combines the efficiency of the TELM, the robustness of the SF-loss function, the flexibility of the capped $L_{2,p}$-norm metric, and the advantages of Fisher regularization. This integrated approach not only takes into account the statistical information of the data but also significantly reduces the impact of noise, thereby enhancing the model’s performance.

(3)

Due to the non-convex nature of the established optimization model, an efficient algorithm based on CCCP [36] is proposed to solve the optimization problem. Moreover, the convergence of the proposed algorithm is proved.

(4)

We performed extensive experiments on artificial datasets, UCI datasets, image datasets, and NDC-large datasets to validate the effectiveness of our proposed algorithm compared to other state-of-the-art algorithms.

The rest of this paper is structured as follows. In Section 2, we briefly review related work on Fisher regularization, the Fisher regularized twin extreme learning machine, the capped $L_{2,p}$-norm metric, and the concave-convex procedure. In Section 3, we provide a comprehensive description of the proposed model and a detailed solution process. The experimental results on multiple datasets are presented in Section 4. Conclusions and suggestions for future work are given in Section 5.

2. Related Work

In this section, we briefly review related work on Fisher regularization, the Fisher regularized twin extreme learning machine, the Capped $L_{2,p}$-norm metric, and the concave-convex procedure.

2.1. Fisher Regularization

Fisher regularization [32] can measure the intra-class divergence within the data, facilitating the development of more effective learning models. With the training set $\mathcal{T}=\left\{(x_1,y_1),\dots ,(x_m,y_m)\right\}$, the Fisher regularization has the form:

$${\parallel f\parallel }_F^2=\sum _{i\in {\mathcal{I}}_{+}}{\left(f\left(x_i\right)-{\overline{f}}_{+}\right)}^2+\sum _{i\in {\mathcal{I}}_{-}}{\left(f\left(x_i\right)-{\overline{f}}_{-}\right)}^2$$

(1)

where f is the prediction function, and $f\left(x_i\right)$ represents the value of f on sample $x_i$; f on all samples forms a vector $f={\left[f\left(x_1\right),f\left(x_2\right),\dots ,f\left(x_m\right)\right]}^T$; ${\overline{f}}_{+}$ and ${\overline{f}}_{-}$ represent the mean of f on all positive and negative samples, respectively; and ${\mathcal{I}}_{+}$ and ${\mathcal{I}}_{-}$ represent the index collections of positive and negative samples.

We can expand Equation (1):

$$\begin{array}{c}{\parallel f\parallel }_F^2=\sum _{i\in {\mathcal{I}}_{+}}{\left(f\left(x_i\right)-{\overline{f}}_{+}\right)}^2+\sum _{i\in {\mathcal{I}}_{-}}{\left(f\left(x_i\right)-{\overline{f}}_{-}\right)}^2\hfill \\ =\sum _{i\in {\mathcal{I}}_{+}}\left(f^2\left(x_i\right)-2·f\left(x_i\right)·{\overline{f}}_{+}+f_{+}^2\right)+\sum _{i\in {\mathcal{I}}_{-}}\left(f^2\left(x_i\right)-2·f\left(x_i\right)·{\overline{f}}_{-}+f_{-}^2\right)\hfill \\ =\left(\sum _{i\in {\mathcal{I}}_{+}}{f}^2\left(x_i\right)-2·m_1·{\overline{f}}_{+}^2+m_1·{\overline{f}}_{+}^2\right)+\left(\sum _{i\in {\mathcal{I}}_{-}}{f}^2\left(x_i\right)-2·m_2·{\overline{f}}_{-}^2+m_2·{\overline{f}}_{-}^2\right)\hfill \\ =\left(\sum _{i\in {\mathcal{I}}_{+}}{f}^2\left(x_i\right)-m_1·{\overline{f}}_{+}^2\right)+\left(\sum _{i\in {\mathcal{I}}_{-}}{f}^2\left(x_i\right)-m_2·{\overline{f}}_{-}^2\right)\hfill \\ =f_{+}^T·f_{+}-{\displaystyle \frac{1}{m_1}}·\left(f_{+}^T·e^T·e·f_{+}\right)+f_{-}^T·f_{-}-{\displaystyle \frac{1}{m_2}}·\left(f_{-}^T·e^T·e·f_{-}\right)\hfill \\ =f_{+}^T·I_{+}·f_{+}-f_{+}^T·M_{+}·f_{+}+f_{-}^T·I_{-}·f_{-}-f_{-}^T·M_{-}·f_{-}\hfill \\ =f_{+}^T·\left(I_{+}-M_{+}\right)·f_{+}+f_{-}^T·\left(I_{-}-M_{-}\right)·f_{-}\hfill \\ ={\left(f_{+},f_{-}\right)}^T·\left[I-\left[\begin{array}{cc}{M}_{+}& 0_1\\ 0_2& M_{-}\end{array}\right]\right]·\left(f_{+},f_{-}\right)\hfill \\ =f^T·(I-G)·f=f^T·N·f\hfill \end{array}$$

(2)

where $f_{+}=\left(f\left(x_1\right),f\left(x_2\right),\dots ,f\left(x_i\right),\dots ,f\left(x_{m_1}\right)\right)$, $i\in {\mathcal{I}}_{+}$, $I_{+}\in R^{m_1\times m_1}$ is the identity matrix. $f_{-}=\left(f\left(x_1\right),f\left(x_2\right),\dots ,f\left(x_i\right),\dots ,f\left(x_{m_2}\right)\right)$, $i\in {\mathcal{I}}_{-}$, $I_{-}\in R^{m_2\times m_2}$ is the identity matrix. $0_1\in 0^{m_1\times m_2}$, $0_2\in 0^{m_2\times m_1}$, $M_{+}\in R^{m_1\times m_1}$, and all the elements in the matrix $M_{+}$ are ${\displaystyle \frac{1}{m_1}}$; $M_{-}\in R^{m_2\times m_2}$, and all the elements in the matrix $M_{-}$ are ${\displaystyle \frac{1}{m_2}}$; $N=I-G$; and $I\in R^{m\times m}$ is the identity matrix $G=\left[\begin{array}{cc}{M}_{+}& 0_1\\ 0_2& M_{-}\end{array}\right]$.

2.2. Fisher Regularized Twin Extreme Learning Machine

Within a supervised classification framework, the training dataset is typically represented as $\mathcal{T}={\left\{\left(x_i,y_i\right)\right\}}_{i=1}^l$, where $x_i\in R^d$, $y_i\in \left\{1,-1\right\}$. The set $\mathcal{T}$ includes $l_1$ positive class samples and $l_2$ negative class samples, where $l=l_1+l_2$.

The TELM is a traditional and highly efficient classifier. Nevertheless, it overlooks the statistical properties contained within the data. Drawing inspiration from Fisher’s concepts, Xue et al. [33] proposed Fisher-TELM (FTELM) by introducing Fisher regularization terms into the TELM learning framework. Specifically, the primal FTELM is given as:

$$\begin{array}{cc}\hfill PrimalFTEL{M}_1:& \underset{{\beta }_1,{\xi }_1}{min}{\displaystyle \frac{1}{2}}{∥{\mathit{H}}_1{\beta }_1∥}_2^2+C_1{\mathit{e}}_2^T{\xi }_1+{\displaystyle \frac{C_2}{2}}{f}_1{\left(x\right)}^T{\mathit{N}}_1{f}_1\left(x\right)\hfill \\ \hfill & s.t.-{\mathit{H}}_2{\beta }_1+{\xi }_1\ge {\mathit{e}}_2\hfill \\ \hfill & {\xi }_1\ge \mathbf{0}\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill PrimalFTEL{M}_2:& \underset{{\beta }_2,{\xi }_2}{min}{\displaystyle \frac{1}{2}}{∥{\mathit{H}}_2{\beta }_2∥}_2^2+C_3{\mathit{e}}_1^T{\xi }_2+{\displaystyle \frac{C_4}{2}}{f}_2{\left(x\right)}^T{\mathit{N}}_2{f}_2\left(x\right)\hfill \\ \hfill & s.t.{\mathit{H}}_1{\beta }_2+{\xi }_2\ge {\mathit{e}}_1\hfill \\ \hfill & {\xi }_2\ge \mathbf{0}\hfill \end{array}$$

(4)

where ${\mathit{N}}_1={\mathit{I}}_{+}-{\mathit{M}}_{+}$, ${\mathit{N}}_2={\mathit{I}}_{-}-{\mathit{M}}_{-}$, ${\beta }_1$ and ${\beta }_2$ represent the output weights connecting the hidden layer to the output layer, and $\mathbf{h}\left(\mathit{x}\right)$ represents the hidden layer output matrix given by $\mathit{h}\left(\mathit{x}\right)=(h_1\left(\mathit{x}\right),h_2\left(\mathit{x}\right),\dots ,h_L\left(\mathit{x}\right))\in {\mathbb{R}}^{1\times L}$. Here, L represents the count of hidden nodes, with each node function $h_i\left(x\right)=G({\sum }_{j=1}^d{x}_j{\omega }_{ji}+b_i)$. G denotes an activation function, with frequently used examples being the sigmoid function $G\left(x\right)={\displaystyle \frac{1}{1+e^{-x}}}$, the ReLU function $G\left(x\right)=max\left(0,x\right)$, etc. In this context, $w_i=\left({\omega }_{i,1},{\omega }_{i,2},\dots ,{\omega }_{i,d}\right)$ stands for input weight, ${\omega }_{j,i}$ denotes the input weight connecting the j-th feature to the i-th hidden node, and $b_i$ is the bias term for the i-th hidden node $(i=1,2,\dots ,$ L). d represents the sample dimension. ${\mathit{H}}_1={\left(\begin{array}{c}h{\left(x_1\right)}^T,h{\left(x_2\right)}^T,\dots ,h{\left(x_{l_1}\right)}^T\end{array}\right)}^T\in {\mathbb{R}}^{l_1\times L}$ and ${\mathit{H}}_2={\left(\begin{array}{c}h{\left(x_1\right)}^T,h{\left(x_2\right)}^T,\dots ,h{\left(x_{l_2}\right)}^T\end{array}\right)}^T\in {\mathbb{R}}^{l_2\times L}$ denote the hidden layer outputs for the positive class and negative class samples, respectively. $C_1$, $C_2$, $C_3$, $C_4>0$ are regularization parameters, while ${\mathit{e}}_1\in R^{m_1}$ and ${\mathit{e}}_2\in R^{m_2}$ are vectors of ones.

According to the representer theorem, $f_1\left(x_1\right)={\mathit{H}}_1{\beta }_1$, $f_2\left(x_2\right)={\mathit{H}}_2{\beta }_2$. Therefore, Equations (3) and (4) can be rephrased as follows:

$$\begin{array}{cc}\hfill & \underset{{\beta }_1,{\xi }_1}{min}{\displaystyle \frac{1}{2}}{∥{\mathit{H}}_1{\beta }_1∥}_2^2+C_1{\mathit{e}}_2^T{\xi }_1+{\displaystyle \frac{C_2}{2}}{\beta }_1^T{\mathit{H}}_1^T{\mathit{N}}_1{\mathit{H}}_1{\beta }_1\hfill \\ \hfill & s.t.-{\mathit{H}}_2{\beta }_1+{\xi }_1\ge {\mathit{e}}_2\hfill \\ \hfill & {\xi }_1\ge \mathbf{0}\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill & \underset{{\beta }_2,{\xi }_2}{min}{\displaystyle \frac{1}{2}}{∥{\mathit{H}}_2{\beta }_2∥}_2^2+C_3{\mathit{e}}_1^T{\xi }_2+{\displaystyle \frac{C_4}{2}}{\beta }_2^T{\mathit{H}}_2^T{\mathit{N}}_2{H}_2{\beta }_2\hfill \\ \hfill & s.t.{\mathit{H}}_1{\beta }_2+{\xi }_2\ge {\mathit{e}}_1\hfill \\ \hfill & {\xi }_2\ge \mathbf{0}\hfill \end{array}$$

(6)

The positive training point can be made to approach the hyperplane $f_1$ as closely as possible by optimizing the first term in the objective function (5). Minimizing the second term ensures that the negative class samples are as far as possible from the positive class hyperplane $f_1$. The last term is a Fisher regularization term that minimizes the within-class scatter. For problem (6), we can also use a similar meaning to explain it.

By incorporating Lagrange multipliers ${\theta }_1$ and ${\lambda }_1$, the dual problem of (5) can be formulated as follows:

$$\begin{array}{cc}\hfill DualFTEL{M}_1:& \underset{{\theta }_1}{min}{\displaystyle \frac{1}{2}}{\theta }_1^T{\mathit{Q}}_1{\theta }_1-{\mathit{e}}_2^T{\theta }_1\hfill \\ \hfill & s.t.\mathbf{0}\le {\theta }_1\le C_1{\mathit{e}}_2\hfill \end{array}$$

(7)

Here, ${\mathit{Q}}_1={\mathit{H}}_2{\left({\mathit{H}}_1^T\left({\mathit{I}}_1+C_2{\mathit{N}}_1\right){\mathit{H}}_1\right)}^{-1}{\mathit{H}}_2^T$. Similarly, we can obtain the dual of (6) as:

$$\begin{array}{cc}\hfill DualFTEL{M}_2:& \underset{{\lambda }_1}{min}{\displaystyle \frac{1}{2}}{\lambda }_1^T{\mathit{Q}}_2{\lambda }_1-{\mathit{e}}_1^T{\lambda }_1\hfill \\ \hfill & s.t.\mathbf{0}\le {\lambda }_1\le C_3{\mathit{e}}_1\hfill \end{array}$$

(8)

where ${\mathit{Q}}_2={\mathit{H}}_1{\left({\mathit{H}}_2^T·\left({\mathit{I}}_2+C_4{\mathit{N}}_2\right){\mathit{H}}_2\right)}^{-1}{\mathit{H}}_1^T$.

By solving problem (7) and problem (8), we get ${\theta }_1$ and ${\lambda }_1$. Then, we get:

$${\beta }_1=-{\left({\mathit{H}}_1^T\left({\mathit{I}}_1+C_2{\mathit{N}}_1\right){\mathit{H}}_1\right)}^{-1}{\mathit{H}}_2^T\theta$$

(9)

$${\beta }_2={\left({\mathit{H}}_2^T\left({\mathit{I}}_2+C_4{\mathit{N}}_2\right){\mathit{H}}_2\right)}^{-1}{\mathit{H}}_1^T\lambda$$

(10)

With ${\beta }_1$ and ${\beta }_2$ determined, we classify a new sample point $\mathit{x}$ using the following decision function:

$$f\left(\mathit{x}\right)=arg\underset{k=1,2}{min}{d}_k\left(\mathit{x}\right)=arg\underset{k=1,2}{min}\left|\mathit{h}\left(\mathit{x}\right){\beta }_k\right|$$

(11)

2.3. Capped $L_{2,p}$-Norm Metric

The squared $L_2$-norm is frequently employed in TELM-related variant classifiers because it is differentiable and easier to optimize. Nevertheless, the squared term heightens the impact of outliers, thereby diminishing the model’s classification performance. Yuan et al. [28] proposed the $L_{2,p}$-norm and capped $L_{2,p}$-norm to enhance the model’s robustness to outliers by making p fall inside the range of (0, 2].

For any vector $s\in R^n$ and parameter $0<p\le 2$, with thresholding parameter $\epsilon \ge 0$, the $L_{2,p}$-norm and capped $L_{2,p}$-norm are the following formulas (12) and (13), respectively.

$$f_1\left(s\right)={\left(\sum _{i=1}^n{s}_i^2\right)}^{{\displaystyle \frac{p}{2}}}$$

(12)

$$f_2\left(s\right)=min\left({\left(\sum _{i=1}^n{s}_i^2\right)}^{{\displaystyle \frac{p}{2}}},\epsilon \right)$$

(13)

We also provide a comparison of the $L_1$-norm, the $L_2$-norm, and the capped $L_{2,p}$-norm (p = 0.5, 1, 1.5, 2) of a scalar in Figure 1. Firstly, from Figure 1, we can see that the $L_1$-norm and $L_2$-norm are unbounded, and the capped $L_{2,p}$-norm is bounded. Secondly, we can observe that the capped $L_{2,p}$-norm can be the capped $L_1$-norm when $p=1$, and the capped $L_{2,p}$-norm can be the capped $L_2$-norm when $p=2$. This indicates that the capped $L_{2,p}$-norm can behave as a capped form of the traditional norm at certain parameter values.

To more intuitively help us understand the characteristics of the capped $L_{2,p}$-norm metric, we also provide a comparison of the $L_{2,p}$-norm and capped $L_{2,p}$-norm for a two-dimensional vector (the high-dimensional situation is similar) as shown in Figure 2. Figure 2a,b are the $L_{2,p}$-norm metric when p takes 1 and 2, respectively, while Figure 2c,d are corresponding capped versions. Figure 2a is an unbounded $L_2$-norm metric, whose surface is a smooth curve surface. Figure 2b is essentially an unbounded squared $L_2$-norm metric, which is also a smooth surface, similar to Figure 2a. However, due to the influence of the square, the surface rises rapidly away from the center. Applying it to a model means that it is very sensitive to data points that are far from the center, which can be noise or outliers. Figure 2a also has a relatively sharp turning point, but the overall rise is slow. This norm metric is less sensitive to outliers compared to the squared $L_2$-norm metric in Figure 2b. Figure 2c,d are the bounded capped $L_2$-norm metric and capped squared $L_2$-norm metric, respectively, characterized by flat regions on their surface when the capped threshold is exceeded. If this metric is applied to the model, it can control the impact of the outliers, and the robustness of the model is enhanced.

In conclusion, the capped $L_{2,p}$-norm metric is a better choice when handling datasets containing outliers or noise. Furthermore, the metric can better strengthen the model’s resilience and improve the classification ability of the model.

2.4. Concave-Convex Procedure

The concave-convex procedure (CCCP) [36] is employed to address optimization problems involving the difference of convex functions. Let $x\in R^n$ be the variable; the optimization problem associated with the CCCP is expressed in the following form:

$$\begin{array}{cc}\hfill & \underset{x}{min}f\left(x\right)\hfill \\ \hfill & s.t.c_i\left(x\right)\le 0,i=1,\dots ,p_1\hfill \\ \hfill & d_j\left(x\right)=0,j=1,\dots ,p_2\hfill \end{array}$$

(14)

where $f\left(x\right)=g\left(x\right)-h\left(x\right)$; $g(.)$ and $h(.)$ are realvalued convex functions; and $p_1$ and $p_2$ denote the number of constraints. Suppose that $h(.)$ is differentiable. The solution to (14) is derived through an iterative process of solving the subsequent series of convex optimization problems:

$$\begin{array}{c}\hfill x^{k+1}\in arg\underset{x}{min}g\left(x\right)-x^T\nabla h\left(x^k\right)\\ \hfill s.t.c_i\left(x\right)\le 0,i=1,\dots ,p_1\\ \hfill d_j\left(x\right)=0,j=1,\dots ,p_2\end{array}$$

(15)

3. Squared Fractional Loss Based Robust Supervised Twin Extreme Learning Machine

In this section, we first put forward a new loss function (SF-loss) and then combine it with the TELM, capped $L_{2,p}$-norm metric, and Fisher regularization to propose a new robust supervised learning framework (RS-SFTELM). We also provide a detailed solution process and a convergence analysis for this model.

3.1. Squared Fractional Loss

Convex loss functions ($L_1$ loss function, $L_2$ loss function, hinge loss function) are commonly utilized in machine learning due to their ability to achieve global optimality. However, their unbounded nature makes them vulnerable to noise and outliers. According to M-estimation theory [34], loss functions with bounded properties or bounded influence functions demonstrate greater robustness to noise and outliers. Thus, we propose a new bounded loss function, called squared fraction loss (abbreviated as SF-loss) in the following:

Definition 1. 

Given a vector u, the SF-loss is defined as

$$L_r\left(u\right)={\displaystyle \frac{u^2}{r{u}^2+1}}$$

(16)

where the parameter $r\in (0,+\infty )$.

Figure 3 shows the $L_r\left(u\right)$ loss function with different parameter r values. From Figure 3, we can see that the parameter r affects the upper bound of the loss function. The more r values, the smaller the upper bounds. In addition, we provide some interesting properties, robustness analysis, and Fisher consistency for our SF-loss function.

3.1.1. The Properties of the SF-Loss Function

Property 1. 

$L_r\left(u\right)$ = 0 if $u=0$. This guarantees that the function $L_r\left(u\right)$ passes through the original point.

Property 2. 

$L_r\left(u\right)$ is bounded, which can ensure better robustness.

Proof. 

$$\underset{u\to \infty }{lim}{L}_r\left(u\right)=\underset{u\to \infty }{lim}{\displaystyle \frac{u^2}{r{u}^2+1}}=\underset{u\to \infty }{lim}{\displaystyle \frac{1}{r+\frac{1}{u^2}}}={\displaystyle \frac{1}{r}}$$

(17)

Therefore, $L_r\left(u\right)$ is a bounded function. □

Property 3. 

$L_r\left(u\right)$ is a differentiable function that can help us optimize better.

Proof. 

$$L_r^{\prime }\left(u\right)={\displaystyle \frac{2u}{{\left(r{u}^2+1\right)}^2}}$$

(18)

Therefore, $L_r\left(u\right)$ is differentiable. □

Property 4. 

$L_r\left(u\right)$ is a symmetrical function.

Proof. 

Calculate $L_r\left(u\right)$ and $L_r(-u)$ separately and obtain:

$$L_r\left(u\right)={\displaystyle \frac{u^2}{r{u}^2+1}}$$

(19)

$$L_r(-u)={\displaystyle \frac{{(-u)}^2}{r{(-u)}^2+1}}={\displaystyle \frac{u^2}{r{u}^2+1}}$$

(20)

Since $L_r\left(u\right)=L_r(-u)$, $L_r\left(u\right)$ is symmetrical. □

Property 5. 

$L_r\left(u\right)$ is a non-convex function.

Proof. 

Let $r=1$, $\lambda ={\displaystyle \frac{1}{2}}$, $u_1=1$, and $u_2=2$. Then, we obtain:

$$L_r\left(\lambda u_1+\left(1-\lambda \right)u_2\right)=L_1({\displaystyle \frac{1}{2}}·1+{\displaystyle \frac{1}{2}}·2)=L_1\left({\displaystyle \frac{3}{2}}\right)={\displaystyle \frac{{\left(\frac{3}{2}\right)}^2}{{\left(\frac{3}{2}\right)}^2+1}}={\displaystyle \frac{2.25}{3.25}}\approx 0.692$$

(21)

$$\lambda L_r\left(u_1\right)+\left(1-\lambda \right)L_r\left(u_2\right)={\displaystyle \frac{1}{2}}·L_1\left(1\right)+{\displaystyle \frac{1}{2}}·L_1\left(2\right)={\displaystyle \frac{1}{2}}·{\displaystyle \frac{1}{2}}+{\displaystyle \frac{1}{2}}·{\displaystyle \frac{4}{5}}={\displaystyle \frac{13}{20}}=0.65$$

(22)

Since $0.692>0.65$, i.e., $L_r\left(\lambda u_1+\left(1-\lambda \right)u_2\right)>\lambda L_r\left(u_1\right)+\left(1-\lambda \right)L_r\left(u_2\right)$, $L_r\left(u\right)$ is a non-convex function. □

3.1.2. Robustness Analysis of SF-Loss Function

Clearly, the new loss function $L_r\left(u\right)$ is bounded. From a robust statistics perspective, the $L_r\left(u\right)$ shows noise insensitivity, which ensures superior robustness. The derivative of $L_r\left(u\right)$ is expressed as:

$$L_r^{\prime }\left(u\right)={\displaystyle \frac{2u}{{\left(r{u}^2+1\right)}^2}}$$

(23)

and we have:

$$\underset{u\to \infty }{lim}{\displaystyle \frac{2u}{{\left(r{u}^2+1\right)}^2}}=0$$

(24)

Hence, according to M-estimation theory [34], the loss function is robust against noise.

3.1.3. Fisher Consistency of SF-Loss Function

An important attribute for a binary classifier f: $X⟶Y$ is whether the classifier satisfies Fisher consistency. Specifically, a classifier f is deemed Fisher consistent if the minimizer of the associated expected risk, as dictated by a loss function L, exhibits the same sign as the Bayes classifier [35]. Thus, the loss function L is termed Fisher consistent if it adheres to this property.

In binary classification problems, the training set is represented by ${\left\{\left(x_i,y_i\right)\right\}}_{i=1}^l$, under the assumption that the samples are independent and the probability measure is $\rho$ on $X\times Y$. Then, the expected risk of classifier f: $X⟶Y$ is defined as:

$$R_{L,\rho }\left(f\right)={\int }_{X\times Y}L(1-yf\left(x\right))d\rho$$

(25)

Here, $L(.)$ denotes the loss function, and $P\left(\chi \right)=P(Y=1(-1)\mid X=\chi )$ represents the conditional probability of the positive (or negative) class when $X=\chi$. Under the condition that $\chi$ is given, the conditional distribution of $\rho$ is indicated by $\rho (y\mid \chi )$. To minimize the expected risk, we introduce the optimization variable q and define the function to minimize the expected risk through the following formula:

$$f_{L,\rho }\left(\chi \right)=arg\underset{q}{min}{\int }_{Y}L(1-yq)d\rho (y\mid \chi )\forall \chi \in X$$

(26)

where $q=f\left(x\right)$ is a function to be optimized to represent the predicted value under a specific condition (i.e., given $\chi$). In the binary classification problem, $\rho \left(y\right|\chi )$ is a binary distribution. Specifically, $Prob(y=1|x)$ and $Prob(y=-1|x)$ denote the probabilities for the positive and negative classes, respectively. The Bayes classifier is given by:

$$f_c\left(\chi \right)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}Prob(y=1|\chi )\ge Prob(y=-1\left|\chi \right)\hfill \\ -1,\hfill & \mathrm{if}Prob(y=1|\chi )<Prob(y=-1\left|\chi \right)\hfill \end{array}\right.$$

(27)

Next, we will examine whether the SF-loss satisfies Fisher consistency.

Property 6. 

Function $f_{L_r,\rho }\left(\chi \right)$, which minimizes SF-loss expected risk among all measurable functions, is equivalent to the Bayes classifier: $f_{L_r,\rho }\left(\chi \right)=f_c\left(\chi \right)$. This means that the SF-loss satisfies Fisher consistency.

Proof. 

For binary classification, we have:

$${\int }_Y{L}_r\left(1-y\left(x\right)q\right)\mathrm{d}\rho \left(y|\chi \right)=L_r\left(1-q\right)Prob\left(y=1|\chi \right)+L_r\left(1+q\right)Prob\left(y=-1|\chi \right)$$

(28)

when $1+q$ and $1-q$ are applied to Formula (16), the following two equations are obtained:

$$L_r\left(1-q\right)={\displaystyle \frac{{\left(1-q\right)}^2}{r{\left(1-q\right)}^2+1}}$$

(29)

and

$$L_r\left(1+q\right)={\displaystyle \frac{{\left(1+q\right)}^2}{r{\left(1+q\right)}^2+1}}$$

(30)

By substituting (29) and (30) into (28), we can obtain:

$${\int }_Y{L}_r\left(1-y\left(x\right)q\right)\mathrm{d}\rho \left(y|\chi \right)={\displaystyle \frac{{\left(1-q\right)}^2}{r{\left(1-q\right)}^2+1}}Prob\left(y=1|\chi \right)+{\displaystyle \frac{{\left(1+q\right)}^2}{r{\left(1+q\right)}^2+1}}Prob\left(y=-1|\chi \right)$$

(31)

Therefore, if $Prob(y=1|\chi )\ge Prob(y=-1\left|\chi \right)$, expected risk receives the minimum value when $q=1$; if $Prob(y=1|\chi )<Prob(y=-1\left|\chi \right)$, expected risk receives the minimum value when $q=-1$. Hence, whichever minimizes the expected risk measured by the SF-loss satisfies $f_{L_r,\rho }\left(\chi \right)=f_c\left(\chi \right)$. This analysis confirms that the minimizer of the associated expected risk, determined by the loss function, has the same sign as the Bayes classifier, thereby proving that this property is satisfied. □

To more clearly compare the performance of our loss function with other advanced loss functions (Hinge loss [20,33], capped $L_1$-norm loss [25,33], and Welsch loss [30]), we present them in Figure 4. We can observe that the hinge loss is unbounded and tends to exaggerate the impact of outliers. Other losses are bounded, so we set specific parameters to ensure that the upper bounds of these losses are consistent. In this case, our proposed loss function exhibits a smoother characteristic and its growth trend is relatively gradual compared to the capped $L_1$-norm loss. Compared to the Welsch loss function, our proposed loss function increases rapidly near the origin (corresponding to normal data) and grows relatively slowly further away from it (representing noisy or outlier data points). This characteristic indicates that our loss function places greater emphasis on increasing the loss for normal points, encouraging the model to make more precise predictions for regular data. Meanwhile, due to the slower growth of loss for outliers, the model becomes less sensitive to them, thus reducing their impact and enhancing the overall robustness of the model.

3.2. Squared Fractional Loss Based Robust Supervised Twin Extreme Learning Machine

In order to better elaborate on the idea of our model improvement, we rewrite problems (5) and (6) of FTELM as follows:

$$\begin{array}{c}\hfill \underset{{\beta }_1}{min}{\displaystyle \frac{1}{2}}{∥H_1{\beta }_1∥}_2^2+C_1\sum _{j=1}^{m_2}{L}_H\left(1+h\left(x_j\right){\beta }_1\right)+{\displaystyle \frac{C_2}{2}}{\beta }_1^T{\mathit{H}}_1^T{\mathit{N}}_1{\mathit{H}}_1{\beta }_1\end{array}$$

(32)

$$\begin{array}{c}\hfill \underset{{\beta }_2}{min}{\displaystyle \frac{1}{2}}{∥H_2{\beta }_2∥}_2^2+C_3\sum _{i=1}^{m_1}{L}_H\left(1-h\left(x_i\right){\beta }_2\right)+{\displaystyle \frac{C_4}{2}}{\beta }_2^T{\mathit{H}}_2^T{\mathit{N}}_2{H}_2{\beta }_2\end{array}$$

(33)

where hinge loss function $L_H\left(u\right)=max\left(0,u\right)$, for (32), $u\triangleq u_j=1+h\left(x_j\right){\beta }_1$ and for (33), $u\triangleq u_i=1-h\left(x_i\right){\beta }_2$. It is worth noting that FTELM uses the squared $L_2$-norm metric and hinge loss, which can exaggerate the influence of noise and outliers. To enhance the robustness of the model, we have substituted the squared $L_2$-norm metric and hinge loss with the capped $L_{2,p}$-norm metric and SF-loss. Therefore, we propose a robust supervised TELM learning framework called SF-loss-based robust supervised TELM (SF-RSTELM). The core problem of our designed model can be described as:

• SF-RSTELM1:

  $$\begin{array}{cc}\hfill & \underset{{\beta }_1}{min}{\displaystyle \frac{1}{2}}\sum _{i=1}^{m_1}min\left({∥h\left(x_i\right){\beta }_1∥}_2^p,{\epsilon }_1\right)+{\displaystyle \frac{C_1}{2}}\sum _{j=1}^{m_2}{L}_r\left(1+h\left(x_j\right){\beta }_1\right)+{\displaystyle \frac{C_2}{2}}{\beta }_1^T{H}_1^T{N}_1{H}_1{\beta }_1\hfill \end{array}$$

  (34)
• SF-RSTELM2:

  $$\begin{array}{cc}\hfill & \underset{{\beta }_2}{min}{\displaystyle \frac{1}{2}}\sum _{j=1}^{m_2}min\left({∥h\left(x_j\right){\beta }_2∥}_2^p,{\epsilon }_3\right)+{\displaystyle \frac{C_3}{2}}\sum _{i=1}^{m_1}{L}_r\left(1-h\left(x_i\right){\beta }_2\right)+{\displaystyle \frac{C_4}{2}}{\beta }_2^T{H}_2^T{N}_2{H}_2{\beta }_2\hfill \end{array}$$

  (35)

  where $C_1$, $C_2$, $C_3$, $C_4>0$ are regularization parameters, and ${\epsilon }_1$ and ${\epsilon }_3$ are the thresholding parameters.

For problem (34), the positive training point approaches the hyperplane $f_1$ as closely as possible by optimizing the first term. Minimizing the second term ensures the negative class samples are as distant as possible from the positive class hyperplane $f_1$. The last term is the Fisher regularization term that minimizes the intra-class divergence from the samples. For problem (35), a similar interpretation can be applied.

In order to solve the problem effectively, we first work on the first term of the objective function. We first use the concave duality theorem [37] to deal with the first term of the objective functions (34) and (35).

Theorem 1. 

Consider a continuous nonconvex function $g\left(\theta \right):R^n\to R$ and suppose $h\left(\theta \right):R^n\to \Omega \subset R^n$ is a map with range Ω. We assume the existence of a concave function $\overline{g}\left(u\right)$ defined on Ω such that $g\left(\theta \right)=\overline{g}\left(h\left(\theta \right)\right)$ is satisfied. Based on this condition, the nonconvex function $g\left(\theta \right)$ can be represented as:

$$g\left(\theta \right)=\underset{\upsilon \in R^n}{inf}\left[{\upsilon }^{T}h\left(\theta \right)-g^{*}\left(\upsilon \right)\right]$$

(36)

Following concave duality [38], $g^{*}\left(\upsilon \right)$ is the concave dual of $g\left(u\right)$ given as:

$$g^{*}\left(\upsilon \right)=\underset{u\in \Omega }{inf}\left[{\upsilon }^{T}u-\overline{g}\left(u\right)\right]$$

(37)

Furthermore, the minimum on the right-hand side of (36) is obtained at ${\upsilon }^{*}$:

$${\upsilon }^{*}={\displaystyle \frac{\partial \overline{g}\left(u\right)}{\partial u}}{|}_{u=h\left(\theta \right)}$$

(38)

According to Theorem 1, we define a concave function $\overline{g}\left(\theta \right):R\to R$ such that $\forall \theta >0$:

$$\overline{g}\left(\theta \right)=min\left({\theta }^{{\displaystyle \frac{p}{2}}},\epsilon \right)$$

(39)

Suppose $h\left(\mu \right)={\mu }^2$. The capped $L_{2,p}$-norm metric can be reformulated as:

$$min\left({∥h\left(x_i\right)\beta ∥}_2^p,\epsilon \right)=\overline{g}\left(h\left(\mu \right)\right)$$

(40)

where $\mu ={∥h\left(x\right)\beta ∥}_2$. Therefore, the capped $L_{2,p}$-norm metric can also be represented as:

$$\sum _{i=1}^{m_1}\overline{g}\left({∥h\left(x_i\right){\beta }_1∥}_2^2\right)$$

(41)

$$\sum _{j=1}^{m_2}\overline{g}\left({∥h\left(x_j\right){\beta }_2∥}_2^2\right)$$

(42)

Let ${\theta }_1=h\left({\mu }_1\right)={∥h\left(x_i\right){\beta }_1∥}_2^2$. According to Equation (36), (40) can also be obtained as:

$$\begin{array}{c}\hfill min\left({∥h\left(x_i\right){\beta }_1∥}_2^p,{\epsilon }_1\right)=\overline{g}\left({∥h\left(x_i\right){\beta }_1∥}_2^2\right)=\underset{f_{ii}>0}{inf}\left[f_{ii}h\left({\mu }_1\right)-g^{*}\left(f_{ii}\right)\right]=\underset{f_{ii}>0}{inf}\left[f_{ii}{\theta }_1-g^{*}\left(f_{ii}\right)\right]\end{array}$$

(43)

The $g^{*}\left(f_{ii}\right)$ is the concave dual of $\overline{g}\left({\theta }_1\right)$, represented as:

$$g^{*}\left(f_{ii}\right)=\underset{{\theta }_1}{inf}\left[f_{ii}{\theta }_1-\overline{g}\left({\theta }_1\right)\right]=\underset{{\theta }_1}{inf}\left\{\begin{array}{c}{f}_{ii}{\theta }_1-{\theta }_1^{{\displaystyle \frac{p}{2}}},{\theta }_1^{{\displaystyle \frac{p}{2}}}<{\epsilon }_1\hfill \\ f_{ii}{\theta }_1-{\epsilon }_1,{\theta }_1^{{\displaystyle \frac{p}{2}}}\ge {\epsilon }_1\hfill \end{array}\right.$$

(44)

By optimizing ${\theta }_1$ for (44), we can obtain:

$$g^{*}\left(f_{ii}\right)=\left\{\begin{array}{c}{f}_{ii}{\left({\displaystyle \frac{2}{p}}{f}_{ii}\right)}^{{\displaystyle \frac{2}{p-2}}}-{\left({\displaystyle \frac{2}{p}}{f}_{ii}\right)}^{{\displaystyle \frac{p}{p-2}}},{\theta }_1^{{\displaystyle \frac{p}{2}}}<{\epsilon }_1\hfill \\ f_{ii}{\epsilon }_1^{{\displaystyle \frac{2}{p}}}-{\epsilon }_1,{\theta }_1^{{\displaystyle \frac{p}{2}}}\ge {\epsilon }_1\hfill \end{array}\right.$$

(45)

Therefore, the capped $L_{2,p}$-norm metric can be further converted to:

$$\begin{array}{cc}\hfill & \underset{{\beta }_1}{min}\sum _{i=1}^{m_1}min\left({∥h\left(x_i\right){\beta }_1∥}_2^p,{\epsilon }_1\right)\hfill \\ \hfill ⟺& \underset{{\beta }_1}{min}\sum _{i=1}^{m_1}\overline{g}\left({∥h\left(x_i\right){\beta }_1∥}_2^2\right)\hfill \\ \hfill ⟺& \underset{{\beta }_1}{min}\sum _{i=1}^{m_1}\underset{f_{ii}\ge 0}{inf}{L}_i\left({\beta }_1,f_{ii},{\epsilon }_1\right)\hfill \\ \hfill ⟺& \underset{\left({\beta }_1,f_{ii}\ge 0\right)}{min}\sum _{i=1}^{m_1}{L}_i\left({\beta }_1,f_{ii},{\epsilon }_1\right)\hfill \end{array}$$

(46)

where

$$L_i\left({\beta }_1,f_{ii},{\epsilon }_1\right)=\left\{\begin{array}{c}{f}_{ii}{\theta }_1-f_{ii}{\left({\displaystyle \frac{2}{p}}{f}_{ii}\right)}^{{\displaystyle \frac{2}{p-2}}}+{\left({\displaystyle \frac{2}{p}}{f}_{ii}\right)}^{{\displaystyle \frac{p}{p-2}}},{\theta }_1^{{\displaystyle \frac{p}{2}}}<{\epsilon }_1\hfill \\ f_{ii}{\theta }_1-f_{ii}{\epsilon }_1^{{\displaystyle \frac{2}{p}}}+{\epsilon }_1,{\theta }_1^{{\displaystyle \frac{p}{2}}}\ge {\epsilon }_1\hfill \end{array}\right.$$

(47)

Similarly, let ${\theta }_2=h\left({\mu }_2\right)={∥h\left(x_j\right){\beta }_2∥}_2^2$. $g^{*}\left(t_{jj}\right)$ is represented as a concave dual function of $\overline{g}\left({\theta }_2\right)$, so the capped $L_{2,p}$-norm metric can be written as:

$$\begin{array}{cc}\hfill & \underset{{\beta }_2}{min}\sum _{j=1}^{m_2}min\left({∥h\left(x_j\right){\beta }_1∥}_2^p,{\epsilon }_3\right)\hfill \\ \hfill ⟺& \underset{{\beta }_2}{min}\sum _{j=1}^{m_2}\overline{g}\left({∥h\left(x_j\right){\beta }_2∥}_2^2\right)\hfill \\ \hfill ⟺& \underset{{\beta }_2}{min}\sum _{j=1}^{m_2}\underset{t_{jj}\ge 0}{inf}{L}_j\left({\beta }_2,t_{jj},{\epsilon }_3\right)\hfill \\ \hfill ⟺& \underset{\left({\beta }_2,t_{jj}\ge 0\right)}{min}\sum _{j=1}^{m_2}{L}_j\left({\beta }_2,t_{jj},{\epsilon }_3\right)\hfill \end{array}$$

(48)

where

$$L_j\left({\beta }_2,t_{jj},{\epsilon }_3\right)=\left\{\begin{array}{c}{t}_{jj}{\theta }_2-t_{jj}{\left({\displaystyle \frac{2}{p}}{t}_{jj}\right)}^{{\displaystyle \frac{2}{p-2}}}+{\left({\displaystyle \frac{2}{p}}{t}_{jj}\right)}^{{\displaystyle \frac{p}{p-2}}},{\theta }_2^{{\displaystyle \frac{p}{2}}}<{\epsilon }_3\hfill \\ t_{jj}{\theta }_2-t_{jj}{\epsilon }_3^{{\displaystyle \frac{2}{p}}}+{\epsilon }_3,{\theta }_2^{{\displaystyle \frac{p}{2}}}\ge {\epsilon }_3\hfill \end{array}\right.$$

(49)

$${\displaystyle \frac{\partial \overline{g}\left(\theta \right)}{\partial \theta }}=\left\{\begin{array}{c}{\displaystyle \frac{p}{2}}{\theta }^{{\displaystyle \frac{p}{2}}-1},0<\theta <{\epsilon }^{{\displaystyle \frac{2}{p}}}\hfill \\ 0,\theta >{\epsilon }^{{\displaystyle \frac{2}{p}}}\hfill \end{array}\right.$$

(50)

Let ${\theta }_1=h\left({\mu }_1\right)={∥h\left(x_i\right){\beta }_1∥}_2^2$, and we can obtain:

$$f_{ii}={\displaystyle \frac{\partial \overline{g}\left({\theta }_1\right)}{\partial {\theta }_1}}{|}_{{\theta }_1={∥h\left(x_i\right){\beta }_1∥}_2^2}=\left\{\begin{array}{c}{\displaystyle \frac{p}{2}}{∥h\left(x_i\right){\beta }_1∥}_2^{p-2},0<{∥h\left(x_i\right){\beta }_1∥}_2^p<{\epsilon }_1\hfill \\ {\sigma }_1,else\hfill \end{array}\right.$$

(51)

Similarly, let ${\theta }_2=h\left({\mu }_2\right)={∥h\left(x_j\right){\beta }_2∥}_2^2$, and we can obtain:

$$t_{jj}={\displaystyle \frac{\partial \overline{g}\left({\theta }_2\right)}{\partial {\theta }_2}}{|}_{{\theta }_2={∥h\left(x_j\right){\beta }_2∥}_2^2}=\left\{\begin{array}{c}{\displaystyle \frac{p}{2}}{∥h\left(x_j\right){\beta }_2∥}_2^{p-2},0<{∥h\left(x_j\right){\beta }_2∥}_2^p<{\epsilon }_3\hfill \\ {\sigma }_2,else\hfill \end{array}\right.$$

(52)

When the variables $f_{ii}$ and $t_{jj}$ are fixed to solve the classifier-related parameters ${\beta }_1$ and ${\beta }_2$, the optimization problems (34) and (35) can be written as:

$$\begin{array}{c}\hfill \underset{{\beta }_1}{min}{\displaystyle \frac{1}{2}}\sum _{i=1}^{m_1}{\displaystyle \frac{2}{p}}{f}_{ii}{∥h\left(x_i\right){\beta }_1∥}_2^2+{\displaystyle \frac{C_1}{2}}\sum _{j=1}^{m_2}{L}_r\left(1+h\left(x_j\right){\beta }_1\right)+{\displaystyle \frac{C_2}{2}}{\beta }_1^T{H}_1^T{N}_1{H}_1{\beta }_1\end{array}$$

(53)

$$\begin{array}{c}\hfill \underset{{\beta }_2}{min}{\displaystyle \frac{1}{2}}\sum _{j=1}^{m_2}{\displaystyle \frac{2}{p}}{t}_{jj}{∥h\left(x_j\right){\beta }_2∥}_2^2+{\displaystyle \frac{C_3}{2}}\sum _{i=1}^{m_1}{L}_r\left(1-h\left(x_i\right){\beta }_2\right)+{\displaystyle \frac{C_4}{2}}{\beta }_2^T{H}_2^T{N}_2{H}_2{\beta }_2\end{array}$$

(54)

Let $F=diag\left({\displaystyle \frac{2}{p}}{f}_{11},{\displaystyle \frac{2}{p}}{f}_{22},\dots ,{\displaystyle \frac{2}{p}}{f}_{m_1{m}_1}\right)$ and $T=diag\left({\displaystyle \frac{2}{p}}{t}_{11},{\displaystyle \frac{2}{p}}{t}_{22},\dots ,{\displaystyle \frac{2}{p}}{t}_{m_2{m}_2}\right)$, so that (53) and (54) are equivalent to (55) and (56), respectively:

$$\begin{array}{c}\hfill \underset{{\beta }_1}{min}{\displaystyle \frac{1}{2}}{\left(H_1{\beta }_1\right)}^{T}F\left(H_1{\beta }_1\right)+{\displaystyle \frac{C_1}{2}}\sum _{j=1}^{m_2}{L}_r\left(1+h\left(x_j\right){\beta }_1\right)+{\displaystyle \frac{C_2}{2}}{\beta }_1^T{H}_1^T{N}_1{H}_1{\beta }_1\end{array}$$

(55)

$$\begin{array}{c}\hfill \underset{{\beta }_2}{min}{\displaystyle \frac{1}{2}}{\left(H_2{\beta }_2\right)}^{T}T\left(H_2{\beta }_2\right)+{\displaystyle \frac{C_3}{2}}\sum _{i=1}^{m_1}{L}_r\left(1-h\left(x_i\right){\beta }_2\right)+{\displaystyle \frac{C_4}{2}}{\beta }_2^T{H}_2^T{N}_2{H}_2{\beta }_2\end{array}$$

(56)

The problems (55) and (56) can be rewritten as the following (57) and (58):

$$\begin{array}{c}\hfill \underset{{\beta }_1}{min}{\displaystyle \frac{1}{2}}{\beta }_1^T{H}_1^T\left(F+C_2{N}_1\right)H_1{\beta }_1+{\displaystyle \frac{C_1}{2}}\sum _{j=1}^{m_2}{L}_r\left(1+h\left(x_j\right){\beta }_1\right)\end{array}$$

(57)

$$\begin{array}{c}\hfill \underset{{\beta }_2}{min}{\displaystyle \frac{1}{2}}{\beta }_2^T{H}_2^T\left(T+C_4{N}_2\right)H_2{\beta }_2+{\displaystyle \frac{C_3}{2}}\sum _{i=1}^{m_1}{L}_r\left(1-h\left(x_i\right){\beta }_2\right)\end{array}$$

(58)

It can be seen that (57) and (58) are non-convex optimization problems because of the non-convexity of the loss function. In this research, we utilize the CCCP technique to handle the non-convexity. Note that the loss function can be expressed as the subtraction of two convex functions, or equivalently, as the sum of a convex function and a concave function. That is, $L_r\left(u\right)=L_{r_1}\left(u\right)+L_{r_2}\left(u\right)$. The specific expressions for the convex function $L_{r_1}\left(u\right)$ and the concave function $L_{r_2}\left(u\right)$ are:

$$\begin{array}{c}\hfill L_{r_1}\left(u\right)=r{u}^2\end{array}$$

(59)

$$\begin{array}{c}\hfill L_{r_2}\left(u\right)=-r{u}^2+{\displaystyle \frac{u^2}{r{u}^2+1}}\end{array}$$

(60)

Figure 5 represents the graphs for $L_{r_1}\left(u\right)$ and $L_{r_2}\left(u\right)$.

Therefore, the above optimization problem can be expressed as:

$$\underset{{\beta }_1}{min}{\displaystyle \frac{1}{2}}{\beta }_1^T{H}_1^T\left(F+C_2{N}_1\right)H_1{\beta }_1+{\displaystyle \frac{C_1}{2}}\sum _{j=1}^{m_2}\left(L_{r_1}\left(1+h\left(x_j\right){\beta }_1\right)+L_{r_2}\left(1+h\left(x_j\right){\beta }_1\right)\right)$$

(61)

$$\begin{array}{c}\hfill \underset{{\beta }_2}{min}{\displaystyle \frac{1}{2}}{\beta }_2^T{H}_2^T\left(E+C_4{N}_2\right)H_2{\beta }_2+{\displaystyle \frac{C_3}{2}}\sum _{i=1}^{m_1}\left(L_{r_1}\left(1-h\left(x_i\right){\beta }_2\right)+L_{r_2}\left(1-h\left(x_i\right){\beta }_2\right)\right)\end{array}$$

(62)

Since (61) and (62) are similar, we will only show the solution process for (61), which can also be expressed as:

$$\underset{{\beta }_1}{min}\underset{L_{vex}\left({\beta }_1\right)}{\underbrace{{\displaystyle \frac{1}{2}}{\beta }_1^T{H}_1^T\left(F+C_2{N}_1\right)H_1{\beta }_1+{\displaystyle \frac{C_1}{2}}{\sum }_{j=1}^{m_2}{L}_{r_1}\left(1+h\left(x_j\right){\beta }_1\right)}}+\underset{L_{cav}\left({\beta }_1\right)}{\underbrace{{\displaystyle \frac{C_1}{2}}{\sum }_{j=1}^{m_2}{L}_{r_2}\left(1+h\left(x_j\right){\beta }_1\right)}}$$

(63)

The solution to the above optimization issue can be obtained by addressing the following equation:

$$\underset{{\beta }_1}{min}L\left({\beta }_1\right)=L_{vex}\left({\beta }_1\right)+L_{cav}\left({\beta }_1\right)$$

(64)

The value of ${\beta }_1$ in $k+1$ iterations is as follows:

$${\beta }_1^{\left(k+1\right)}=arg\underset{{\beta }_1}{min}\left\{L_{vex}\left({\beta }_1\right)+\nabla L_{cav}\left({\beta }_1^{\left(k\right)}\right){\beta }_1\right\}$$

(65)

Let ${\delta }_1^{\left(k\right)}=\nabla L_{cav}\left({\beta }_1^{\left(k\right)}\right)$, where ${\delta }_1^{\left(k\right)}={\left[{\delta }_{1_1}^k,{\delta }_{1_2}^k,\dots ,{\delta }_{1j}^k,\dots ,{\delta }_{1_{m_2}}^k\right]}^T\in R^{m_2}$.

$${\delta }_{1j}^k={\displaystyle \frac{\partial L_{cav}}{\partial u_j}}=-2r{u}_j+{\displaystyle \frac{2u_j}{{\left(r{u}_j^2+1\right)}^2}}$$

(66)

where $u_j=1+h\left(x_j\right){\beta }_1$. Therefore, we can get:

$$\nabla L_{cav}\left({\beta }_1^{\left(k\right)}\right){\beta }_1={\delta }_1^{\left(k\right)T}{H}_2{\beta }_1$$

(67)

Solving (65) is equivalent to solving the following subproblem:

$$\begin{array}{cc}\hfill & \underset{{\beta }_1}{min}{\displaystyle \frac{1}{2}}{\beta }_1^T{H}_1^T\left(F+C_2{N}_1\right)H_1{\beta }_1+{\displaystyle \frac{C_1}{2}}r{\xi }_1^T{\xi }_1+{\delta }_1^{\left(k\right)T}{H}_2{\beta }_1\hfill \\ \hfill & s.t.-H_2{\beta }_1+{\xi }_1=e_2\hfill \end{array}$$

(68)

Here, we introduce Lagrange multipliers ${\lambda }_1$ for problem (68). Next, its Lagrange function is given by:

$$\begin{array}{cc}\hfill L({\beta }_1,{\xi }_1,{\lambda }_1,{\theta }_1)=& {\displaystyle \frac{1}{2}}{\beta }_1^T{H}_1^T\left(F+C_2{N}_1\right)H_1{\beta }_1+{\displaystyle \frac{C_1}{2}}r{\xi }_1^T{\xi }_1+{\delta }_1^{\left(k\right)T}{H}_2{\beta }_1\hfill \\ \hfill & -{\lambda }_1^T\left(-H_2{\beta }_1+{\xi }_1-e_2\right)\hfill \end{array}$$

(69)

Following the Karush–Kuhn–Tucker (KKT) conditions, we derive the following constraints:

$$\begin{array}{c}\hfill KKT:\left\{\begin{array}{cc}& {\displaystyle \frac{\partial L}{\partial {\beta }_1}}=H_1^T\left(F+C_2{N}_1\right)H_1{\beta }_1+H_2^T{\delta }_1^{\left(k\right)}+H_2^T{\lambda }_1=0\hfill \\ & {\displaystyle \frac{\partial L}{\partial {\xi }_1}}=C_{1}r{\xi }_1-{\lambda }_1=0\hfill \\ & {\lambda }_1^T\left(-H_2{\beta }_1+{\xi }_1-e_2\right)=0\hfill \end{array}\right.\end{array}$$

(70)

From the KKT condition, we can obtain:

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}& {\beta }_1=-{\left(H_1^T\left(F+C_2{N}_1\right)H_1\right)}^{-1}{H}_2^T\left({\delta }_1^{\left(k\right)}+{\lambda }_1\right).\hfill \\ & C_{1}r{\xi }_1={\lambda }_1.\hfill \end{array}\right.\end{array}$$

(71)

Bring (71) into (68), and we can obtain the dual problem of the original problem:

$$\begin{array}{cc}\hfill & \underset{{\lambda }_1}{min}{\displaystyle \frac{1}{2}}{\left({\delta }_1^{\left(k\right)}+{\lambda }_1\right)}^T{H}_2{\left(H_1^T\left(F+C_2{N}_1\right)H_1\right)}^{-1}{H}_2^T\left({\delta }_1^{\left(k\right)}+{\lambda }_1\right)-e_2^T{\lambda }_1\hfill \end{array}$$

(72)

Using a similar approach, we can obtain the dual problem of Equation (58):

$$\begin{array}{cc}\hfill & \underset{{\lambda }_2}{min}{\displaystyle \frac{1}{2}}{\left({\lambda }_2-{\delta }_2^{\left(k\right)}\right)}^T{H}_1{\left(H_2^T\left(T+C_1{N}_2\right)H_2\right)}^{-1}{H}_1^T\left({\lambda }_2-{\delta }_2^{\left(k\right)}\right)-e_1^T{\lambda }_2\hfill \end{array}$$

(73)

After solving (72) and (73) to obtain the optimal solutions ${\lambda }_1$ and ${\lambda }_2$, we can obtain:

$$\begin{array}{c}\hfill {\beta }_1=-{\left(H_1^T\left(F+C_2{N}_1\right)H_1\right)}^{-1}{H}_2^T\left({\delta }_1^{\left(k\right)}+{\lambda }_1\right)\end{array}$$

(74)

$$\begin{array}{c}\hfill {\beta }_2={\left(H_2^T\left(T+C_4{N}_2\right)H_2\right)}^{-1}{H}_1^T\left({\lambda }_2-{\delta }_2^{\left(k\right)}\right)\end{array}$$

(75)

Therefore, the decision function of SF-RSTELM is given by:

$$\begin{array}{c}\hfill f\left(x\right)=arg\underset{k=1,2}{min}{d}_k\left(x\right)=arg\underset{k=1,2}{min}\left|{\beta }_k^{T}h\left(x\right)\right|.\end{array}$$

(76)

Based on the previous discussion, we provide a detailed description of the implementation steps of the proposed method in Algorithm 1.

Algorithm 1 The procedure of SF-RSTELM

Input: The training set $\mathcal{T}=\left\{(x_i,y_i)\right|1\le i\le m\}$, where $x_i\in {\mathbb{R}}^d$ and $y_i=\{1,-1\}$;

Parameters $C_1$, $C_2$, $C_3$, $C_4>0$ and ${\epsilon }_1$, ${\epsilon }_2$, ${\epsilon }_3$, ${\epsilon }_4>0$, $\epsilon >0$. ${\sigma }_1$,${\sigma }_2$.

Activation function $G\left(x\right)$; The number of hidden nodes L.

Maximum number of iterations $k_{max}$.

Output:${\beta }_1^{*}$, ${\beta }_2^{*}$.

Step:

1: Initialize $F^{\left(0\right)}\in {\mathbb{R}}^{m_1\times m_1}$ and $T^{\left(0\right)}\in {\mathbb{R}}^{m_2\times m_2}$; ${\delta }_1^{\left(0\right)}$ and ${\delta }_2^{\left(0\right)}$

2: Compute the graph matrix $N_1$, $N_2$.

3: Set $t=0$.

4: While

Calculate ${\lambda }_1^{\left(k\right)}$ and ${\lambda }_2^{\left(k\right)}$ for the dual problem (72) and (73), respectively.

Then get the solution ${\beta }_1^{\left(k\right)}$, ${\beta }_2^{\left(k\right)}$ by

${\beta }_1^{\left(k\right)}=-{\left(H_1^T\left(F^{\left(k\right)}+C_2{N}_1\right)H_1\right)}^{-1}{H}_2^T\left({\delta }_1^{\left(k\right)}+{\lambda }_1\right)$

${\beta }_2^{\left(k\right)}={\left(H_2^T\left(T^{\left(k\right)}+C_4{N}_2\right)H_2\right)}^{-1}{H}_1^T\left({\lambda }_2-{\delta }_2^{\left(k\right)}\right)$

Update the matrices $F^{(k+1)}$, $T^{(k+1)}$, ${\delta }_1^{(k+1)}$ and ${\delta }_2^{(k+1)}$ by (51), (52) and (66).

if $k>k_{max}$ or $\left|{\beta }_i^{\left(k\right)}-{\beta }_i^{(k-1)}\right|\le \epsilon$ $\left(i=1,2\right)$

break

else

$k=k+1$

end while

5: Construct the following decision functions:

$f\left(x\right)=arg{min}_{k=1,2}{d}_k\left(x\right)=arg{min}_{k=1,2}\left|{\beta }_k^{T}h\left(x\right)\right|.$

3.3. Convergence Analysis

Theorem 2. 

Utilizing the CCCP technique to address problem (63), the resulting sequence $\left\{{\beta }_1^{\left(k\right)}\right\}$ converges.

Proof of Theorem 2. 

At the iteration point of step $k+1$, the following inequality holds:

$$L_{vex}\left({\beta }_1^{\left(k\right)}\right)+\nabla L_{cav}{\left({\beta }_1^{\left(k\right)}\right)}^T{\beta }_1^{\left(k\right)}\ge L_{vex}\left({\beta }_1^{(k+1)}\right)+\nabla L_{cav}{\left({\beta }_1^{\left(k\right)}\right)}^T{\beta }_1^{(k+1)}$$

(77)

It can be written as

$$L_{vex}\left({\beta }_1^{\left(k\right)}\right)-L_{vex}\left({\beta }_1^{(k+1)}\right)\ge \nabla L_{cav}{\left({\beta }_1^{\left(k\right)}\right)}^T\left({\beta }_1^{(k+1)}-{\beta }_1^{\left(k\right)}\right)$$

(78)

Due to the concavity of $L_{cav}\left(.\right)$, we have

$$\nabla L_{cav}{\left({\beta }_1^{\left(k\right)}\right)}^T\left({\beta }_1^{(k+1)}-{\beta }_1^{\left(k\right)}\right)\ge L_{cav}\left({\beta }_1^{(k+1)}\right)-L_{cav}\left({\beta }_1^{\left(k\right)}\right)$$

(79)

By combining the above inequalities, we have

$$L_{vex}\left({\beta }_1^{\left(k\right)}\right)+L_{cav}\left({\beta }_1^{\left(k\right)}\right)\ge L_{vex}\left({\beta }_1^{(k+1)}\right)+L_{cav}\left({\beta }_1^{(k+1)}\right)$$

(80)

Accordingly, the objective value of problem (63) decreases monotonically with each iteration and remains non-negative, thereby proving the convergence of the sequence. □

Theorem 3. 

Algorithm 1 converges to a local optimum to the problems in (34) and (35).

Proof of Theorem 3. 

Taking problem (34) as an example, the analysis for problem (35) follows a similar approach.

First and foremost, let us recall the formulation of our framework, namely Equation (34).

$$\begin{array}{c}\hfill \underset{{\beta }_1,{\xi }_1}{min}{\displaystyle \frac{1}{2}}\sum _{i=1}^{m_1}min\left({∥h\left(x_i\right){\beta }_1∥}_2^p,{\epsilon }_1\right)+{\displaystyle \frac{C_1}{2}}\sum _{j=1}^{m_2}{L}_r\left(1+h\left(x_j\right){\beta }_1\right)+{\displaystyle \frac{C_2}{2}}{\beta }_1^T{H}_1^T{N}_1{H}_1{\beta }_1\end{array}$$

(81)

For convenience, let $J_1={\displaystyle \frac{C_1}{2}}{\sum }_{j=1}^{m_2}{L}_r\left(1+h\left(x_j\right){\beta }_1\right)$ and $J_2={\displaystyle \frac{C_2}{2}}{\beta }_1^T{H}_1^T{N}_1{H}_1{\beta }_1$. When ${∥h\left(x_i\right){\beta }_1∥}_2^p\le {\epsilon }_1$, we represent the Lagrange function of (81) as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_1\left({\beta }_1\right)={\displaystyle \frac{1}{2}}\sum _{i=1}^{m_1}{∥h\left(x_i\right){\beta }_1∥}_2^p+J_1+J_2\end{array}$$

(82)

Then, we differentiate ${\mathcal{L}}_1\left({\beta }_1\right)$ with respect to ${\beta }_1$:

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial {\mathcal{L}}_1\left({\beta }_1\right)}{\partial {\beta }_1}}=\sum _{i=1}^{m_1}{\displaystyle \frac{1}{2}}p{\left({∥h\left(x_i\right){\beta }_1∥}_2\right)}^{p-1}{\displaystyle \frac{h\left(x_i\right){\beta }_1{h}^{\prime }\left(x_i\right)}{{∥h\left(x_i\right){\beta }_1∥}_2}}+{\displaystyle \frac{\partial J_1}{\partial {\beta }_1}}+{\displaystyle \frac{\partial J_2}{\partial {\beta }_1}}=0\end{array}$$

(83)

According to (51), we bring $f_{ii}$ into Formula (83):

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial {\mathcal{L}}_1\left({\beta }_1\right)}{\partial {\beta }_1}}=H_1^{T}F{H}_1{\beta }_1+{\displaystyle \frac{\partial J_1}{\partial {\beta }_1}}+{\displaystyle \frac{\partial J_2}{\partial {\beta }_1}}=0\end{array}$$

(84)

Similarly, we obtain the Lagrangian function of problem (55):

$$\begin{array}{c}\hfill {\mathcal{L}}_2\left({\beta }_1\right)={\displaystyle \frac{1}{2}}{\left(H_1{\beta }_1\right)}^{T}F\left(H_1{\beta }_1\right)+J_1+J_2\end{array}$$

(85)

Taking the derivative of ${\mathcal{L}}_2\left({\beta }_1\right)$ with respect to ${\beta }_1$:

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial {\mathcal{L}}_2\left({\beta }_1\right)}{\partial {\beta }_1}}=H_1^{T}F{H}_1{\beta }_1+{\displaystyle \frac{\partial J_1}{\partial {\beta }_1}}+{\displaystyle \frac{\partial J_2}{\partial {\beta }_1}}=0\end{array}$$

(86)

It is noted that Formula (84) is equal to Formula (86) when determining the optimal solution ${\beta }_1$. Furthermore, the optimal solution ${\beta }_1$ meets the KKT condition of model (34). By solving problem (55), we can determine the optimal solution for problem (34). Thus, Algorithm 1 is capable of converging to a local optimum, making it feasible to obtain the local minimum of problem (34). □

4. Numerical Experiments

Within this part, we performed a comparison of SF-RSTELM with several algorithms, such as TELM [20], FTELM [33], $C{L}_1$-FTELM [33], FRTELM [25], and CWTELM [30]. To assess the performance of the proposed algorithm, experiments are carried out on four distinct types of databases: artificial datasets, UCI datasets, image datasets, and NDC large datasets. In addition, we demonstrate the convergence of the proposed algorithm through experimental analysis.

4.1. Experimental Setting

4.1.1. Operating Environment

All experiments were carried out using MATLAB (2021a) (MathWorks, Natick, United States) on a personal computer (PC) equipped with an Intel Core-i7 processor (2.5 GHz) and 16 GB random-access memory (RAM).

4.1.2. Benchmark Approaches

We have selected five advanced algorithms as benchmarks to compare with the SF-RSTELM proposed in this paper. These algorithms are:

• TELM [20]: Using the hinge loss function and squared $L_2$-norm metric.
• FTELM [33]: The Fisher regularization term is introduced into TELM, and relates to the statistical information of intra-class samples.
• $C{L}_1$-FTELM [33]: Capped $L_1$-norm loss and metric are introduced into FTELM.
• FRTELM [25]: Capped $L_1$-norm loss and metric are introduced into TELM.
• CWTELM [30]: Replace the hinge loss function and squared $L_2$-norm metric in TELM with Welsch loss function and capped $L_{2,p}$-norm metric.

4.1.3. Parameter Selection

In the training process of the model, the selection of parameters is crucial because it will affect the classification performance of the model. Here are some parameters that are respectively required for each model. Since both the comparison model and our model are twin models, which contain two optimization problems, the two optimization models are carried out separately. Therefore, with regard to the parameter selection problem, we will only explain the parameter selection method for one of the optimization problems, and the other model is similar.

• TELM: the count of hidden layer nodes L, the regularization parameter $C_1$.
• FTELM: the count of hidden layer nodes L, the regularization parameters $C_1$, $C_2$.
• $C{L}_1$-FTELM: the count of hidden layer nodes L, the regularization parameters $C_1$, $C_2$, the capped parameter ${\epsilon }_1$ in the metric, and the capped parameter ${\epsilon }_2$ in the loss function.
• FRTELM: the count of hidden layer nodes L, the regularization parameter $C_1$, the capped parameter ${\epsilon }_1$ in the metric, and the capped parameter ${\epsilon }_2$ in the loss function.
• CWTELM: the count of hidden layer nodes L, the regularization parameter $C_1$, the capped parameters ${\epsilon }_1$ and the parameter p of the metric, and the parameter $\sigma$ in the loss function.
• SF-RSTELM: the count of hidden layer nodes L, the regularization parameters $C_1$, $C_2$, the capped parameters ${\epsilon }_1$ and the parameter p of capped $L_{2,p}$-norm metric, and the parameter r in the loss function.

Due to the different number of parameters in different models, we will adopt different parameter selection strategies. For models with fewer than three parameters (TELM and FTELM), we will use grid search and ten-fold cross-validation to explore the parameter space and find the optimal parameter combination. However, when dealing with models with more than three parameters ($C{L}_1$-FTELM, FRTELM, CWTELM, and SF-RSTELM), directly applying grid search may lead to a significant increase in computational load. To overcome this challenge, we first fix some parameters to narrow down the search space based on initial experimental results and domain expertise. The fixed parameter strategy is as follows: the count of hidden nodes L is fixed, but L should be different for different datasets, and the range of L is $\left\{i\times {10}^j\mid i=1,2,\dots ,6,j=1,2\right\}$. For a fair contrast, we fix the parameters to ensure that the above algorithms with bounded loss functions have the same upper bound on their loss. Specifically, we set the capped $L_1$-norm loss parameter to 2 in $C{L}_1$-FTELM and FRTELM, the parameter $\sigma$ to 2 in CWTELM for Welsch loss, and the parameter r to 0.5 in our model’s SF-loss. Moreover, for the aforementioned models that utilize bounded metrics, we set all their upper bounds to be equal. That is, the capped parameter ${\epsilon }_1$ is set to 0.001. In addition to fixing the above parameters, we still choose to use ten-fold cross-validation and grid search to find the best values for the remaining parameters. Specifically, the regularization parameters $C_1$ and $C_2$ are chosen from $\{i\times {10}^j\mid i=1,2,\dots ,6,\mathrm{and}$ $j=-3,\dots ,2\}$, and the parameter p is chosen from $\left(0,2\right]$.

4.1.4. Evaluation Criteria

To assess the validity of the model, we employ accuracy ($ACC$) and the $F_1$-score as evaluation criteria. Specifically, these criteria are specified as follows:

$$ACC={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(87)

$$F_1={\displaystyle \frac{2TP}{2TP+FP+FN}}$$

(88)

where $TP$, $TN$, $FP$, and $FN$ refer to true positives, true negatives, false positives, and false negatives, respectively. Both $ACC$ and the $F_1$-score serve to evaluate the model’s generalization capability, and the higher they are, the better the performance. To guarantee the reliability of our experiments, we repeat all experimental procedures 10 times, and the final experimental result is the mean of the 10 repetitions.

4.2. Experiments on the Artificial Datasets

In this subsection, experiments are initially performed on the Two Moons, XOR, and Banana datasets. Both the Two Moons and Banana datasets comprise 400 samples, while the XOR dataset contains 100 samples. Figure 6 visually depicts the two-dimensional distribution graphs of these three artificial datasets. Class 1 is shown as a red ‘∘’, and class 2 is shown as a blue ‘⋄’.

To test the performance of the proposed method, Figure 7 shows the accuracy of five comparison algorithms as well as the proposed algorithm under conditions of no noise and a noise ratio of 20%. The noise is introduced by randomly selecting training samples and perturbing their features with Gaussian noise that follows a normal distribution $N\left(0,{\sigma }^2\right)$. Specifically, for the training data X, $X+\tilde{X}$ is used instead of X, where $\tilde{X}$ represents the noise matrix that adheres to a normal distribution with a mean of 0 and a variance of ${\sigma }^2$.

As can be seen from Figure 7a, the accuracy of SF-RSTELM is highest among the six methods when no noise is added to the dataset. Our model has the highest accuracy, which can be explained in three ways. First, it uses a bounded capped $L_{2,p}$-norm metric, whose p-value can be flexibly adjusted to suit different data. Second, we use a bounded loss that effectively controls the upper bound. In addition, we consider the intra-class divergence information of the data. The remaining algorithms are sorted in descending order of accuracy as $C{L}_1$-FTELM, CWTELM, FTELM, FRTELM, TELM. The accuracy of $C{L}_1$-FTELM is slightly lower than that of our model, probably because the capped $L_1$-norm metric is not as flexible as the capped $L_{2,p}$-norm metric we use. Although CWTELM has also improved the metric and loss, it still falls slightly below $C{L}_1$-FTELM due to its failure to consider the statistical properties of the data. FTELM is merely an extension of TELM, with the addition of a Fisher regularization term. FRTELM improved the metric and loss; however, the intra-class divergence information of the data is not taken into account. When noise is added to the dataset (Figure 7b), the performance of all algorithms is degraded, but SF-RSTELM’s accuracy is still the highest. This shows that our algorithm has the strongest noise immunity and robustness.

4.3. Experiments on the UCI Datasets

In this subsection, we evaluate the performance of our model on nine UCI datasets from http://archive.ics.uci.edu/ml/datasets.php (accessed on 15 February 2024) and contrast it with five other state-of-the-art algorithms.

Table 1. Characteristics of UCI datasets.

Datasets	Instances	Attributes	Datasets	Instances	Attributes
Australian	690	14	Vote	435	16
Ionosphere	351	34	WDBC	569	30
German	1000	24	wholesalesta	400	7
Pima	768	8	Sonar	208	60
QSAR	1055	41

provides a detailed overview of the characteristics of the UCI datasets employed. All features within these datasets are normalized to a scale of $\left[0,1\right]$ to ensure consistency.

Initially, we assess the steadiness of our SF-RSTELM without adding extra noises.

Table 2. Experimental results on UCI datasets. The best results are marked in bold.

Datasets	TELM [20]	FTELM [33]	C${\mathit{L}}_1$-FTELM [33]	CWTELM [30]	FRTELM [25]	SF-RSTELM
	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)
	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)
	Time(s)	Time(s)	Time(s)	Time(s)	Time(s)	Time(s)
Australian	85.66 ± 0.37	86.37 ± 0.29	86.49 ± 0.37	86.25 ± 0.82	86.03 ± 0.94	86.76 ± 0.56
	86.27 ± 0.33	85.70 ± 1.04	86.06 ± 0.76	86.14 ± 0.47	85.97 ± 0.39	86.55 ± 0.64
	0.596	0.651	3.739	5.939	4.638	6.856
Vote	94.26 ± 0.55	94.98 ± 0.69	95.74 ± 0.37	95.25 ± 0.57	94.83 ± 0.66	95.86 ± 0.37
	93.46 ± 0.83	95.89 ± 0.57	95.28 ± 0.17	96.50 ± 0.46	95.80 ± 0.56	96.87 ± 1.64
	0.233	0.794	1.009	1.843	3.036	3.604
Ionosphere	88.67 ± 1.36	90.25 ± 1.13	90.36 ± 1.64	90.34 ± 1.73	90.14 ± 0.85	90.67 ± 0.93
	84.87 ± 2.03	86.45 ± 1.95	88.33 ± 2.75	87.38 ± 1.78	86.26 ± 0.94	89.41 ± 1.12
	0.458	0.237	2.254	2.358	3.329	3.217
WDBC	96.13 ± 0.34	97.13 ± 0.17	97.05 ± 0.65	96.63 ± 0.38	96.50 ± 0.83	97.28 ± 0.98
	95.37 ± 0.51	96.07 ± 0.21	95.78 ± 0.34	95.98 ± 0.62	95.94 ± 0.87	97.03 ± 0.56
	0.719	0.689	4.359	3.159	4.656	5.289
German	74.52 ± 1.28	74.86 ± 2.96	76.42 ± 0.92	74.38 ± 2.74	74.63 ± 0.79	74.80 ± 0.68
	70.36 ± 0.87	71.28 ± 2.77	72.90 ± 0.53	73.21 ± 0.41	72.18 ± 0.47	72.73 ± 0.32
	0.878	0.654	8.744	7.934	7.218	8.306
wholesalesta	87.30 ± 0.72	88.01 ± 0.44	89.91 ± 0.51	89.43 ± 1.04	88.72 ± 1.34	90.00 ± 1.20
	76.76 ± 1.71	82.30 ± 0.49	82.26 ± 0.27	83.87 ± 1.86	81.23 ± 2.72	85.16 ± 2.26
	1.869	0.481	3.647	4.385	3.287	4.473
Pima	76.38 ± 1.72	76.48 ± 1.38	76.97 ± 0.45	78.13 ± 2.27	76.22 ± 1.11	76.79 ± 0.58
	74.52 ± 1.28	76.74 ± 1.08	77.96 ± 0.32	78.82 ± 0.45	76.90 ± 0.64	75.32 ± 0.68
	0.969	1.282	4.561	7.548	5.078	6.631
Sonar	67.76 ± 0.54	69.65 ± 2.21	69.83 ± 0.46	68.45 ± 3.01	68.37 ± 2.12	70.50 ± 3.67
	65.83 ± 0.66	72.10 ± 2.86	68.26 ± 0.15	65.35 ± 3.41	66.64 ± 1.84	72.82 ± 2.72
	0.611	1.055	1.376	1.537	1.718	1.476
QSAR	85.61 ± 0.74	86.43 ± 0.82	87.44 ± 2.43	87.16 ± 0.36	86.35 ± 1.93	86.81 ± 2.68
	77.64 ± 1.11	78.36 ± 1.24	82.23 ± 0.31	81.35 ± 0.74	81.36 ± 0.71	81.59 ± 0.23
	0.784	1.522	12.884	9.572	11.667	19.101

presents the results of the experiments, with the best results highlighted in bold. Here, Time(s) denotes the average execution time for each algorithm when optimized with the best parameters. In ACC ± S, ACC denotes the mean learning accuracy, while S represents the standard deviation. Table 2 shows that SF-RSTELM outperforms the other five methods in terms of accuracy and $F_1$ on most datasets, except for the German, Pima, and QSAR datasets. This is because SF-RSTELM not only incorporates the Fisher regularization term, which considers intra-class divergence, but also leverages the parameter adjustability of SF-loss and the flexibility of the capped $L_{2,p}$-norm metric.

Moreover, to illustrate the noise-resistant properties of SF-RSTELM, we introduce Gaussian noise into the training subset, affecting their features to generate noise. The experimental outcomes at noise levels of 15% and 25% are displayed in

Table 3. Experimental results on UCI datasets with 15% Gaussian noise. The best results are marked in bold.

Datasets	TELM [20]	FTELM [33]	C${\mathit{L}}_1$-FTELM [33]	CWTELM [30]	FRTELM [25]	SF-RSTELM
	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)
	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)
	Time(s)	Time(s)	Time(s)	Time(s)	Time(s)	Time(s)
Australian	82.10 ± 0.88	83.07 ± 0.90	85.44 ± 0.22	84.25 ± 0.34	84.29 ± 0.63	85.59± 0.75
	83.88 ± 0.84	84.60 ± 0.78	86.64 ± 0.25	84.14 ± 0.81	84.68 ± 0.61	86.17 ± 0.49
	0.481	0.727	3.624	5.619	4.256	6.499
Vote	91.17 ± 1.04	92.54 ± 1.16	93.65 ± 0.45	92.56 ± 0.23	92.43 ± 0.30	93.72 ± 0.91
	92.84 ± 0.87	93.88 ± 0.76	92.53 ± 0.26	91.62 ± 0.58	91.32 ± 0.27	93.26 ± 0.69
	0.320	0.586	1.146	2.052	2.940	3.296
Ionosphere	85.50 ± 1.80	85.88 ± 1.71	85.97 ± 0.89	85.34 ± 1.73	85.38 ± 1.33	87.94 ± 1.79
	82.96 ± 0.61	84.34 ± 0.43	83.08 ± 0.53	85.21 ± 1.78	83.39 ± 1.75	85.56 ± 0.57
	0.231	0.164	2.897	2.358	2.428	3.299
WDBC	94.89 ± 0.63	95.13 ± 0.41	94.36 ± 0.54	94.29 ± 0.87	93.37 ± 0.46	95.36 + 0.51
	93.15 ± 0.88	93.09 ± 0.57	92.66 ± 0.21	91.84 ± 0.34	90.24 ± 0.77	93.28 ± 0.82
	0.392	0.227	4.087	4.527	4.560	5.498
German	70.79 ± 0.73	71.59 ± 1.19	72.87 ± 0.58	71.24 ± 0.56	70.50 ± 0.66	72.30 ± 0.65
	68.35 ± 0.15	67.57 ± 0.96	71.32 ± 0.17	69.54 ± 0.24	69.88 ± 0.73	70.54 ± 0.18
	0.786	1.446	8.582	7.934	7.572	8.081
wholesalesta	84.19 ± 1.82	83.65 ± 1.32	85.72 ± 0.19	84.91 ± 0.87	84.67 ± 0.95	86.05 ± 0.74
	78.45 ± 2.50	82.25 ± 0.78	82.15 ± 0.32	81.13 ± 1.45	81.29 ± 0.79	82.34 ± 1.65
	0.632	0.253	3.362	4.073	3.395	4.455
Pima	69.07 ± 1.03	71.87 ± 1.09	73.09 ± 0.48	72.45 ± 0.71	70.75 ± 0.35	71.58 ± 1.03
	66.15 ± 0.57	69.45 ± 0.23	70.17 ± 0.23	70.05 ± 0.30	69.34 ± 0.45	69.81 ± 0.73
	0.497	1.815	5.178	8.237	6.583	7.871
Sonar	63.90 ± 0.82	65.25 ± 0.75	64.27 ± 0.65	64.13 ± 0.46	63.95 ± 0.52	65.50 ± 2.44
	64.19 ± 0.94	64.67 ± 2.34	64.73 ± 0.24	64.23 ± 0.67	64.01 ± 1.22	65.63 ± 1.25
	0.433	0.652	1.686	1.874	1.634	1.947
QSAR	78.08 ± 1.49	80.05 ± 1.42	82.35 ± 0.35	82.56 ± 0.76	81.21 ± 0.37	82.05 ± 0.84
	79.35 ± 1.37	77.25 ± 0.57	80.56 ± 0.28	81.74 ± 0.87	80.45 ± 0.17	79.69 ± 2.20
	0.846	1.731	13.932	8.528	10.462	17.196

and

Table 4. Experimental results on UCI datasets with 25% Gaussian noise. The best results are marked in bold.

Datasets	TELM [20]	FTELM [33]	C${\mathit{L}}_1$-FTELM [33]	CWTELM [30]	FRTELM [25]	SF-RSTELM
	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)
	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)
	Time(s)	Time(s)	Time(s)	Time(s)	Time(s)	Time(s)
Australian	79.97 ± 0.74	80.50 ± 1.25	81.82 ± 0.31	81.12 ± 0.54	80.34 ± 0.93	82.50 ± 0.84
	78.56 ± 1.41	79.13 ± 0.20	82.09 ± 0.65	81.23 ± 0.34	80.39 ± 0.79	83.09 ± 0.82
	0.369	0.549	2.937	4.457	3.210	5.239
Vote	90.40 ± 0.90	90.14 ± 0.65	92.52 ± 0.46	91.12 ± 0.76	90.88 ± 1.05	92.67 ± 1.09
	93.73 ± 0.82	91.83 ± 0.91	94.48 ± 0.81	91.72 ± 0.83	92.97 ± 0.75	92.87 ± 0.77
	0.299	0.312	1.391	2.131	3.724	2.984
Ionosphere	80.44 ± 0.92	83.06 ± 1.55	84.18 ± 0.67	84.13 ± 0.76	83.23 ± 0.65	84.71 ± 1.99
	79.24 ± 1.46	81.29 ± 0.38	81.26 ± 0.34	81.05 ± 0.23	81.25 ± 0.12	81.89 ± 0.86
	0.254	0.157	2.721	2.480	4.008	3.365
WDBC	91.91 + 0.34	91.93 + 1.39	92.26 + 0.48	92.03 + 0.56	91.82 + 0.42	93.04 + 0.64
	89.71 ± 1.54	89.76 ± 0.45	90.14 ± 0.86	89.47 ± 0.35	87.21 ± 0.62	90.34 ± 0.76
	0.285	0.486	4.228	5.162	3.014	5.271
German	70.66 ± 0.91	70.09 ± 0.64	71.12 ± 0.46	70.37 ± 1.08	69.46 ± 0.78	71.83 ± 0.43
	67.23 ± 0.83	68.72 ± 0.31	67.33 ± 0.28	67.53 ± 0.36	67.24 ± 0.41	68.98 ± 0.65
	0.734	1.379	9.411	8.475	8.306	9.916
wholesalesta	81.44 ± 1.13	80.05 ± 1.65	81.16 ± 0.23	80.52 ± 0.34	80.25 ± 0.31	83.33 ± 1.02
	78.23 ± 0.68	79.21 ± 0.84	77.34 ± 0.75	78.47 ± 0.75	78.02 ± 1.40	79.45 ± 0.93
	0.493	0.367	3.737	4.782	5.434	5.837
Pima	65.71 ± 0.22	66.78 ± 0.93	68.36 ± 0.57	67.23 ± 0.84	66.79 ± 0.14	65.86 ± 0.82
	67.30 ± 0.17	67.37 ± 0.56	68.65 ± 0.26	67.18 ± 0.72	65.37 ± 0.23	67.73 ± 0.59
	0.429	1.518	6.551	8.169	5.176	8.117
Sonar	60.75 ± 0.74	61.15 ± 0.26	62.23 ± 0.14	61.23 ± 0.52	61.15 ± 0.21	62.55 ± 1.26
	61.53 ± 0.31	61.90 ± 0.88	62.17 ± 0.56	61.45 ± 1.31	62.43 ± 1.61	63.23 ± 0.67
	0.538	0.399	1.372	1.549	1.752	1.382
QSAR	74.63 ± 1.27	76.45 ± 0.65	81.35 ± 0.35	81.68 ± 0.81	81.05 ± 0.54	79.13 ± 0.76
	72.36 ± 1.10	72.94 ± 0.46	80.82 ± 0.28	80.93 ± 0.77	80.54 ± 0.23	76.56 ± 1.84
	0.606	1.572	12.847	9.458	11.284	18.193

. It is clear that with rising noise levels, the learning accuracy of all algorithms declines. However, SF-RSTELM maintains higher accuracy and $F_1$ scores compared to the other five methods, except on a few datasets. This also demonstrates the strong noise resistance capability of our model.

To demonstrate the robustness of SF-RSTELM at varying noise levels (10%, 15%, 20%, and 25%), experiments are conducted on three datasets (Australian, Vote, and Ionosphere). Figure 8 illustrates the accuracy variation line charts of six algorithms on these three datasets across different noise levels. For the original dataset $\mathit{X}$, we change it with $\mathit{X}+\mu \overline{\mathit{X}}$, where $\overline{\mathit{X}}$ is the Gaussian variable. Here, $\mu ={\displaystyle \frac{{\rho \parallel X\parallel }_F}{\parallel \overline{X}{\parallel }_F}}$, and $\mu$ is a noise ratio. The value of $\mu \in \{0,0.1,0.15,0.2,0.25\}$. As is evident from Figure 8, the performance of all algorithms decreases as noise levels rise, but the accuracy of our model decreases the slowest. This further indicates that SF-RSTELM possesses superior classification accuracy compared to the other five methods. This superiority in noise handling is likely attributed to the combined effectiveness of the SF-loss function, the capped $L_{2,p}$-norm metrics, and the Fisher regularization term.

4.4. Experiments on the Image Datasets

In this part, we will perform experiments using high-dimensional image datasets to assess and compare the noise resistance and classification accuracy of our model SF-RSTELM with five other algorithms. We utilized the following three image datasets: COIL-20 http://www.cad.zju.edu.cn/home/dengcai/Data/FaceData.html (accessed on 15 February 2024), USPS http://www.cad.zju.edu.cn/home/dengcai/Data/FaceData.html (accessed on 15 February 2024), and MNIST http://www.cad.zju.edu.cn/home/dengcai/Data/FaceData.html (accessed on 15 February 2024). Since these image datasets are essentially a problem with multiple classes, we employed the “one-vs-rest” strategy to convert them to multiple binary classification problems.

Table 5. Characteristics of image datasets.

Datasets	COIL-20	USPS	MNIST
Instances	1440	9298	70,000
Attributes	1024	256	784
Image Resolution	$32\times 32$ pixels	$16\times 16$ pixels	$28\times 28$ pixels
Input Dimension	1024	256	784
Description	20 objects, each rotated on a turntable with images captured every 5 degrees, resulting in 72 images per object.	Contains a total of 9298 images of handwritten digits.	It has a training set of 60,000 examples, and a test set of 10,000 examples.

shows the features of the three image datasets. Figure 9 demonstrates examples from the three high-dimensional image datasets. Specifically, due to the large size of the MNIST dataset, we only select the first 2000 samples to participate in the experiment. These image datasets are utilized to evaluate the performance of our models in multi-class classification tasks.

The specific experimental results are presented in

Table 6. Experimental results on image datasets. The best results are marked in bold.

Datasets	TELM [20]	FTELM [33]	C${\mathit{L}}_1$-FTELM [33]	CWTELM [30]	FRTELM [25]	SF-RSTELM
	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)
	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)
	Time(s)	Time(s)	Time(s)	Time(s)	Time(s)	Time(s)
COIL20	88.06 ± 0.89	90.38 ± 0.52	90.63 ± 0.28	91.62 ± 0.39	90.25 ± 0.74	92.46 ± 0.25
	87.61 ± 0.60	88.15 ± 1.56	89.34 ± 0.75	89.45 ± 0.21	89.50 ± 0.67	90.82 ± 0.71
USPS	97.25 ± 0.28	97.14 ± 0.74	98.27 ± 0.65	98.49 ± 0.75	97.45 ± 0.96	98.73 ± 0.38
	95.57 ± 0.47	94.83 ± 0.36	95.51 ± 0.59	96.63 ± 0.38	95.58 ± 0.35	97.25 ± 0.17
MNIST	89.92 ± 0.35	90.23 ± 0.81	90.89 ± 0.38	89.92 ± 0.32	89.61 ± 0.23	91.26 ± 0.79
	88.24 ± 0.51	87.34 ± 0.75	89.17 ± 0.29	87.38 ± 0.41	87.23 ± 0.16	89.03 ± 0.85

. From the experiment results, it is evident that SF-RSTELM has the highest accuracy on all three datasets.

Table 7. Experimental results on image datasets with 15% Gaussian noise. The best results are marked in bold.

Datasets	TELM [20]	FTELM [33]	C${\mathit{L}}_1$-FTELM [33]	CWTELM [30]	FRTELM [25]	SF-RSTELM
	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)
	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)
	Time(s)	Time(s)	Time(s)	Time(s)	Time(s)	Time(s)
COIL-20	86.35 ± 0.56	87.87 ± 0.58	89.25 ± 0.75	88.78 ± 0.91	88.46 ± 0.64	89.57 ± 0.56
	85.51 ± 0.92	85.25 ± 0.29	87.85 ± 0.28	87.25 ± 0.29	87.67 ± 0.71	88.04 ± 0.63
USPS	94.54 ± 0.32	95.32 ± 0.56	95.81 ± 0.26	96.58 ± 0.82	95.34 ± 0.68	96.65 ± 0.76
	93.12 ± 0.79	93.54 ± 0.93	94.78 ± 0.57	95.45 ± 0.73	94.16 ± 0.45	95.47 ± 0.64
MNIST	87.48 ± 0.83	88.74 ± 0.46	88.93 ± 0.76	88.13 ± 0.25	87.94 ± 0.57	89.18 ± 0.42
	84.62 ± 0.21	86.37 ± 0.84	87.32 ± 0.82	86.84 ± 0.18	86.15 ± 0.74	87.56 ± 0.27

shows the results after adding 15% Gaussian noise. It can be seen that the ACC and $F_1$ value of all models are significantly reduced, but our model is still the highest, which also indicates that our model has good noise resistance capability, classification ability, and ability to process high-dimensional image datasets.

4.5. Experiments on the NDC Large Datasets

In this subsection, to evaluate the stability of our proposed model on large-scale datasets. We have conducted a comparative analysis of our algorithm with five other algorithms using the NDC datasets generated by David Musicant’s NDC data generator http://www.cs.wisc.edu/musicant/data/ndc (accessed on 15 February 2024). Detailed descriptions of the NDC datasets are presented in

Table 8. Characteristics of NDC datasets.

Datasets	Instances	Attributes
NDC-5k	5000	35
NDC-10k	10,000	35
NDC-15k	15,000	35

.

Table 9. Experimental results on NDC large datasets. The best results are marked in bold.

Datasets	TELM [20]	FTELM [33]	C${\mathit{L}}_1$-FTELM [33]	CWTELM [30]	FRTELM [25]	SF-RSTELM
	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)	ACC ± S(%)
	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)	${\mathit{F}}_{\mathbf{1}}$ ± S(%)
	Time(s)	Time(s)	Time(s)	Time(s)	Time(s)	Time(s)
NDC-5k	86.38 ± 0.26	87.63 ± 0.23	90.84 ± 0.81	90.79 ± 0.23	87.65 ± 0.78	90.95 ± 0.47
	85.89 ± 0.91	88.47 ± 0.39	91.78 ± 0.39	91.60 ± 0.88	87.71 ± 0.31	91.83 ± 0.39
NDC-10k	87.39 ± 0.73	89.72 ± 0.84	91.37 ± 0.45	92.74 ± 0.29	88.97 ± 0.65	92.91 ± 0.23
	86.82 ± 0.57	88.90 ± 0.61	90.72 ± 0.24	91.69 ± 0.82	87.83 ± 0.18	91.80 ± 0.35
NDC-15k	87.13 ± 0.36	88.21 ± 0.13	90.69 ± 0.71	90.32 ± 0.14	88.05 ± 0.53	89.95 ± 0.46
	86.87 ± 0.85	87.56 ± 0.38	90.94 ± 0.80	90.72 ± 0.52	86.48 ± 0.29	88.73 ± 0.71

summarizes the experimental results of the six algorithms on three large-scale NDC datasets.

As can be seen from Table 9, our model SF-RSTELM has the highest accuracy and $F_1$ value except for the NDC-15k dataset. Overall, SF-RSTELM is more stable than the other five algorithms. This is mainly due to the advantages of SF-RSTELM, which not only incorporates the statistical attributes of the data, but also uses the bounded and flexibly adjustable metric and loss function, which effectively controls the disturbance from noise and outliers. Therefore, from this set of experimental results, we can see that our model is also effective for large datasets.

4.6. Convergence Curve

We also perform experiments on four UCI datasets (Australian, QSAR, WDBC, and Vote) to confirm the convergence of the proposed Algorithm 1. Figure 10 demonstrates the convergence case of the objective function value with the increasing number of iterations. As can be seen from Figure 10, the objective function value converges relatively quickly to a stable fixed value.

This observation verifies the effectiveness of our algorithm in converging the objective function to a local optimum within a finite number of iterations, thereby demonstrating its convergence properties.

5. Conclusions and Future Works

In this paper, we first propose a new kind of SF-loss function that exhibits favorable characteristics including boundedness, smoothness, symmetry, noise insensitivity, and Fisher consistency. Then, SF-RSTELM is proposed by integrating the capped $L_{2,p}$-norm metric, SF-loss, and Fisher regularization term. SF-RSTELM not only integrates the Fisher regularization term, addressing the intra-class divergence of the data, but also exploits the parameter adjustability of SF-loss and the flexibility of capped $L_{2,p}$-norm metrics to reduce the influence of noise and outliers. Moreover, an efficient iterative algorithm is proposed to solve the model, and the convergence of the algorithm is proved. Experimental results on multiple datasets demonstrate the efficiency of the proposed model. Specifically, our model was able to achieve higher ACC and $F_1$ scores on most datasets, with improvements ranging from 0.28% to 4.5% compared to other state-of-the-art algorithms.

In the future, we will continue to study the improvement of the algorithm. Because the model constructed in this paper represents a non-convex optimization problem, we convert it into a series of convex problems to solve by the CCCP method, resulting in a long training time, so it is necessary to find a fast solution method in future research. Moreover, transforming this paper’s model from supervised to semi-supervised learning remains an important direction for future studies.