Pattern Classification Based on RBF Networks with Self-Constructing Clustering and Hybrid Learning

Abstract

Radial basis function (RBF) networks are widely adopted to solve problems in the field of pattern classification. However, in the construction phase of such networks, there are several issues encountered, such as the determination of the number of nodes in the hidden layer, the form and initialization of the basis functions, and the learning of the parameters involved in the networks. In this paper, we present a novel approach for constructing RBF networks for pattern classification problems. An iterative self-constructing clustering algorithm is used to produce a desired number of clusters from the training data. Accordingly, the number of nodes in the hidden layer is determined. Basis functions are then formed, and their centers and deviations are initialized to be the centers and deviations of the corresponding clusters. Then, the parameters of the network are refined with a hybrid learning strategy, involving hyperbolic tangent sigmoid functions, steepest descent backpropagation, and least squares method. As a result, optimized RBF networks are obtained. With this approach, the number of nodes in the hidden layer is determined and basis functions are derived automatically, and higher classification rates can be achieved. Furthermore, the approach is applicable to construct RBF networks for solving both single-label and multi-label pattern classification problems.

1. Introduction

Pattern classification is a process related to categorization, concerned with classifying patterns into one or more categories. It plays an important role in many fields, such as media, medicine, science, business, and security [1,2,3,4,5]. A classification problem can be converted to multiple binary classification problems in two ways, one-vs.-rest (OvR) and one-vs.-one (OvO), by creating several systems each of which is responsible for one category. However, OvR may suffer from unbalanced learning, since typically the set of negative training data is much larger than the set of positive training data. In contrast, OvO may suffer from ambiguities: two or more categories may receive the same number of votes and one category has to be picked arbitrarily. Therefore, researchers tend to seek other ways to solve classification problems.

In general, classification tasks can be divided into two kinds: single-label and multi-label. Single-label classification concerns classifying a pattern to only one category, but multi-label classification may classify a pattern to one or more categories. Many algorithms, based on multilayered perceptrons (MLP) [6], decision trees [7,8], k-nearest neighbors (KNN) [9], probability-based classifiers [10], support vector machines (SVM) [11,12], or random vector functional-link (RVFL) networks [13,14], have been developed for pattern classification.

Most developed algorithms are for single-label classification. KNN [9] is a lazy learning algorithm. It is based on similarity between input patterns and training data. The category to which the input pattern belongs to is determined by its k-nearest neighbors. Decision tree methods [7,15] build decision trees, e.g., ID3 and C4.5, from a set of training data using the concept of information entropy. An unseen pattern traverses down from the root of the tree until reaching a leaf which tells the category to which the unseen is assigned. Probability-based classifiers [16,17] assume independence among features. These classifiers can be trained for the estimation of parameters necessary for classification. MLP [18,19,20] is a feedforward multi-layer network model consisting of several layers of nodes, with two adjacent layers fully connected to each other. Each node in the network is associated with an activation function, linear or nonlinear. RVFL networks [13,14,21] are also fully-connected feedforward networks, with only one single hidden layer. The weights between input and hidden nodes are assigned to random values. The values of the weights between hidden and output nodes, instead, are learned from training patterns. An RBF network [22,23,24,25,26,27] is a two-layer network, using radial basis functions as the activation functions in the hidden layer. The output of the network is a linear combination of the outputs from the hidden layer. In SVM [11,12], training patterns are mapped into the points in a high-dimensional space and the points of different categories are separated in the space by a gap as wide as possible. For an unseen pattern, the same mapping is performed and the classification is predicted according to the side of the gap it falls on. For solving multi-label classification problems [28], two approaches are generally adopted [29]: In one approach, a multi-label classification task is transformed to several single-label classification tasks which are instead solved by single-label classification methods. In the other approach, the capability of a specific single-label classification algorithm is extended to handle directly the multi-label data [30,31,32,33].

Since first introduced in 1988 by Broomhead and Lowe [34], RBF networks have been becoming popular in classification applications. However, as pointed out in [34,35], in the construction phase of such networks, there are several issues encountered, such as the determination of the number of nodes in the hidden layer, the form and initialization of the basis functions, and the learning of the parameters involved in the networks. In this paper, we present a novel approach for constructing RBF networks for pattern classification problems. An iterative self-constructing clustering algorithm is used to produce a desired number of clusters from the training data. Accordingly, the number of nodes in the hidden layer is determined. Basis functions are then formed, and their centers and deviations are initialized to be the centers and deviations of the corresponding clusters. Then the parameters of the network are refined with a hybrid learning strategy, involving hyperbolic tangent sigmoid functions, steepest descent backpropagation and least squares method. As a result, optimized RBF networks are obtained. Our approach can offer advantages in practicality. The number of nodes in the hidden layer is determined and basis functions are derived automatically, and higher classification rates can be achieved through the hybrid learning process. Furthermore, the approach is applicable to construct RBF networks for solving both single-label and multi-label pattern classification problems. Experimental results have shown that the proposed approach can be used to solve classification tasks effectively.

We have been working on RBF networks for years, and have developed different techniques [26,27,36,37]. This paper is mainly based on the MS thesis of the first author, Z.-R. He (who preferred to use a different English name, Tsan-Jung He, in the thesis) [38], supervised by S.-J. Lee, with the following contributions:

• An iterative self-constructing clustering algorithm is applied to determine the number of hidden nodes and associated basis functions.
• The centers and deviations of the basis functions are refined through the steepest descent backpropagation method.
• Tikhonov regularization is applied in the optimization process to maintain the robustness of the output parameters.
• The hyperbolic tangent sigmoid function is used as the activation function of the output nodes when learning the parameters of basis functions.

The rest of this paper is organized as follows: Section 2 gives an overview of the related work. The proposed approach is described in detail in Section 3, Section 4 and Section 5. Basis functions are derived by clustering from training data and a hybrid learning is used to optimize network parameters. Experimental results are presented in Section 6. The effectiveness of the proposed approach on single-label and multi-label classification is demonstrated. Finally, concluding remarks are given in Section 7.

2. Related Work

Many single-label classification algorithms have been proposed. The extended nearest neighbor (ENN) method [39], unlike the classic KNN, makes the prediction in a “two-way communication” style. It considers not only what are the nearest neighbors of the input pattern, but also those of which the input pattern is their nearest neighbors. The Natural Neighborhood Based Classification Algorithm (NNBCA) [40] provides a good classification result without artificially selecting the neighborhood parameter. Unlike KNN, it predicts different k for different samples. An enhanced general fuzzy min-max neural network (EGFM) classification model is proposed in [19] to perform supervised classification of data. New hyperbox expansion, overlap, and contraction rules are used to overcome some unidentified cases in some regions. SaE-ELM [41] is an improved algorithm of RVFL networks. In SaE-ELM, the hidden node parameters are optimized by the self-adaptive differential evolution algorithm, whose trial vector generation strategies and associated control parameters are self-adapted in a strategy pool by learning from their previous experiences in generating promising solutions, and the network output weights are calculated using the Moore–Penrose generalized inverse. In [42], basis functions in an RBF network are interpreted as probability density functions. The weights are seen as prior probabilities. Models that output class conditional densities or mixture densities were proposed. In [24], an RBF network based on KNN with adaptive selection radius is proposed. An adaptive selection radius is calculated according to the population density sampling method. The RBF network is trained to locate the sound source by solving nonlinear equations about the time difference between arrival and source of the sound.

Several problem transformation methods exist for multi-label classification. One popular transformation method, called binary relevance [43], transforms the original training dataset into p datasets, where p is the number of categories involved with the dataset. Each resulting dataset contains all the patterns of the original dataset, with each pattern assigned to one of the two labels: “belonging to” or “not belonging to” a particular category. Since the resulting datasets are all single-labeled, any single-label classification technique is applicable to them. The label powerset (LP) transformation [44] method creates one binary classifier for every label combination present in the training set. For example, eight possible label combinations are created if the number of categories associated with the original dataset is three. Ensemble methods were developed to create multi-label ensemble classifiers [45,46]. A set of multi-class single-label classifiers are created. For an input example, each classifier outputs a single class. These predictions are then combined by an ensemble method.

Some classification algorithms/models have been adapted to the multi-label task, without requiring problem transformations. Clare [29] is an adapted C4.5 version for multi-label classification. Other decision tree classification methods based on multi-valued attributes were also developed [47]. Zhang et al. [32] propose several multi-label learning algorithms including back-propagation multi-label learning (BP-MLL) [48], multi-label k-nearest neighbors (ML-KNN) [49] and multi-label RBF (ML-RBF) [50]. BP-MLL is a multi-label version of the back-propagation neural network. To take care of multi-labels, label co-occurrence is incorporated into the pairwise ranking loss function. However, it has a complex global error function to be minimized. ML-KNN is a lazy learning algorithm which requires a big run-time search. ML-RBF is a multi-label RBF neural network which is an extension of the traditional RBF learning algorithm. Multi-label with Fuzzy Relevance Clustering (ML-FRC), proposed by Lee and Jiang [51], is a fuzzy relevance clustering based method for the task of multi-label text categorization. Kurata et al. [52] treat some of the nodes in the final hidden layer as dedicated neurons for each training pattern assigned to more than one label. These dedicated neurons are initialized to connect to the corresponding co-occurring labels with stronger weights than to others. A multi-label metamorphic relations prediction approach, named RBF-MLMR [33], is proposed to find a set of appropriate metamorphic relations for metamorphic testing. It uses soot analysis tool to generate the control flow graph and labels. The extracted nodes and the path properties constitute multi-label data sets for the control flow graph. Then, a multi-label RBF network prediction model is established to predict for multiple metamorphic relations.

3. Proposed Approach

Let $X=\left\{\left(x^{\left(i\right)},y^{\left(i\right)}\right)\right|1\le i\le N\}$ be a finite set of $N$ training patterns, where $x^{\left(i\right)}\in {ℝ}^n$ is an input vector with n feature values, i.e., $x^{\left(i\right)}={\left[x_1^{\left(i\right)}{x}_2^{\left(i\right)}\cdots x_n^{\left(i\right)}\right]}^T$, and $y^{\left(i\right)}$ is the corresponding category vector with m components, i.e., $y^{\left(i\right)}={\left[y_1^{\left(i\right)}{y}_2^{\left(i\right)}\cdots y_m^{\left(i\right)}\right]}^T$, of pattern $i$, defined as

$$y_f^{\left(i\right)}=\{\begin{array}{l}+1,\mathrm{if} x^{\left(i\right)} \mathrm{belong} \mathrm{to} \mathrm{category} f;\\ -1,\mathrm{if} x^{\left(i\right)} \mathrm{does} \mathrm{not} \mathrm{belong} \mathrm{to} \mathrm{category} f\end{array}$$

(1)

for $1\le f\le m$. Note that $m$ denotes the number of categories associated with the training dataset X. For convenience, the categories are labelled as 1, 2, ⋯, $m$ respectively. For single-label classification, one of the elements in $y^{\left(i\right)}$ is +1 and the other m−1 elements are −1. For a multi-label classification, two or more components in $y^{\left(i\right)}$ can be +1. Pattern classification is concerned with constructing a predicting model from X, and for any input vector x the model can respond with output $\widehat{y}$ showing which categories x belongs to.

In this work, we construct RBF networks as predicting models for classification. The data for training and testing the models will be collected from the openly accessible websites. The number of hidden nodes will be determined by clustering on the training data. The parameters of an RBF network will be learned by applying a hybrid learning algorithm. The performance of the constructed models will be analyzed by five-fold cross validation. Experiments for comparing performance with other methods will be conducted.

Our RBF network architecture, shown in Figure 1, is a two-layer network consisting of one hidden layer and one output layer. There are n input nodes, each node receiving and feeding forward one feature value of the input vector to the hidden layer. The hidden layer has J nodes, each with a basis function as its activation function. Each input node is fully connected to every node in the hidden layer. The output layer has m nodes, each corresponding to one distinct category. The hidden and output layers are also fully connected, i.e., each node in the hidden layer is connected to each node in the output layer. The weight between node $j$, $1\le j\le J$, of the hidden layer and node $f$, $1\le f\le m$, of the output layer is denoted by w_f,j for $1\le j\le J$ and $1\le f\le m$.

When an input vector $x={\left[x_1{x}_2\cdots x_n\right]}^T$ is presented to the input nodes of the network, the network produces the output vector $\widehat{y}={\left[{\widehat{y}}_1{\widehat{y}}_2\cdots {\widehat{y}}_m\right]}^T$ at the output layer. The network operates as follows.

• Input node $\mathit{i}$, $\mathbf{1}\le \mathit{i}\le \mathit{n}$, takes ${\mathit{x}}_{\mathit{i}}$ and dispatches it to every node in the hidden layer.
• Hidden layer. Hidden node $\mathit{j}$, $\mathbf{1}\le \mathit{j}\le \mathit{J}$, is responsible for computing the value of its designated basis function as

  $$B_j\left(x\right)={\displaystyle \prod }_{k=1}^n\exp \left[-{\left(\frac{x_k-c_{k,j}}{\alpha v_{k,j}}\right)}^2\right]$$

  (2)

  where $c_{k,j}$ and $v_{k,j}$ denote the center and deviation, respectively, of the kth dimension. Note that $\alpha$ is a constant for controlling the scope of the function. Let

  $$h_j=B_j\left(x\right)$$

  (3)

  which is forwarded to the output layer.
• Output layer. Node $\mathit{f}$, $\mathbf{1}\le \mathit{f}\le \mathit{m}$, in this layer calculates the weighted sum

  $$s_f=w_{f,1}{h}_1+w_{f,2}{h}_2+\cdots +w_{f,J}{h}_J+b_f$$

  (4)

  where $w_{f,1},w_{f,2},\cdots ,w_{f,J}$ are the weights and $b_f$ is the bias associated with it. Then it provides the network output ${\widehat{y}}_f$ by

  $${\widehat{y}}_f=\mathrm{sign}\left(s_f\right)$$

  (5)

  where sign $(·)$ is the sign function defined as

  $$\mathrm{sign}\left(s_f\right)=\{\begin{array}{c}+1,\mathrm{if}{s}_f\ge 0;\\ -1,\mathrm{if}{s}_f<0.\end{array}$$

  (6)
  Therefore, the network output $\widehat{y}$ for the input vector x is

  $$\widehat{y}={\left[\begin{array}{ccc}\begin{array}{cc}{\widehat{y}}_2& {\widehat{y}}_2\end{array}& \cdots & {\widehat{y}}_m\end{array}\right]}^T$$

  (7)

Each component in $\widehat{y}$ is either +1 or –1. If component $f$ is +1, $1\le f\le m$, x is predicted to belong to category $f$. Note that one component or more in $\widehat{y}$ can be +1. Therefore, the proposed network architecture is applicable to solving both single-label and multi-label pattern classification problems.

We describe below how the network of Figure 1 is created from the given training set. Two phases, network setup and parameter refinement, are involved. In the first phase, the network structure is built and the initial values of the parameters are set. Then the parameters of the network are optimally refined in the second phase.

4. Network Setup Phase

In this phase, the number of nodes and basis functions in the hidden layer are determined automatically. Furthermore, the centers and deviations of the basis functions, and the weights and biases associated with the output layer, are initialized appropriately.

4.1. Initialization of Basis Functions

To determine the number of nodes and basis functions in the hidden layer, we find a group of clusters in the training set $X$ through a clustering process. Firstly, we divide the training set $X$ into $m$ training subsets $X_1,X_2,\cdots ,X_m$ as follows:

$$X_h=\left\{x^{\left(i\right)}|1\le i\le N\wedge y_h^{\left(i\right)}=1\right\}$$

(8)

for $1\le h\le m$. Then, each subset is fed to an iterative self-constructing clustering algorithm [53] and is grouped into $K_1,\cdots ,K_m$ clusters, respectively. A cluster contains patterns which are similar to each other, and is characterized by its center and deviation.

Let $K=K_1+\cdots +K_m$. Therefore, we have $K$ clusters in total and let them be denoted as $C_1,C_2,\cdots ,C_K$ with centers ${\mathrm{c}}_k={\left[c_{1,k}\cdots c_{n,k}\right]}^T$ and deviations ${\mathrm{v}}_k={\left[v_{1,k}\cdots v_{n,k}\right]}^T$, $1\le k\le K$. Then we set the number of hidden nodes, $J$, to be equal to $K$, i.e., $J=K$. For hidden node $j$, its associated basis function, $B_j\left(x\right)$, is also determined, with its center and deviation initialized to be the center and deviation of cluster $C_j$, i.e., we set

$$B_j\left(x\right)={\displaystyle \prod }_{i=1}^n\exp \left[-{\left(\frac{x_i-c_{i,j}}{\alpha v_{i,j}}\right)}^2\right]$$

(9)

for $1\le j\le J$, with $c_{i,j}$ and $v_{i,j}$ being the center and deviation, respectively, in the ith dimension of $C_j$.

4.2. Initialization of Weights and Biases

Next, we derive initial values for the weights $w_{f,j}$, $1\le j\le J$, $1\le f\le m$, between the hidden layer and the output layer, and for the biases $b_f$, $1\le f\le m$, in the output layer. Consider the sum of squared error of the whole training set $X$:

$${\displaystyle \sum }_{i=1}^N\parallel y^{\left(i\right)}-{\widehat{y}}^{\left(i\right)}{\parallel }^2={\displaystyle \sum }_{i=1}^N{\displaystyle \sum }_{f=1}^m{\left[y_f^{\left(i\right)}-{\widehat{y}}_f^{\left(i\right)}\right]}^2$$

(10)

By Equations (4) and (5), we have

$${\widehat{y}}_f^{\left(i\right)}=\mathrm{sign}\left(s_f^{\left(i\right)}\right),$$

(11)

$$s_f^{\left(i\right)}=w_{1,f}{h}_1^{\left(i\right)}+w_{2,f}{h}_2^{\left(i\right)}+\cdots +w_{J,f}{h}_J^{\left(i\right)}+b_f$$

(12)

where $h_J^{\left(i\right)}=B_j\left(x^{\left(i\right)}\right)$, for $1\le f\le m$ and $1\le i\le N$. For pattern $i$, we have

$$\begin{array}{l}{s}_1^{\left(i\right)}=w_{1,1}{h}_1^{\left(i\right)}+w_{1,2}{h}_2^{\left(i\right)}+\cdots +w_{1,J}{h}_J^{\left(i\right)}+b_1\\ s_2^{\left(i\right)}=w_{2,1}{h}_1^{\left(i\right)}+w_{2,2}{h}_2^{\left(i\right)}+\cdots +w_{2,J}{h}_J^{\left(i\right)}+b_2\\ \cdots =\cdots ,\\ s_m^{\left(i\right)}=w_{m,1}{h}_1^{\left(i\right)}+w_{m,2}{h}_2^{\left(i\right)}+\cdots +w_{m,J}{h}_J^{\left(i\right)}+b_m\end{array}$$

(13)

In total, there are N × m such equations. Although only the sign of $s_f^{\left(i\right)}$ matters in Equation (11), to be robust we’d like $s_f^{\left(i\right)}$ as close to $y_f^{\left(i\right)}$, which is either +1 or −1, as possible. Therefore, we demand

$$\begin{array}{l}{y}_1^{\left(i\right)}=w_{1,1}{h}_1^{\left(i\right)}+w_{1,2}{h}_2^{\left(i\right)}+\cdots +w_{1,J}{h}_J^{\left(i\right)}+b_1\\ y_2^{\left(i\right)}=w_{2,1}{h}_1^{\left(i\right)}+w_{2,2}{h}_2^{\left(i\right)}+\cdots +w_{2,J}{h}_J^{\left(i\right)}+b_2\\ \cdots =\cdots ,\\ y_m^{\left(i\right)}=w_{m,1}{h}_1^{\left(i\right)}+w_{m,2}{h}_2^{\left(i\right)}+\cdots +w_{m,J}{h}_J^{\left(i\right)}+b_m\end{array}$$

(14)

for $1\le i\le N$. Equation (14) can be rewritten as

$$\parallel AW-Y{\parallel }^2$$

(15)

where

$$A={\left[\begin{array}{cccc}{h}_1^{\left(1\right)}& h_1^{\left(2\right)}& \cdots & h_1^{\left(N\right)}\\ h_2^{\left(1\right)}& h_2^{\left(2\right)}& \cdots & h_2^{\left(N\right)}\\ ⋮& ⋮& \ddots & ⋮\\ h_J^{\left(1\right)}& h_J^{\left(2\right)}& \cdots & h_J^{\left(N\right)}\\ 1& 1& \cdots & 1\end{array}\right]}^T,$$

(16)

$$W={\left[\begin{array}{ccc}{w}_{1,1}& w_{1,2}& \begin{array}{ccc}\cdots & w_{1,J}& b_1\end{array}\\ w_{2,1}& w_{2,2}& \begin{array}{ccc}\cdots & w_{2,J}& b_2\end{array}\\ \begin{array}{c}⋮\\ w_{m,1}\end{array}& \begin{array}{c}⋮\\ w_{m,2}\end{array}& \begin{array}{ccc}\begin{array}{c}\ddots \\ \cdots \end{array}& \begin{array}{c}⋮\\ w_{m,J}\end{array}& \begin{array}{c}⋮\\ b_m\end{array}\end{array}\end{array}\right]}^T,$$

(17)

$$Y={\left[\begin{array}{ccc}{y}_1^{\left(1\right)}& y_1^{\left(2\right)}& \begin{array}{cc}\cdots & y_1^{\left(N\right)}\end{array}\\ y_2^{\left(1\right)}& y_2^{\left(2\right)}& \begin{array}{cc}\cdots & y_2^{\left(N\right)}\end{array}\\ \begin{array}{c}⋮\\ y_m^{\left(1\right)}\end{array}& \begin{array}{c}⋮\\ y_m^{\left(2\right)}\end{array}& \begin{array}{cc}\ddots & ⋮\\ \cdots & y_m^{\left(N\right)}\end{array}\end{array}\right]}^T$$

(18)

with size of $N\times \left(J+1\right)$, $\left(J+1\right)\times m$, and $N\times m$, respectively.

To improve the robustness of the system [54], instead of minimizing Equation (15), we would like to minimize the penalized residual sum of squares:

$$\parallel AW-Y{\parallel }^2+\lambda \parallel W{\parallel }^2$$

(19)

where $\lambda$ is a pre-defined non-negative coefficient to balance the robustness against collinearity and the capability of fitting the desired outputs well. To find the solution to the minimization problem, we first rewrite Equation (19) as

$$\parallel AW-Y{\parallel }^2+\lambda \parallel W{\parallel }^2=\parallel AW-Y{\parallel }^2+\parallel \sqrt{\lambda }IW-\mathbf{0}{\parallel }^2=\parallel \left[\begin{array}{c}A\\ \sqrt{\lambda }I\end{array}\right]W-\left[\begin{array}{c}Y\\ \mathbf{0}\end{array}\right]{\parallel }^2$$

where $\mathbf{I}$ is the $\left(J+1\right)$ × $\left(J+1\right)$ identity matrix and $\mathbf{0}$ is the $\left(J+1\right)\times m$ zero matrix. Let $A^{\prime }=\left[\begin{array}{c}A\\ \sqrt{\lambda }I\end{array}\right]$ and $Y^{\prime }=\left[\begin{array}{c}Y\\ 0\end{array}\right]$. Then Equation (19) becomes

$$\parallel A^{\prime }W-Y^{\prime }{\parallel }^2$$

(20)

By the least squares method [55], the solution minimizing Equation (19) is

$$\begin{gathered} \begin{array}{l}\widehat{W}={\left\{{\left(A^{\prime }\right)}^T{A}^{\prime }\right\}}^{-1}{A}^{\prime }{Y}^{\prime }\\ ={\left\{{\left[\begin{array}{c}A\\ \sqrt{\lambda }I\end{array}\right]}^T\left[\begin{array}{c}A\\ \sqrt{\lambda }I\end{array}\right]\right\}}^{-1}\left[\begin{array}{c}A\\ \sqrt{\lambda }I\end{array}\right]\left[\begin{array}{c}Y\\ \mathbf{0}\end{array}\right]\end{array} \\ ={\left\{\left[\begin{array}{cc}{A}^T& \sqrt{\lambda }I\end{array}\right]\left[\begin{array}{c}A\\ \sqrt{\lambda }I\end{array}\right]\right\}}^{-1}\left[\begin{array}{cc}{A}^T& \sqrt{\lambda }I\end{array}\right]\left[\begin{array}{c}Y\\ \mathbf{0}\end{array}\right] \\ ={\left\{A^{T}A+\lambda I\right\}}^{-1}{A}^{T}Y \end{gathered}$$

(21)

Note that the solution is unique even when $A$ has dependent columns. This solution, $\widehat{W}$, provides initial values for the weights $w_{f,j}$, $1\le j\le J$, $1\le f\le m$, and biases $b_f$, $1\le f\le m$, which can be refined later.

5. Parameter Refinement Phase

In this phase, the optimal values of the parameters of the network are learned from the training dataset $X$. Such parameters include the centers ${\mathrm{c}}_j={\left[c_{1,j}\cdots c_{n,j}\right]}^T$ and deviations ${\mathrm{v}}_j={\left[v_{1,j}\cdots v_{n,j}\right]}^T$, $1\le j\le J$, associated with the basis functions in the hidden layer, and the weights $w_{f,j}$, $1\le j\le J$, $1\le f\le m$, between the hidden layer and the output layer, and the biases and biases $b_f$, $1\le f\le m$, associated with the output nodes in the output layer. A hybrid learning process is applied iteratively. In each iteration, we first treat all the weights and biases as fixed and use the steepest descent backpropagation to update the centers and deviations associated with the basis functions. Then we treat all the centers and deviations associated with the basis functions as fixed and use the least squares method to update the weights and biases associated with the output layer. The process is iterated until convergence is reached.

5.1. Centers and Deviations

First, we describe how to refine the centers and deviations associated with the basis functions in the hidden layer. To make the refinement possible, we express ${\widehat{y}}_f$ in Equation (5) in terms of the hyperbolic tangent sigmoid function which is analytic, i.e.,

$${\widehat{y}}_f\approx a_f=g\left(s_f\right)=\frac{e^{\beta s_f}-e^{-\beta s_f}}{e^{\beta s_f}+e^{-\beta s_f}}$$

(22)

for $1\le f\le m$, where $\beta$ is a slope-controlling constant and $s_f$ is defined in Equation (4). Note that $a_f$ provides a good approximation to the sign function and it is differentiable at all points.

Let us denote the initial values of the centers and deviations, shown in Equation (9), as $c_{k,j}\left(0\right)$ and $v_{k,j}\left(0\right)$, $1\le k\le n$, $1\le j\le J$, and the initial values of the weights and biases, obtained in Equation (21), as $w_{f,i}\left(0\right)$ and $b_f\left(0\right)$, $1\le f\le m$, $1\le i\le J$. As in [56], the mean squared error for the training set $X$ can be estimated as

$$\begin{array}{cc}\widehat{E}\left(k\right)\hfill & =\parallel y\left(k\right)-\widehat{y}\left(k\right){\parallel }^2\hfill \\ & \approx \parallel y\left(k\right)-a\left(k\right){\parallel }^2={\displaystyle {\displaystyle \sum }_{l=1}^m}{\left[y_l\left(k\right)-a_l\left(k\right)\right]}^2\hfill \end{array}$$

(23)

where $\left(x\left(k\right),y\left(k\right)\right)$ can be any pattern in the training dataset $X$. By steepest descent backpropagation, the centers and deviations of the basis functions in the hidden layer are updated as

$$c_{i,j}\left(k+1\right)=c_{i,j}\left(k\right)-\eta \frac{\partial \widehat{E}\left(k\right)}{\partial c_{i,j}},1\le i\le n,1\le j\le J$$

(24)

$$v_{i,j}\left(k+1\right)=v_{i,j}\left(k\right)-\eta \frac{\partial \widehat{E}\left(k\right)}{\partial v_{i,j}},1\le i\le n,1\le j\le J$$

(25)

for $k\ge 0$, where $\eta$ is the learning rate. From Equation (23), we have

$$\frac{\partial \widehat{E}\left(k\right)}{\partial c_{i,j}}=-2{\displaystyle {\displaystyle \sum }_{l=1}^m}\left[y_l\left(k\right)-a_l\left(k\right)\right]\frac{\partial a_l\left(k\right)}{\partial c_{i,j}}$$

(26)

$$\frac{\partial \widehat{E}\left(k\right)}{\partial v_{i,j}}=-2{\displaystyle {\displaystyle \sum }_{l=1}^m}\left[y_l\left(k\right)-a_l\left(k\right)\right]\frac{\partial a_l\left(k\right)}{\partial v_{i,j}}$$

(27)

By chain rule, we have

$$\frac{\partial a_l\left(k\right)}{\partial c_{i,j}}=\frac{\partial a_l\left(k\right)}{\partial s_l}\frac{\partial s_l\left(k\right)}{\partial h_j}\frac{\partial h_j\left(k\right)}{\partial c_{i,j}}$$

(28)

Since $a_l\left(k\right)=g\left(s_l\left(k\right)\right),s_l\left(k\right)=w_{l,1}\left(k\right)h_1\left(k\right)+\cdots +w_{l,J}\left(k\right)h_J\left(k\right)+b_l\left(k\right)$, and $h_j\left(k\right)={\displaystyle \prod }_{r=1}^n{e}^{-{\left(\frac{x_r\left(k\right)-c_{r,j}\left(k\right)}{\alpha v_{r,j}\left(k\right)}\right)}^2}$, we have

$$\begin{array}{cc}\frac{\partial a_l\left(k\right)}{\partial s_l}\hfill & =\dot{g}\left(s_l\left(k\right)\right)\hfill \\ & =\frac{\partial \left[e^{\beta s_l\left(k\right)}-e^{-\beta s_l\left(k\right)}\right]}{\partial s_l}\frac{1}{e^{\beta s_l\left(k\right)}-e^{-\beta s_l\left(k\right)}}+\left[e^{\beta s_l\left(k\right)}-e^{-\beta s_l\left(k\right)}\right]\frac{\partial {\left[e^{\beta s_l\left(k\right)}-e^{-\beta s_l\left(k\right)}\right]}^{-1}}{\partial s_l}\hfill \\ & =\beta \left(1-a_l^2\left(k\right)\right)\hfill \end{array}$$

(29)

$$\frac{\partial s_l\left(k\right)}{\partial h_j}=w_{l,j}\left(k\right)$$

(30)

$$\begin{array}{cc}\frac{\partial h_j\left(k\right)}{\partial c_{i,j}}\hfill & =\frac{\partial e^{-{\left(\frac{x_i\left(k\right)-c_{i,j}\left(k\right)}{\alpha v_{i,j}\left(k\right)}\right)}^2}}{\partial c_{i,j}}{\displaystyle {\displaystyle \prod }_{r\ne i}^n}{e}^{-{\left(\frac{x_r\left(k\right)-c_{r,j}\left(k\right)}{\alpha v_{r,j}\left(k\right)}\right)}^2}\hfill \\ & =\frac{\partial \left[-{\left(\frac{x_i\left(k\right)-c_{i,j}\left(k\right)}{\alpha v_{i,j}\left(k\right)}\right)}^2\right]}{\partial c_{i,j}}{e}^{-{\left(\frac{x_i\left(k\right)-c_{i,j}\left(k\right)}{\alpha v_{i,j}\left(k\right)}\right)}^2}{\displaystyle {\displaystyle \prod }_{r\ne i}^n}{e}^{-{\left(\frac{x_r\left(k\right)-c_{r,j}\left(k\right)}{\alpha v_{r,j}\left(k\right)}\right)}^2}\hfill \\ & =\frac{2\left[x_i\left(k\right)-c_{i,j}\left(k\right)\right]}{{\left[\alpha v_{i,j}\left(k\right)\right]}^2}{h}_j\left(k\right)\hfill \end{array}$$

(31)

Similarly, we have

$$\frac{\partial a_l\left(k\right)}{\partial v_{i,j}}=\frac{\partial a_l\left(k\right)}{\partial s_l}\frac{\partial s_l\left(k\right)}{\partial h_j}\frac{\partial h_j\left(k\right)}{\partial v_{i,j}}$$

(32)

$$\frac{\partial h_j\left(k\right)}{\partial v_{i,j}}=2\frac{{\left[x_i\left(k\right)-c_{i,j}\left(k\right)\right]}^2}{{\alpha }^2{v}_{i,j}^3\left(k\right)}{h}_j\left(k\right)$$

(33)

Therefore, we have

$$c_{i,j}\left(k+1\right)=c_{i,j}\left(k\right)+4\eta \frac{x_i\left(k\right)-c_{i,j}\left(k\right)}{{\left[\alpha v_{i,j}\left(k\right)\right]}^2}{h}_j\left(k\right){\displaystyle \sum }_{l=1}^m\left[y_l\left(k\right)-a_l\left(k\right)\right]\dot{\dot{g}\left(s_l\left(k\right)\right)}{w}_{l,j}\left(k\right)$$

(34)

$$v_{i,j}\left(k+1\right)=v_{i,j}\left(k\right)+4\eta \frac{{\left[x_i\left(k\right)-c_{i,j}\left(k\right)\right]}^2}{{\alpha }^2{v}_{i,j}^3\left(k\right)}{h}_j\left(k\right){\displaystyle \sum }_{l=1}^m\left[y_l\left(k\right)-a_l\left(k\right)\right]\dot{\dot{g}\left(s_l\left(k\right)\right)}{w}_{l,j}\left(k\right)$$

(35)

for 1 ≤ i ≤ n, 1 ≤ j ≤ J. In matrix form this becomes:

$${\mathrm{c}}_j\left(k+1\right)={\mathrm{c}}_j\left(k\right)+4\eta h_j\left(k\right)D_{1,j}\left(k\right)q_{1,j}\left(k\right)w_j{\left(k\right)}^T\dot{G}\left(s\left(k\right)\right)e\left(k\right)$$

(36)

$${\mathrm{v}}_j\left(k+1\right)={\mathrm{v}}_j\left(k\right)+4\eta h_j\left(k\right)D_{2,j}\left(k\right)q_{2,j}\left(k\right)w_j{\left(k\right)}^T\dot{G}\left(s\left(k\right)\right)e\left(k\right)$$

(37)

for $1\le j\le J$, where

$$q_{1,j}\left(k\right)={\left[\begin{array}{ccc}{x}_1\left(k\right)-c_{1,j}\left(k\right)& x_2\left(k\right)-c_{2,j}\left(k\right)& \begin{array}{cc}\cdots & x_n\left(k\right)-c_{n,j}\left(k\right)\end{array}\end{array}\right]}^T$$

(38)

$$q_{2,j}\left(k\right)={\left[\begin{array}{ccc}{\left[x_1\left(k\right)-c_{1,j}\left(k\right)\right]}^2& {\left[x_2\left(k\right)-c_{2,j}\left(k\right)\right]}^2& \begin{array}{cc}\cdots & {\left[x_n\left(k\right)-c_{n,j}\left(k\right)\right]}^2\end{array}\end{array}\right]}^T$$

(39)

$$D_{1,j}\left(k\right)=\left[\begin{array}{c}\begin{array}{cccccc}\frac{1}{{\left[\alpha v_{1,j}\left(k\right)\right]}^2}& 0& 0& \cdots & 0& 0\\ 0& \frac{1}{{\left[\alpha v_{2,j}\left(k\right)\right]}^2}& 0& \cdots & 0& 0\\ ⋮& ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& 0& \cdots & 0& \frac{1}{{\left[\alpha v_{n,j}\left(k\right)\right]}^2}\end{array}\end{array}\right]$$

(40)

$$D_{2,j}\left(k\right)=\left[\begin{array}{c}\begin{array}{cccccc}\frac{1}{{\left[{\alpha }^2{v}_{1,j}^3\left(k\right)\right]}^2}& 0& 0& \cdots & 0& 0\\ 0& \frac{1}{{\left[{\alpha }^2{v}_{2,j}^3\left(k\right)\right]}^2}& 0& \cdots & 0& 0\\ ⋮& ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& 0& \cdots & 0& \frac{1}{{\left[{\alpha }^2{v}_{n,j}^3\left(k\right)\right]}^2}\end{array}\end{array}\right]$$

(41)

$$w_j\left(k\right)={\left[\begin{array}{ccc}{w}_{1,j}\left(k\right)& w_{2,j}\left(k\right)& \begin{array}{cc}\cdots & w_{m,j}\left(k\right)\end{array}\end{array}\right]}^T$$

(42)

$$\dot{G}\left(s\left(k\right)\right)=\left[\begin{array}{cccccc}\dot{g}\left(s_1\left(k\right)\right)& 0& 0& \cdots & 0& 0\\ 0& \dot{g}\left(s_2\left(k\right)\right)& 0& \cdots & 0& 0\\ ⋮& ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& 0& \cdots & 0& \dot{g}\left(s_m\left(k\right)\right)\end{array}\right]$$

(43)

$$e\left(k\right)={\left[\begin{array}{ccc}{y}_1\left(k\right)-a_1\left(k\right)& y_2\left(k\right)-a_2\left(k\right)& \begin{array}{cc}\cdots & y_m\left(k\right)-a_m\left(k\right)\end{array}\end{array}\right]}^T$$

(44)

Note that the algorithm described above is the stochastic steepest descent backpropagation which involves incremental training. However, we perform batch training in which the complete gradient is computed, after all inputs are applied to the network, before the centers and deviations are updated. To implement the batch training, in each iteration the individual gradients for all the inputs in the training set are averaged to get the total gradient, and the update equations become

$$c_{i,j}\left(k+1\right)=c_{i,j}\left(k\right)-\frac{\eta }{N}{\displaystyle \sum }_{p=1}^N\frac{\partial {\widehat{E}}_p\left(k\right)}{\partial c_{i,j}}$$

(45)

$$v_{i,j}\left(k+1\right)=v_{i,j}\left(k\right)-\frac{\eta }{N}{\displaystyle \sum }_{p=1}^N\frac{\partial {\widehat{E}}_p\left(k\right)}{\partial v_{i,j}}$$

(46)

for $1\le i\le n$, $1\le j\le J$, where ${\widehat{E}}_p\left(k\right)$ is the mean squared error $\widehat{E}\left(k\right)$ induced from pattern $p$, $\left(x^{\left(p\right)},y^{\left(p\right)}\right)$. In matrix form, the batch training becomes:

$$c_j\left(k+1\right)=c_j\left(k\right)+4\frac{\eta }{N}{\displaystyle \sum }_{p=1}^N\left\{h_j\left(p\right)D_{1,j}\left(p\right)q_{1,j}\left(p\right)w_j{\left(p\right)}^T\dot{G}\left(s\left(p\right)\right)e\left(p\right)\right\}$$

(47)

$$V_j\left(k+1\right)=v_j\left(k\right)+4\frac{\eta }{N}{\displaystyle \sum }_{p=1}^N\left\{h_j\left(p\right)D_{2,j}\left(p\right)q_{2,j}\left(p\right)w_j{\left(p\right)}^T\dot{G}\left(s\left(p\right)\right)e\left(p\right)\right\}$$

(48)

for $1\le j\le J$.

5.2. Weights and Biases

Next, we describe how to refine weights and biases associated with the output layer in each iteration. For pattern $i$, $1\le i\le N$, by Equation (13), we want

$$\begin{gathered} y_1^{\left(i\right)}=w_{1,1}\left(k+1\right)h_1^{\left(i\right)}\left(k+1\right)+\cdots +w_{1,J}\left(k+1\right)h_J^{\left(i\right)}\left(k+1\right)+b_1\left(b+1\right) \\ {y}_2^{\left(i\right)}=w_{2,1}\left(k+1\right)h_1^{\left(i\right)}\left(k+1\right)+\cdots +w_{2,J}\left(k+1\right)h_J^{\left(i\right)}\left(k+1\right)+b_2\left(b+1\right) \\ \cdots =\cdots \\ {y}_m^{\left(i\right)}=w_{m,1}\left(k+1\right)h_1^{\left(i\right)}\left(k+1\right)+\cdots +w_{m,J}\left(k+1\right)h_J^{\left(i\right)}\left(k+1\right)+b_m\left(b+1\right) \end{gathered}$$

where

$$h_j^{\left(i\right)}\left(k+1\right)={\displaystyle \prod }_{r=1}^n{e}^{-{\left(\frac{x_r^{\left(i\right)}-c_{r,j}\left(k+1\right)}{\alpha v_{r,j}\left(k+1\right)}\right)}^2}$$

(49)

for $1\le j\le J$ and $1\le i\le N$. Note that $c_{r,j}\left(k+1\right)$ and $v_{r,j}\left(k+1\right)$ are the newly updated values obtained in the first part of the current iteration. Let

$$A\left(k+1\right)=[\begin{array}{cccc}{h}_1^{\left(1\right)}\left(k+1\right)& h_1^{\left(2\right)}\left(k+1\right)& \cdots & h_1^{\left(N\right)}\left(k+1\right)\\ h_2^{\left(1\right)}\left(k+1\right)& h_2^{\left(2\right)}\left(k+1\right)& \cdots & h_2^{\left(N\right)}\left(k+1\right)\\ ⋮& ⋮& \ddots & ⋮\\ h_J^{\left(1\right)}\left(k+1\right)& h_J^{\left(2\right)}\left(k+1\right)& \cdots & h_J^{\left(N\right)}\left(k+1\right)\\ 1& 1& \cdots & 1\end{array}{]}^T$$

(50)

and

$$W(k+1)=[\begin{array}{ccccc}{w}_{1,1}\left(k+1\right)& w_{1,}{}_2\left(k+1\right)& \cdots & w_{1,J}\left(k+1\right)& b_1\left(k+1\right)\\ w_{2,1}\left(k+1\right)& w_{2,2}\left(k+1\right)& \cdots & w_{2,J}\left(k+1\right)& b_2\left(k+1\right)\\ ⋮& ⋮& \ddots & ⋮& ⋮\\ w_{m,1}\left(k+1\right)& w_{m,2}\left(k+1\right)& \cdots & w_{m,J}\left(k+1\right)& b_m\left(k+1\right)\end{array}{]}^T$$

(51)

and $Y$ be the same matrix as shown in Equation (18). As before, we’d like to minimize

$$\parallel A\left(k+1\right)W\left(k+1\right)-Y{\parallel }^2+\lambda \parallel W\left(k+1\right){\parallel }^2$$

(52)

The solution is ${\left\{A^T\left(k+1\right)A\left(k+1\right)+\lambda I\right\}}^{-1}{A}^T\left(k+1\right)Y$ which provides the new values of the weights and biases, i.e., $w_{f,j}\left(k+1\right)$ and $b_f\left(k+1\right)$, $1\le j\le J$, $1\le f\le m$.

6. Experimental Results

In this section, experimental results are presented to show the effectiveness of the proposed approach. Comparison results with other classification methods are also presented. For convenience, our approach is abbreviated as RBF-ISCC (RBF with Iterative Self-Constructing Clustering). We used a computer with Intel(R) Core(TM) i7 [email protected] and 16 GB of RAM to conduct the experiments.

A five-fold cross validation was adopted to measure the performance of a classifier in the following experiments. Each dataset was randomly divided into five disjoint subsets, in such a way that the data of each category were evenly involved in five folds. Then, a classifier performed five runs for each dataset. In each run, four of the five subsets were used for training in the training phase and the remaining subset was used for testing in the testing phase. The results of the five runs were then averaged to indicate the performance of the classifier on the dataset. Note that the data for training were different from the data for testing in each run. The pre-defined constants, such as α of our method, were determined in the training phase and were not changed in the testing phase.

6.1. Experiment 1—Single-Label Data Sets

In this experiment, we would like to show the performance of single-label classification of different methods. Average accuracy (AACC), macro-averaged recall (MAREC), macro-averaged precision (MAPRE), and macro-averaged F-measure (MAFM) are adopted metrics for performance evaluation [57]. For category $i$, $1\le i\le m$, let TP_i, FP_i, FN_i, TN_i are the number of true positives, false positives, false negatives, and true negatives, respectively. AACC, MAREC, MAPRE, and MAFM are defined as

$$\begin{gathered} \mathrm{AACC}=\frac{{{\displaystyle \sum }}_{i=1}^m\frac{{\mathrm{TP}}_i+{\mathrm{TN}}_i}{{\mathrm{TP}}_i+{\mathrm{FP}}_i+{\mathrm{FN}}_i+{\mathrm{TN}}_i}}{m} \\ \mathrm{MAREC}=\frac{{{\displaystyle \sum }}_{i=1}^m\frac{{\mathrm{TP}}_i}{{\mathrm{TP}}_i+{\mathrm{FN}}_i}}{m} \\ \mathrm{MAPRE}=\frac{{{\displaystyle \sum }}_{i=1}^m\frac{{\mathrm{TP}}_i}{{\mathrm{TP}}_i+{\mathrm{FP}}_i}}{m} \\ \mathrm{MAFM}=\frac{2\times \mathrm{MAREC}\times \mathrm{MAPRE}}{\mathrm{MAREC}+\mathrm{MAPRE}} \end{gathered}$$

Clearly, higher values of the above metrics indicate a better classification. Several benchmark data sets are used in this experiment.

Table 1. Single-label data sets for experiments.

Data Set	# Features	# Categories	# Patterns	Cardinality
Liver Disorders	7	2	345	1.000
Heart	13	2	270	1.000
Ionosphere	34	2	351	1.000
Sonar	60	2	208	1.000
Iris	4	3	150	1.000
Lung Cancer	56	3	32	1.000
Soybean	35	4	47	1.000
Glass	9	6	214	1.000
Ecoli	8	8	336	1.000
Yeast-SL	8	10	1484	1.000

lists the characteristics of these data sets. For example, the dataset “Liver Disorders” contains 345 patterns, and each patterns has 7 features. There are two categories involved in the dataset. Note that cardinality denotes the average number of categories each pattern belongs to. Since each pattern belongs to one and only one category, the cardinality of the dataset is 1.000. The data sets of Table 1 are taken from the repository of the machine learning database at the University of California at Irvine (UCI) [58].

Table 2. Testing average accuracy (AACC) obtained for single-label data sets.

Data Set	LinearSVC	SVC-RBF	Support Vector Machines (SVM)	NN	Radial Basis Function (RBF)	RBF with Iterative Self-Constructing Clustering (RBF-ISCC)
Liver Disorders	0.7246	0.6812	0.7101	0.6232	0.5507	0.7536
Heart	0.7593	0.7593	0.7778	0.7407	0.7037	0.7778
Ionosphere	0.8873	0.9014	0.9014	0.8873	0.8310	0.9296
Sonar	0.8095	0.8571	0.8810	0.8810	0.8095	0.8333
Iris	0.9111	0.9778	0.9778	0.9778	0.9778	0.9778
Lung Cancer	0.5714	0.6190	0.5238	0.7143	0.4286	0.7143
Soybean	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
Glass	0.8837	0.8915	0.7907	0.8915	0.8915	0.8992
Ecoli	0.9632	0.9596	0.9044	0.9485	0.9412	0.9614
Yeast-SL	0.9101	0.9232	0.8532	0.9051	0.8956	0.9249
Ave-AACC	0.8420	0.8570	0.8320	0.8569	0.8030	0.8772

and

Table 3. Testing macro-averaged F-measure (MAFM) obtained for single-label data sets.

Data Set	LinearSVC	SVC-rbf	SVM	NN	RBF	RBF-ISCC
Liver Disorders	0.7189	0.6662	0.6983	0.6134	0.5599	0.7461
Heart	0.7587	0.7587	0.7768	0.7373	0.6973	0.7776
Ionosphere	0.8768	0.8920	0.8920	0.8750	0.8119	0.9266
Sonar	0.8132	0.8623	0.8808	0.8808	0.8091	0.8337
Iris	0.8421	0.9682	0.9682	0.9682	0.9682	0.9682
Lung Cancer	0.4915	0.2000	0.1667	0.6377	0.1333	0.5000
Soybean	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
Glass	0.6266	0.6325	0.4332	0.6015	0.7213	0.6325
Ecoli	0.8936	0.8679	0.5893	0.7079	0.7281	0.8663
Yeast-SL	0.4251	0.6614	0.2427	0.5426	0.5111	0.5450
Ave-MAFM	0.7447	0.7509	0.6648	0.7564	0.6940	0.7796

shows the testing AACC and MMFM, respectively, obtained by different methods, including LinearSVC [59], SVC-rbf (Support Vector Classifier with RBF kernel), SVM [11], NN (Neural Network) [60], RBF [22], and RBF-ISCC for the data sets in Table 1. SVC-rbf, support vector classification with RBF kernel, is implemented based on libsvm [61] and using one-vs.-rest scheme for classification. LinearSVC, Linear support vector classification, is similar to SVC with linear kernel, but implemented in terms of liblinear [59] rather than libsvm. We also used one-vs.-rest scheme for classification for LinearSVC. SVM is a support vector machine based on the MATLAB toolbox (http://www.mathworks.com). NN is a three-layer neural network, using the hyperbolic tangent sigmoid transfer function (tanh) in the hidden layer. NN optimizes the log-loss function using LBFGS (Limited-memory BFGS) [60], which is a BFGS (Broyden–Fletcher–Goldfarb–Shanno) based optimizer in the family of quasi-Newton methods. The codes of LinearSVC, SVC-rbf, and NN are downloaded from the scikit-learn library (http://scikit-learn.org) [62,63]. RBF is a radial basis function network downloaded from the website http://neupy.com/apidocs. In the tables, the best performance for each data set is boldfaced. The average matric values obtained by each method are shown in the bottom line of Table 2 and Table 3. Note that RBFISCC performs best, having the highest average AACC, 87.72% and the highest average MAFM, 77.96%. In particular, RBF-ISCC gets the highest AACC and MAFM for 8 and 5, respectively, out of 10 data sets.

Table 4. Testing macro-averaged recall (MAREC) obtained for single-label data sets.

Data Set	LinearSVC	SVC-rbf	SVM	NN	RBF	RBF-ISCC
Liver Disorders	0.7198	0.6586	0.6931	0.6134	0.5603	0.7448
Heart	0.7417	0.7417	0.7625	0.7250	0.6917	0.7792
Ionosphere	0.8491	0.8691	0.8691	0.8674	0.7783	0.9365
Sonar	0.8048	0.8523	0.8795	0.8795	0.8091	0.8341
Iris	0.8000	0.9667	0.9667	0.9667	0.9667	0.9667
Lung Cancer	0.6111	0.3333	0.3333	0.6111	0.1667	0.5000
Soybean	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
Glass	0.6063	0.5913	0.3786	0.5730	0.7071	0.5611
Ecoli	0.8548	0.8470	0.5881	0.7132	0.7000	0.8235
Yeast-SL	0.3593	0.6280	0.2326	0.5692	0.5524	0.4280
Ave-MAREC	0.7347	0.7488	0.6704	0.7519	0.6932	0.7574

and

Table 5. Testing macro-averaged precision (MAPRE) obtained for single-label data sets.

Data Set	LinearSVC	SVC-rbf	SVM	NN	RBF	RBF-ISCC
Liver Disorders	0.7179	0.6739	0.7036	0.6134	0.5594	0.7474
Heart	0.7766	0.7766	0.7917	0.7500	0.7029	0.7761
Ionosphere	0.9064	0.9162	0.9162	0.8827	0.8486	0.9169
Sonar	0.8221	0.8726	0.8822	0.8822	0.8091	0.8333
Iris	0.8889	0.9697	0.9697	0.9697	0.9697	0.9697
Lung Cancer	0.4111	0.1429	0.1111	0.6667	0.1111	0.5000
Soybean	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
Glass	0.6482	0.6799	0.5062	0.6333	0.7361	0.7250
Ecoli	0.9144	0.8898	0.5905	0.7027	0.7586	0.9139
Yeast-SL	0.5205	0.6989	0.2537	0.5183	0.4755	0.7499
Ave-MAPRE	0.7606	0.7621	0.6725	0.7619	0.6971	0.8132

show the MAREC and MAPRE values, respectively, obtained by different methods for all the data sets.

For these data sets, we set $\eta =0.1$ in RBF-ISCC. The other constants are shown in

Table 6. Pre-defined constants for RBF-ISCC for single-label data sets.

Data Set	α	β	λ		α	β	λ
Liver Disorders	4	2	0	Heart	4	8	0.1
Ionosphere	4	2	0	Sonar	4	8	0
Iris	4	4	0	Lung Cancer	6	2	0
Soybean	4	2	0	Glass	2	2	0.1
Ecoli	4	16	0	Yeast-SL	4	16	0

. As mentioned earlier, their values are determined in the training phase and are not changed in the testing phase.

6.2. Experiment 2—Multi-Label Data Sets

In this experiment, we show the performance of different methods on multi-label classification datasets. Exact match ratio (EMR), labeling F-measure (LFM), and Hamming loss (HL) are adopted for performance evaluation [57]. Let N_t be the number of patterns. EMR, LFM, and HL are defined as

$$\begin{gathered} \mathrm{EMR}=\frac{{{\displaystyle \sum }}_{i=1}^{N_t}I\left(y^{\left(i\right)},{\widehat{y}}^{\left(i\right)}\right)}{N_t} \\ \mathrm{LFM}=\frac{{{\displaystyle \sum }}_{i=1}^{N_t}\frac{2{{\displaystyle \sum }}_{h=1}^m\left(\frac{1+y_h^{\left(i\right)}}{2}\right)\left(\frac{1+{\widehat{y}}_h^{\left(i\right)}}{2}\right)}{{{\displaystyle \sum }}_{h=1}^m\left[\frac{1+y_h^{\left(i\right)}}{2}+\frac{1+{\widehat{y}}_h^{\left(i\right)}}{2}\right]}}{N_t} \\ \mathrm{HL}=\frac{{{\displaystyle \sum }}_{i=1}^{N_t}{H}_d\left(y^{\left(i\right)},{\widehat{y}}^{\left(i\right)}\right)}{m{N}_t} \end{gathered}$$

where $I\left(a,b\right)=1$ if a is identical to b and $I\left(a,b\right)=0$ otherwise, and $H_d\left(a,b\right)$ is the Hamming distance between a and b. Note that higher values in EMR and LFM, and lower value in HL, indicate a better classification. Several benchmark data sets, taken from the MULAN (Multi-Label Learning) library [30], are used in this experiment. The characteristics of these data sets are shown in

Table 7. Multi-label data sets for experiments.

Data Set	# Features	# Categories	# Patterns	Cardinality
Birds	260	19	645	1.014
Scene	294	6	2407	1.074
Emotions	72	6	593	1.869
Flags	19	7	194	3.392
Yeast-ML	103	14	2417	4.237

. A pattern may belong to more than one category in average, so the cardinality of a data set can be greater than 1. For instance, the cardinality of “Yeast-ML” is 4.237, indicating each pattern belonging to 4.237 categories in average.

Table 8. Testing exact match ratio (EMR) obtained for multi-label data sets.

Data Set	ML-KNN	MLPC	ML-SVC	ML-RBF	RBF-ISCC
Emotions	0.1832	0.2673	0.2030	0.2129	0.3020
Scene	0.4576	0.5518	0.3721	0.5418	0.5360
Yeast-ML	0.1045	0.0788	0.1368	0.0305	0.1526
Birds	0.4675	0.4675	0.4180	0.4892	0.5232
Flags	0.1538	0.1538	0.1385	0.1385	0.1846
Ave-EMR	0.2733	0.3038	0.2537	0.2826	0.3400

,

Table 9. Testing labeling F-measure (LFM) obtained for multi-label data sets.

Data Set	ML-KNN	MLPC	ML-SVC	ML-RBF	RBF-ISCC
Emotions	0.4812	0.6266	0.5716	0.4926	0.5893
Scene	0.4880	0.6734	0.5545	0.5924	0.5860
Yeast-ML	0.5633	0.5412	0.6024	0.4066	0.6203
Birds	0.4675	0.6225	0.5737	0.5372	0.5890
Flags	0.6329	0.6885	0.6830	0.6955	0.6827
Ave-LFM	0.5270	0.630	0.5970	0.5449	0.6135

and

Table 10. Testing Hamming loss (HL) obtained for multi-label data sets.

Data Set	ML-KNN	MLPC	ML-SVC	ML-RBF	RBF-ISCC
Emotions	0.2302	0.2137	0.2343	0.2195	0.2021
Scene	0.1130	0.1142	0.1568	0.0984	0.0994
Yeast-ML	0.2136	0.2605	0.2108	0.2711	0.2017
Birds	0.0510	0.0595	0.1154	0.0838	0.0422
Flags	0.3099	0.2659	0.2791	0.2835	0.2967
Ave-HL	0.1835	0.1828	0.1993	0.1913	0.1684

show the testing EMR, LFM, and HL obtained by different methods, including ML-KNN (K-nearest neighbors classifier for multi-label) [49], MLPC (Multi-layer Perceptron classifier) [64], ML-SVC (Support vector classifier for multi-label), ML-RBF (RBF network for multi-label) [50], and RBF-ISCC, for the data sets in Table 7. The codes of ML-KNN, MLPC, and ML-SVC are downloaded from the scikitlearn library, while the codes of ML-RBF is downloaded from the public domain: http://cse.seu.edu.cn/people/zhangml/resources.htm. For these data sets, we set $\eta =0.1$ in RBF-ISCC, except $\eta =1.2$ for “Scene”. The other constants are shown in

Table 11. Pre-defined constants for RBF-ISCC for multi-label data sets.

Data Set	α	β	λ		α	β	λ
Emotions	7	2	0.1	Scene	12	4	0
Yeast-ML	7	4	0	Birds	10	2	0
Flags	2	2	10

. Note that their values are determined in the training phase and are not changed in the testing phase.

The average metric values obtained by each method are shown in the bottom line of Table 8, Table 9 and Table 10. From these tables, we can see that RBF-ISCC performs the best in average EMR and average HL for these data sets.

Table 12. Ranking of different methods for multi-label data sets.

Data Set	ML-KNN	MLPC	ML-SVC	ML-RBF	RBF-ISCC
Ave-EMR	4	2	5	3	1
Ave-LFM	5	1	3	4	2
Ave-HL	3	2	5	4	1
Ave-Rank	4.00	1.67	4.33	3.67	1.33

ranks different methods based on the average results in Table 8, Table 9 and Table 10. Clearly, RBF-ISCC is ranked the first place.

6.3. Discussions

Our proposed approach includes two phases, network setup and parameter refinement. In the network setup phase, the iterative self-constructing clustering algorithm is used to produce J clusters. Assume that P₁ iterations is performed. In each iteration, each n-dimensional instance is compared against each of the J existing clusters. Therefore, the complexity of this phase is O(nNJP₁). In the parameter refinement phase, a hybrid learning, integrating the steepest descent and least squares methods, is applied to refine the parameters associated with the network. Let P₂ be the number of iterations performed in this phase. The complexity of the least squares method is O(NJ²), while the complexity of steepest descent is O(nNJ). Therefore, the complexity of the parameter refinement phase is O(nNJP₂+NJ²P₂). As a result, the total complexity of our approach is O(nNJP₁ + nNJP₂ + NJ²P₂). We feel that it may not be appropriate to compare the performance from the speed viewpoint among different methods. The programs of the other existing methods are taken from different websites. They might be implemented in different languages or with different optimizing mechanisms. For support-vector based classifiers, e.g., LinearSVC and SVC-rbf, the quadratic programming solver takes a complexity between O(nN²) and O(nN³) [11,59]. For NN with k hidden layers each having h neurons, the time complexity of backpropagation is O(nNmh^kP) where P is the number of iterations [60]. For KNN, the brute-force implementation takes a complexity of O(nN²). The computational time of each method is usually of the order of seconds for data sets of small or moderate size.

High-dimensional datasets may cause data sparsity and computational intractability, leading to low forecasting accuracy of classification algorithms. Two strategies, correlation analysis and dimensionality reduction, can be employed to reduce the dimensionality of a dataset. By correlation analysis, the correlation, e.g., Pearson product-moment correlation coefficient [65], for each pair of variables, is calculated. Variables are sorted out according to their relevance to the target to be predicted. Four levels of correlation, uncorrelated, weakly correlated, moderately correlated, and strongly correlated, are defined [66]. It was shown in [67] that by ignoring weakly correlated or uncorrelated variables, forecasting accuracy can be improved. Principal component analysis (PCA) [68] is a popular method for dimensionality reduction. PCA reduces the dimensionality of a dataset while retaining the variation present in the dataset up to the maximum extent. This is done by transforming the original variables to a new set of variables known as the principal components. Dimensionality reduction is done by choosing a small number of principal components. Some strategies were proposed to determine an appropriate number of principal components. One common strategy chooses g principal components so that g is as small as possible and the cumulative energy is above a certain threshold.

To address the scalability concern about N, two methods—sampling [69] and clustering [70]—can be adopted. We may take a random sample, with size N_s, from the N instances, such that N_s is smaller than N. We may apply clustering to reduce the training set size. The N instances are grouped into clusters and the obtained clusters are used as the representatives for training the model. Another approach through exploiting the parallel or distributed facility of the computing system has long been adopted to address the scalability problem [71]. The N instances are divided and dispatched to different CPUs or computing nodes which can run simultaneously to improve the training efficiency of the classification task.

7. Concluding Remarks

We have presented a novel approach of constructing RBF networks for solving supervised classification problems. An iterative self-constructing clustering algorithm is used to determine the number of nodes in the hidden layer. Basis functions are formed, and their centers and deviations are initialized to be the centers and deviations of the corresponding clusters. Then, the parameters of the network are refined with a hybrid learning strategy, involving the steepest descent backpropagation and least squares method. Hyperbolic tangent sigmoid functions and Tikhonov regularization are employed. As a result, optimized RBF networks are obtained. The proposed approach is applicable to construct RBF networks for solving both single-label and multi-label pattern classification problems. Experimental results have shown that the proposed approach can be used to solve classification tasks effectively.

Note that all the dimensions of a training pattern are treated to be equally important, i.e., no different weights are involved. It might be beneficial to incorporate a kind of weighting mechanism which suggests different weights in different dimensions [72]. Furthermore, the clustering algorithm may encounter difficulties finding useful clusters in high-dimensional data sets. Dimensionality reduction [65,68,69,70] may need to be applied in this case. We will take these as our future work.