Semi-Supervised Classification via Hypergraph Convolutional Extreme Learning Machine

Abstract

Extreme Learning Machine (ELM) is characterized by simplicity, generalization ability, and computational efficiency. However, previous ELMs fail to consider the inherent high-order relationship among data points, resulting in being powerless on structured data and poor robustness on noise data. This paper presents a novel semi-supervised ELM, termed Hypergraph Convolutional ELM (HGCELM), based on using hypergraph convolution to extend ELM into the non-Euclidean domain. The method inherits all the advantages from ELM, and consists of a random hypergraph convolutional layer followed by a hypergraph convolutional regression layer, enabling it to model complex intraclass variations. We show that the traditional ELM is a special case of the HGCELM model in the regular Euclidean domain. Extensive experimental results show that HGCELM remarkably outperforms eight competitive methods on 26 classification benchmarks.

1. Introduction

Extreme Learning Machine (ELM) [1,2] was developed as a simple but effective learning model for classification and regression problems. As it is a special form of random vector functional-link network (RVFL) [3], ELM suggests that the hidden layer parameters of a neural network play an important role but does not need update during training [4,5]. Inspired by this, a large number of ELM variants have been proposed and widely applied to biomedical data analysis [6], computer vision [7], system modeling and prediction [8,9], and so on.

The key to the classic ELM is random mapping [10,11]. Despite helpful, the random mapping often suffers from poor robustness due to its randomness. To remedy the drawback, a number of works have been devoted to seeking the optimal hidden parameters. Wu et al. [12] presented a multi-objective evolutionary ELM to jointly optimize the structural risk and empirical risk of ELM. Similarly, many popular heuristic search methods, including differential evolution [13], are adopted for this purpose. However, the heuristic search is often time-consuming. Alternatively, kernel ELM (KELM) [4,14] implicitly implements the ELM hidden mapping in the reproducing kernel Hilbert space, which usually results in more stable performance. Considering that the random mapping also provides good diversity, it allows us to design an ensemble of ELMs [5,15]. It has been proven to be useful to improve the robustness of single ELM. Another tendency to enhance ELM is to make ELM deeper [16,17]. Compared with popular deep learning models, e.g., Convolutional Neutral Networks (CNNs) [18], deep ELMs lack the potential to capture deep semantics from large-scale complex data.

The above-mentioned ELMs typically belong to supervised ELM, which desires adequate training samples for model training. For most general situations, where training samples are significantly limited, the auxiliary information of unlabeled data is surprisingly helpful to improve generalization ability [19,20]. To this end, Semi-supervised ELM (SS-ELM) [21] has drawn increasing attention. SS-ELM adopts a frequently used scheme, i.e., graph Laplacian regularization, to incorporate manifold structure. To combine multiple graphs into SS-ELM, Yi et al. [22] introduced an adaptive multiple graph regularized SS-ELM approach. These efforts have demonstrated that structured information is beneficial for ELM. Nonetheless, the simple graph can only reveal the pairwise relationship of data points, thus ignoring the high-order relationship. Moreover, these methods basically work in the Euclidean domain, leading to indirect and insufficient use of structured information.

Recently, Graph Neural Networks (GNNs) [23] is emerging as a powerful technique for generalizing neural networks on graph-structured data. Technically, GNNs capture the dependence of graphs via message passing between the nodes of graphs [24]. Graph Convolutional Network (GCN) [25] is one of the most popular GNN models for node-level tasks [26] and graph-level tasks [27]. It has been proven that GCN is equivalent to a fixed low-pass filter [28,29] followed by a linear classifier. Thus, it often suffers from the over-smoothing problem when model depth increased. More recently, some works have attempted to extending GCN into hypergraph scenarios [30,31] so that the high-order relationship of data can be embedded. Despite considerable success, current GCNs highly rely on arduous model update via gradient descent.

Depending on the observation that GCN is equivalent to a multilayer perceptron multiplied by an adjacent matrix, [32], Zhang et al. proposed a Graph Convolutional ELM (GCELM) for semi-supervised learning. As a pioneer work, GCELM follows the same pipeline as ELM but extends it into the non-Euclidean domain, enabling ELM to deal with graph-structured data directly. However, GCELM mainly focuses on the graph embedding of pairwise connections, resulting in failure to capture intraclass variations between data points. This motivates us to further develop an enhanced version of HGCELM. We refer to the method as Hypergraph Convolutional Extreme Learning Machine (HGCELM). Specifically, HGCELM contains a random hypergraph convolutional layer that produces hypergraph embeddings, and a hypergraph regression layer with a closed-form solution.

To sum up, the main contributions of this paper are two-fold:

• We propose a simple but effective hypergraph convolutional ELM, i.e., HGCELM, for semi-supervised classification. The HGCELM method not only inherits all the advantages from ELM but enables ELM to model the high-order relationship of data. The successful attempt signifies that structured information, especially high-order relationships, among data is important for ELM, which offers an alternative orientation for ELM representation learning.
• We have shown that the traditional ELMs are the special cases of HGCELM on the Euclidian data. We conduct extensive experiments on 26 popular datasets for semi-supervised classification task. Comparisons with state-of-the-art methods demonstrate that the proposed GCELM can achieve superior performance.

The rest of the paper is organized as follows. In Section 2, we briefly review hypergraph learning, ELMs, and graph neural networks. In Section 3, we systematically introduce the framework, formulation, and implementation of the proposed method. Experimental evaluations and comparisons are presented in Section 4, followed by the conclusions and future work given in Section 5.

2. Preliminary and Related Work

2.1. Notations

Through this paper, symbols for vectors are boldface lowercase italics (e.g., x), symbols for matrices are boldface uppercase roman letters (e.g., $\mathbf{X}$), and symbols for scalars are italics (e.g., $x_{ij}$). Let $\mathcal{X}={\left\{x_i\in {\mathbb{R}}^m;{\mathbf{y}}_i\in {\mathbb{R}}^C\right\}}_{i=1}^N$ be the sample set consisting of Nm-dimensional data points and C distinct classes, in which $x_i$ and ${\mathbf{y}}_i$ denote i-th data point and its one-hot target vector, respectively. For clarity, we denote the labeled data and unlabeled data with a subscript $\mathcal{L}$ and $\mathcal{U}$, e.g., ${\mathbf{X}}_{\mathcal{L}}$ and ${\mathbf{X}}_{\mathcal{U}}$ are the feature matrices for training and test, respectively. ${\mathbf{I}}_N$ signifies an identity matrix with the size of $N\times N$. The matrix Frobenius norm is defined as ${∥\mathbf{X}∥}_F={\left({\sum }_{ij}{\left|x_{ij}\right|}^2\right)}^{1/2}$. The main notations involved in this paper and corresponding definitions are given in

Table 1. Important notations used in this paper and their definitions.

Notation	Definition
N	The number of data points.
m	The number of features.
C	The number of classes.
$\mathbf{X}$	The feature matrix of dataset, ${\mathbb{R}}^{N\times m}$.
$\mathbf{Y}$	The label matrix with one-hot encoding, ${\mathbb{R}}^{N\times C}$.
$N_{\mathcal{T}}$	The number of labeled samples.
$\mathcal{T}$	The labeled set.
$\mathcal{U}$	The unlabeled set.
$\mathsf{\Theta }$	The hidden layer parameter matrix, ${\mathbb{R}}^{m\times L}$.
$\mathit{\beta }$	The output layer parameter matrix, ${\mathbb{R}}^{L\times C}$.
$\mathbf{Z}$	The matrix of latent representation, ${\mathbb{R}}^{N\times L}$.
${\mathbf{Z}}^{†}$	The Moore–Penrose generalized inverse of matrix $\mathbf{Z}$.
L	The number of hidden neurons.
$\mathcal{V}$	The set of vertices in the hypergraph.
$\mathcal{E}$	The set of hyperedges in the hypergraph.
$\mathbf{W}$	The diagonal matrix of the hyperedge weights, ${\mathbb{R}}^{\left|\mathcal{E}\right|\times \left|\mathcal{E}\right|}$.
$\mathcal{G}$	A hypergraph $\mathcal{G}=\left(\mathcal{V},\mathcal{E},\mathbf{W}\right)$.
$\mathbf{H}$	The incidence matrix of the hypergraph, ${\mathbb{R}}^{N\times \left|\mathcal{E}\right|}$.
$d\left(v\right)$	The degree of the vertex v.
$\delta \left(e\right)$	The degree of the hyperedge e.
${\mathbf{D}}_v$	The diagonal matrix of the vertex degrees, ${\mathbb{R}}^{N\times N}$.
${\mathbf{D}}_e$	The diagonal matrix of the hyperedge degrees, ${\mathbb{R}}^{\left|\mathcal{E}\right|\times \left|\mathcal{E}\right|}$.
$\mathbf{L}$	The hypergraph Laplacian matrix, ${\mathbb{R}}^{N\times N}$.

.

2.2. Hypergraph Preliminary

Hypergraph is the generalization of the simple graph, in which an edge can join any number of vertices. We refer to edges in a hypergraph as hyperedges. By contrast, we denote the regular graph in which each edge only connects two vertexes as a simple graph. A visual comparison between the simple graph and the hypergraph is illustrated in Figure 1. It can be seen that hypergraph can reveal a more complex data relationship, which is superior to the simple graph. For a learning task, hypergraph is usually used to represent the high-order intraclass variations among data points.

Formally, let $\mathcal{G}=\left(\mathcal{V},\mathcal{E},\mathcal{W}\right)$ be a hypergraph composed of a vertex set $v\in \mathcal{V}$ with the size of N, a hyperedge set $e\in \mathcal{E}$ with the size $\left|\mathcal{E}\right|$, and a weight set of hyperedge $\mathcal{W}$, where the weight of hyperedge e is indicated as $w\left(e\right)$. A hypergraph is often described as an incidence matrix $\mathbf{H}\in {\mathbb{R}}^{N\times \left|\mathcal{E}\right|}$ whose elements indicate whether a vertex joins a corresponding hyperedge (e.g., Figure 1b). Mathematically, the incidence matrix is defined by

$$h\left(v,e\right)=\left\{\begin{array}{cc}1\hfill & v\in e\hfill \\ 0\hfill & v\notin e.\hfill \end{array}\right.$$

(1)

The degree of a vertex v and the degree of a hyperedge e are given as follows, respectively

$$d\left(v\right)=\sum _{e\in \mathcal{E}}w\left(e\right)h\left(v,e\right),$$

(2)

$$\delta \left(e\right)=\sum _{v\in \mathcal{V}}h\left(v,e\right),$$

(3)

By analogy with the simple graph, spectral analysis can be used as an efficient tool for the analysis of hypergraph. The normalized hypergraph Laplacian matrix [33,34] is calculated by

$$\mathbf{L}=\mathbf{I}-{\mathbf{D}}_v^{-1/2}\mathbf{HW}{\mathbf{D}}_e^{-1}{\mathbf{H}}^T{\mathbf{D}}_v^{-1/2}.$$

(4)

For a semi-supervised learning task, hypergraph is usually used by incorporating with an empirical error [35], as follows

$$\underset{\mathbf{F}}{argmin}{\mathcal{R}}_{emp}\left(\mathbf{F}\right)+\lambda \mathrm{tr}\left({\mathbf{F}}^T\mathbf{LF}\right),$$

(5)

where ${\mathcal{R}}_{emp}\left(\mathbf{F}\right)$ denotes the empirical error term over a problem-dependent prediction $\mathbf{F}$.

2.3. ELMs

The basic ELM can be interpreted as two components, i.e., random hidden mapping and ridge regression classifier. Formally, ELM’s hidden layer can be expressed as

$$\mathbf{Z}=\sigma \left(\mathbf{X}\mathsf{\Theta }+\mathbf{b}\right),$$

(6)

where $\mathbf{Z}$ is the hidden layer output matrix parameterized by hidden weight matrix $\mathsf{\Theta }$ and bias vector $\mathbf{b}$, $\sigma$ denotes a nonlinear activation function such as Sigmoid. In the second stage, ELM computes prediction by

$$\mathbf{Y}=\mathbf{Z}\mathit{\beta }.$$

(7)

Here, $\mathit{\beta }$ is the output weight matrix. Since $\mathbf{Z}$ is known to the output layer, Equation (7) essentially is a least-squares optimization problem and can be solved as

$$\mathit{\beta }={\mathbf{Z}}^{†}{\mathbf{Y}}_{\mathcal{T}}.$$

(8)

ELM avoids iterative parameters tuning, thus significantly faster than gradient descent-based neural networks. SS-ELM is the semi-supervised version of ELM by introducing a graph Laplacian regularization term [21]. Its formulation is given by

$$\underset{\mathit{\beta }}{argmin}∥{\mathbf{Z}}_{\mathcal{T}}\mathit{\beta }-{\mathbf{Y}}_{\mathcal{T}}∥{}_F^2+\lambda \mathrm{tr}\left({\mathit{\beta }}^T{\mathbf{Z}}^T\mathbf{L}\mathbf{Z}\mathit{\beta }\right).$$

(9)

SS-ELM also has a closed-form solution. It should be noted that the graph structure information is considered in SS-ELM but it essentially works in the regular Euclidean domain.

2.4. GCNs

There is an increasing interest in generalizing convolutions to the graph domain [23]. The recent development of GNNs allows us to efficiently approximate convolution on graph-structured data. GNNs can typically divide into two categories [23,28]: spectral convolutions [36], which perform convolution by transforming node representations into the spectral domain using the graph Fourier transform or its extensions, and spatial convolutions [37], which perform convolution by sampling from neighborhood signals.

In [25], Kift et al. developed a Graph Convolutional Network (GCN) by simplifying the spectral convolution with the 1th-order Chebyshev polynomials and setting the largest eigenvalue of the normalized graph Laplacian to be 2. Formally, GCN defines spectral convolution over a graph as follows

$$\mathbf{Z}=\sigma \left({\tilde{\mathbf{D}}}^{-1/2}\tilde{\mathbf{A}}{\tilde{\mathbf{D}}}^{-1/2}\mathbf{X}\mathsf{\Theta }\right).$$

(10)

Here, $\tilde{\mathbf{A}}={\mathbf{I}}_N+\mathbf{A}$ is the so-called augmented normalized adjacency, $\tilde{\mathbf{D}}$ is given by ${\tilde{D}}_{ii}={\sum }_j\widetilde{A_{ij}}$. However, the current GNNs highly rely on gradient descend optimizers, which is often time-consuming and prone to suffering from a locally optimal solution. To overcome the shortcomings, Zhang et al. [32] proposed a randomization-based GCN (i.e., GCELM) by combining the advantages of ELM with GCN. Instead of updating all trainable parameters, GCELM employs a random graph convolutional layer and keeps it fixed. Thus, this allows GCELM to calculate the closed-form solution for the training phase, and further resulting in a faster learning speed.

3. HGCELM

In Figure 2, we provide an illustration of the proposed HGCELM framework. The framework first constructs a hypergraph from the given dataset, and then feeds it into the HGCELM model consisting of a random hypergraph convolutional layer and a hypergraph convolutional regression layer. The details for HGCELM are introduced as follows.

3.1. Hypergraph Construction

We represent the high-order relationship of a dataset by constructing a hypergraph $\mathcal{G}$. For this purpose, each data point $x_i$ is treated as a vertex v, and furthermore, we consider that $x_i$ is the center vertex and its k nearest neighbors are associated with the hyperedge e. As a result, each hyperedge connects $k+1$ vertices. The incidence matrix of the hypergraph is defined by

$$h\left(v_i,e_j\right)=\left\{\begin{array}{cc}1\hfill & x_j\in {\mathcal{N}}_k\left(x_i\right)\hfill \\ 0\hfill & otherwise\hfill \end{array}\right..$$

(11)

The degree of the vertex set $\mathcal{V}$ and the degree of the hyperedge set $\mathcal{E}$ can be expressed in diagonal matrix forms, i.e., ${\mathbf{D}}_v$ and ${\mathbf{D}}_e$. There are various ways to assign the weights for hyperedges, such as the sum of similarities of vertices within a hyperedge [34]. In this paper, we view all hyperedges as equal weights and thus can be indicated as an identity matrix $\mathbf{I}\in {\mathbb{R}}^{\left|\mathcal{E}\right|\times \left|\mathcal{E}\right|}$. The samples belonging to the same class often have a higher probability of being assigned to ${\mathcal{N}}_k$. Therefore, it is reasonable to use hypergraph to describe the intraclass variations of data.

3.2. Random Hypergraph Convolution

Inspired by the previous GCELM [32], we propose a novel ELM mapping, called random hypergraph convolution (RGC), to incorporate the high-order relationship among data into the feature mapping. We define the random hypergraph convolution following that proposed in [30,31], except that ours does not need to be updated iteratively. It is expressed as follows:

$$\mathbf{Z}=\sigma \left(\mathbf{S}\mathbf{X}\mathsf{\Theta }\right),$$

(12)

where $\mathbf{S}={\mathbf{D}}_v^{-1/2}\mathbf{H}{\mathbf{D}}_e^{-1}{\mathbf{H}}^{-1}{\mathbf{D}}_v^{-1/2}$ referring to as augmented normalized incidence matrix by imposing a symmetric normalization. It should be noted that $\mathbf{S}$ incorporates structured information of data and can be precomputed during implementation. Thus, the random hypergraph convolution enables ELM to embed the high-order information. Following the ELM’s theory, the filter parameters of the random hypergraph convolution, $\mathsf{\Theta }$, is randomly generated under a specific probability distribution, e.g., Gaussian distribution ${\mathsf{\Theta }}_{ij}\sim \mathbb{N}\left(0,1\right)$.

3.3. Hypergraph Convolutional Regression

Based on the hidden hypergraph embedding $\mathbf{Z}$, we use a hypergraph convolutional regression layer to predict labels. Formally, the layer can be written as:

$$\mathbf{Y}=\mathbf{S}\mathbf{Z}\mathit{\beta }.$$

(13)

To solve $\mathit{\beta }$, the above equation can be rewritten as the following ridge regression problem:

$$\underset{{\mathit{\beta }}^{*}}{argmin}{∥\mathit{\beta }∥}_F,s.t.,\tilde{\mathbf{Y}}=\mathbf{S}\mathbf{Z}\mathit{\beta }.$$

(14)

Here, $\tilde{\mathbf{Y}}=\left[{\mathbf{Y}}_{\mathcal{T}};{\mathbf{Y}}_{\mathcal{U}}\right]$ is an augmented training target matrix. Since ${\mathbf{Y}}_{\mathcal{U}}$ is invisible during training stage, thus it is set to be a 0 matrix. Let $\mathbf{M}$ be a diagonal mask matrix with its first $N_{\mathcal{T}}$ diagonal elements $M_{ii}=1$ and the rest equal to 0. We further rewrite Equation (14) as

$$\underset{{\mathit{\beta }}^{*}}{argmin}\frac{1}{2}{∥\mathbf{M}\left(\tilde{\mathbf{Y}}-\mathbf{S}\mathbf{Z}\mathit{\beta }\right)∥}_F^2+\frac{\lambda }{2}{∥\mathit{\beta }∥}_F^2.$$

(15)

It is easy to prove that Equation (15) holds a optimal solution and its closed form is expressed

$${\mathit{\beta }}^{*}={\left({\mathbf{Z}}^T{\mathbf{S}}^T\mathbf{M}\mathbf{S}\mathbf{Z}+\lambda \mathbf{I}\right)}^{-1}{\mathbf{Z}}^T{\mathbf{S}}^T\mathbf{M}\tilde{\mathbf{Y}}.$$

(16)

The labels of the unlabeled data points can be determined depended on the obtained ${\mathit{\beta }}^{*}$, which is given by

$${\overline{\mathbf{Y}}}_{\mathcal{U}}={\left[\mathbf{S}\mathbf{Z}\right]}_{\mathcal{U}}{\mathit{\beta }}^{*}.$$

(17)

We show the overall learning steps of the proposed HGCELM in Algorithm 1. Since there is no iteration in HGCELM, it would be computationally efficient for training such model.

Algorithm 1: HGCELM

Input: Dataset $\mathbf{X}$, training labels ${\mathbf{Y}}_{\mathcal{T}}$, hyper-parameters $\lambda$ and L. 1 Construct hypergraph $\mathcal{G}=\left(\mathcal{V},\mathcal{E},\mathcal{W}\right)$ by Equation (11); 2 Generate $\mathsf{\Theta }$ using the standard normal distribution; 3 Calculate random hypergraph embedding: $\mathbf{Z}=\sigma \left(\mathbf{S}\mathbf{X}\mathsf{\Theta }\right)$; 4 Solve hypergraph conv regression: ${\mathit{\beta }}^{*}={\left({\mathbf{Z}}^T{\mathbf{S}}^T\mathbf{M}\mathbf{S}\mathbf{Z}+\lambda \mathbf{I}\right)}^{-1}{\mathbf{Z}}^T{\mathbf{S}}^T\mathbf{M}\tilde{\mathbf{Y}}$; 5 Predict test levels by Equation (17);  Output: ${\overline{\mathbf{Y}}}_{\mathcal{U}}$.

3.4. Computation Complexity and Connection to Existing Methods

We give a theoretical analysis for the computation complexity of our method. Ignoring the hypergraph generation procedure and assuming $L\ll N$, the time complexity of HGCELM is approximately $O\left(N^3\right)$. Compared with ELM, HGCELM has a higher time complexity, which is determined by the hypergraph convolution. Fortunately, the augmented normalized incidence matrix can be precomputed and is sparse. Thus, the hypergraph embedding can be efficiently implemented. HGCELM is more efficient than GCN since it does not use iteration.

Our HGCELM is closely related to the classic ELMs. First, HGCELM maintains the advantages of ELMs, i.e., fast learning speed achieved by a closed-form solution. Second, our HGCELM will degenerate to be the classic ELM by casting away $\mathbf{S}$. Consequently, the classic ELM is a special case of the HGCELM model in the regular Euclidean domain. Furthermore, our HGCELM remedies the drawback that traditional ELMs cannot deal with structured data.

4. Results and Discussion

In this section, we conduct experiments to verify the effectiveness of the proposed HGCELM.

4.1. Experimental Configurations

4.1.1. Datasets

We evaluate our method on 26 widely used classification datasets taken from University of California at Irvine (UCI) repository. For each dataset, we scale its feature values into $\left[0,1\right]$ using a min-max standardization method and randomly choose 5 samples per class as the training set and the rest as the test set. The details of these datasets are reported in

Table 2. Details of the benchmark datasets.

Dataset	#Classes	#Instance	#Feature	#Train	#Test	Dataset	#Classes	#Instance	#Feature	#Train	#Test
austra	2	680	14	10	680	weather	2	22	4	10	12
australian	2	690	14	70	620	Wine	3	178	13	19	159
breast	2	277	9	29	248	X8D5K	5	1000	8	100	900
cleve	2	296	13	30	266	zoo	7	101	16	13	88
diabetes	2	768	8	78	690	cloud	2	1024	10	103	921
dnatest	3	1186	180	120	1066	bupa	2	345	6	10	335
german	2	1000	24	100	900	air	3	359	64	37	322
heart	2	270	13	27	243	segmentation	7	210	18	21	189
ionosphere	2	351	32	10	341	pima In. D.	2	768	8	10	758
iris	3	150	4	15	135	Xinp	3	178	13	19	159
sonar	2	208	60	10	198	wdbc	2	569	30	58	511
vote	2	435	16	10	425	ecoli label	2	335	2	10	325
WBC	2	683	9	10	673	appendicitis	2	106	7	12	94

.

4.1.2. Baseline Methods

We compare our method with eight baselines including the basic ELM [2], KELM [38], SS-ELM [21], Transductive Support Vector Machine (TSVM) [19], Self-training Semi-supervised ELM (ST-ELM) [39], Laplacian Support Vector Machine (LapSVM) [20], GCELM [32], and GCN [25]. For a fair comparison with previous works, we set 50 hidden neurons for these methods that contain hidden layer, i.e., GCELM, ELM, SS-ELM, ST-ELM, and GCN. The hyperparameter settings of these methods are provided in

Table 3. Hyperparameter settings of different methods.

Method	Hyper-Parameters
HGCELM (Ours)	$L=50,\lambda ={10}^{-3}$
GCELM [32]	$L=50,\lambda ={10}^{-3}$
ELM [2]	$L=50,\lambda ={10}^{-5}$
KELM [38]	$\lambda ={10}^{-5}$
SS-ELM [21]	$L=50,{\lambda }_1={10}^{-5},{\lambda }_2={10}^{-2}$
TSVM [19]	$C={10}^3,C^{*}=1.5,kernel=RBF$
ST-ELM [39]	$L=50,{\lambda }_1={\lambda }_2={10}^{-5}$
LapSVM [20]	${\lambda }_1=0.001,{\lambda }_2=0.05$
GCN [25]	$L=50,lr=0.02,epoch=100$

. We implement all the methods with Python 3.5 running on an Intel i5-6500 3.20 GHz CPU with 8.00 GB RAM.

4.2. Qualitative Study

4.2.1. Comparisons with Baselines

Table 4. Experimental results of different algorithms on 26 different datasets (MEAN ± STD, preferably in BOLD).

Data Sets	HGCELM	GCELM	SS-ELM	ELM	KELM	TSVM	ST-ELM	LapSVM	GCN
austra	82.57 ± 3.72	80.78 ± 2.99	76.24 ± 8.14	73.32 ± 10.48	79.47 ± 4.12	77.43 ± 10.59	75.17 ± 9.99	55.70 ± 0.21	71.38 ± 8.36
australian	81.90 ± 4.73	78.46 ± 4.82	78.10 ± 6.24	71.35 ± 11.25	79.32 ± 6.11	75.38 ± 10.81	78.57 ± 9.56	75.79 ± 6.61	75.36 ± 7.56
breast	66.52 ± 10.39	61.61 ± 8.61	55.30 ± 7.67	52.00 ± 7.38	55.56 ± 8.97	55.77 ± 6.78	53.18 ± 10.68	71.54 ± 0.00	60.17 ± 6.18
cleve	74.51 ± 2.09	72.90 ± 6.42	72.88 ± 5.30	68.95 ± 7.31	72.38 ± 3.79	72.80 ± 5.70	68.85 ± 8.30	54.20 ± 0.00	70.17 ± 6.14
diabetes	69.22 ± 3.75	70.10 ± 3.09	66.27 ± 8.92	57.31 ± 8.51	65.71 ± 5.64	64.97 ± 5.26	58.43 ± 8.56	65.30 ± 0.06	62.93 ± 6.62
dnatest	78.55 ± 2.34	67.87 ± 5.03	51.60 ± 4.63	50.77 ± 4.79	50.81 ± 4.26	60.99 ± 2.94	42.86 ± 8.51	48.27 ± 4.10	59.33 ± 3.95
german	67.38 ± 2.93	67.10 ± 10.01	54.48 ± 5.35	57.95 ± 9.42	51.07 ± 7.48	59.52 ± 6.89	59.64 ± 7.63	70.21 ± 0.03	59.54 ± 5.32
heart	78.50 ± 2.83	76.53 ± 3.63	72.54 ± 6.06	69.15 ± 9.07	73.25 ± 8.99	70.23 ± 7.77	67.21 ± 8.62	55.77 ± 0.00	70.00 ± 8.58
ionosphere	90.32 ± 0.00	81.07 ± 7.51	75.98 ± 4.31	74.65 ± 8.96	75.57 ± 6.18	77.05 ± 5.43	77.20 ± 5.68	64.52 ± 0.00	71.32 ± 8.21
iris	95.56 ± 1.72	94.32 ± 3.20	80.04 ± 4.79	80.70 ± 10.76	93.04 ± 3.88	93.00 ± 3.70	81.04 ± 8.36	92.70 ± 2.58	91.19 ± 3.61
sonar	67.93 ± 4.60	67.44 ± 6.64	63.28 ± 5.10	64.66 ± 6.37	62.56 ± 5.37	66.41 ± 5.27	62.21 ± 5.10	46.46 ± 0.00	67.22 ± 7.09
vote	90.12 ± 2.60	84.19 ± 5.72	83.75 ± 3.49	84.39 ± 6.17	87.98 ± 1.97	87.97 ± 3.59	87.65 ± 6.21	38.35 ± 0.00	86.62 ± 3.87
WBC	97.21 ± 0.27	96.53 ± 0.74	92.30 ± 1.83	89.35 ± 5.63	95.30 ± 2.78	95.40 ± 1.90	92.14 ± 3.50	65.26 ± 0.07	95.93 ± 2.28
weather	84.17 ± 6.92	76.94 ± 14.38	69.72 ± 15.14	73.61 ± 12.74	69.17 ± 11.21	76.39 ± 14.29	70.00 ± 15.00	41.67 ± 0.00	50.83 ± 11.66
Wine	96.38 ± 1.38	91.25 ± 2.28	89.37 ± 3.33	79.75 ± 6.72	90.31 ± 3.92	90.51 ± 3.34	79.59 ± 7.68	88.45 ± 1.60	90.84 ± 2.79
X8D5K	100.00 ± 0.00	100.00 ± 0.00	100.00 ± 0.00	96.93 ± 3.07	99.96 ± 0.06	99.98 ± 0.05	99.93 ± 0.21	100.00 ± 0.00	100.00 ± 0.00
zoo	100.00 ± 0.00	99.22 ± 1.32	98.02 ± 1.45	98.18 ± 1.51	98.65 ± 1.60	99.48 ± 1.09	99.69 ± 0.94	99.95 ± 0.28	100.00 ± 0.00
cloud	90.38 ± 2.92	90.18 ± 3.68	89.92 ± 3.22	73.38 ± 10.32	88.48 ± 4.36	89.71 ± 4.65	75.95 ± 9.54	81.32 ± 4.86	88.10 ± 3.90
bupa	57.37 ± 2.72	56.33 ± 6.93	52.55 ± 4.01	54.07 ± 6.63	56.15 ± 4.98	55.95 ± 5.62	55.41 ± 5.85	41.79 ± 0.00	51.72 ± 3.71
air	81.72 ± 6.33	71.30 ± 6.25	66.52 ± 4.50	67.44 ± 5.83	70.89 ± 4.18	70.68 ± 6.38	69.53 ± 7.55	72.49 ± 7.63	79.65 ± 5.78
segmentation	84.51 ± 2.00	83.22 ± 3.14	78.91 ± 3.42	65.71 ± 7.74	83.01 ± 3.22	81.85 ± 4.53	67.05 ± 8.36	80.84 ± 3.51	80.74 ± 3.61
pima In. D.	68.97 ± 3.19	68.45 ± 5.55	67.28 ± 7.67	58.43 ± 7.06	63.52 ± 6.78	64.84 ± 8.34	62.35 ± 5.65	35.29 ± 1.03	65.47 ± 4.70
Xinp	97.33 ± 0.85	93.48 ± 1.95	93.04 ± 2.31	80.31 ± 6.64	92.76 ± 3.00	93.90 ± 1.76	81.56 ± 8.59	92.45 ± 1.32	94.97 ± 1.82
wdbc	94.44 ± 1.42	92.13 ± 0.16	88.94 ± 4.59	81.20 ± 8.63	91.72 ± 4.20	91.15 ± 4.02	89.51 ± 5.39	62.97 ± 0.00	90.86 ± 2.44
ecoli_label	67.45 ± 1.78	64.46 ± 7.16	67.09 ± 2.29	57.92 ± 11.95	61.17 ± 12.58	58.92 ± 13.97	63.28 ± 9.71	61.80 ± 8.44	63.14 ± 8.21
appendicitis	86.98 ± 2.25	85.83 ± 2.80	73.33 ± 14.10	58.12 ± 11.00	69.06 ± 12.88	63.70 ± 15.29	57.86 ± 15.23	16.67 ± 0.00	76.15 ± 10.68
Average	82.69 ± 2.84	80.4 ± 4.61	75.64 ± 5.14	71.26 ± 7.78	76.9 ± 5.29	77.53 ± 5.94	72.62 ± 7.68	65.9 ± 1.57	76.63 ± 5.12
W/T/L	–	24/1/1	25/1/0	26/0/0	26/0/0	26/0/0	26/0/0	24/1/1	24/2/0

reports the test accuracy of different methods over the 26 datasets. Each method is evaluated for 30 independent runs. At the bottom of the table, two summative metrics are calculated, that is, an arithmetic mean of test accuracy over the 26 datasets and a statistical value for how many datasets our HGCELM win/tie/loss (W/T/L) than a competitor. On the basis of these results, we conclude that the proposed HGCELM method consistently outperforms the other baselines. Compared with GCELM, HGCELM achieves $2.29\%$ improvement in terms of the averaged accuracy with a lower standard deviation. It demonstrates that hypergraph is more helpful for semi-supervised learning. This is because the hypergraph convolution aims to embed the intraclass variations, while the graph convolution focuses only on the pairwise relationship. There is also a significant improvement comparing HCGELM against the classic semi-supervised methods (i.e., SS-ELM, TSVM, ST-ELM, and LapSVM). Specifically, our method wins on 25, 26, 26, and 24 datasets, respectively. Notably, ELM and KELM are purely supervised, thus their classification accuracy is lower than most semi-supervised methods caused by the limited training samples. Despite being optimized, GCN cannot achieve better test accuracy than our HGCELM. It further signifies the effectiveness and reliability of our proposal.

4.2.2. Performance with Varying Training Size

To investigate the performance of HGCELM, we visualize the test accuracy under varying training sizes. In this experiment, we increase the training size from $5\%$ to $50\%$ and present the corresponding results of 30 evaluations with a box plot. As shown in Figure 3, the test accuracy of HGCELM tends to increase when the training sample increased. Meanwhile, HGCELM becomes more robust and more stable. When the training size is larger than $25\%$, HGCELM’s accuracy closes to its highest accuracy and tends to be convergent. For the Iris dataset, HGCELM can achieve a remarkable accuracy (higher than $90\%$) using only 5 samples per class ($10\%$). It means that HGCELM has the ability to explore and make better use of the useful information from unlabeled data.

4.2.3. Analysis on Decision Boundaries

To provide an intuitive understanding of the superiority of our HGCELM, we visualize the decision boundaries of ELM, SS-ELM, GCELM, and HGCELM. In Figure 4, we synthesize three types of representative data distributions, i.e., linear separable data (the first row), linear inseparable data with half circles (the second row), and linear inseparable data with concentric circles (the third row), each of which contains 100 samples with two classes. The complexity of these datasets gradually increases from top to bottom. In this experiment, we select 10 samples for training and initialize each classifier with 10 hidden neurons. Owing to the simplicity, all the four classifiers can correctly find a reliable decision boundary on the linear separable dataset. However, for the first linear inseparable dataset, semi-supervised methods (i.e., HGCELM, GCELM, and SS-ELM) can obtain more generalized decision boundaries than the supervised method (i.e., ELM). However, spectral convolution-based methods (HGCELM and GCELM) are superior to the graph Laplacian-based method (SS-ELM) in terms of decision boundary, as well as test accuracy. From the second linear inseparable dataset, HGCELM can accurately classify the data points with a better decision boundary, while the other three methods show relatively poorer ones. In particular, ELM fails to work on this dataset under the same experiment settings. This is because the ELM cannot use the unlabeled samples, making it more likely to overfitting the limited training set. Although SS-ELM shows same accuracy as HGCELM, its decision boundary cannot separate two circles. We can naturally conclude that HGCELM has a better generalization ability than the other methods. The ability is benefited from the fact that the high-order relationship among the whole data points can improve the quality of decision making. It should be noticed that due to the geometric property of data distribution, the Euclidean-distance-based graph construction strategy would result in an inappropriate graph structure on the linear inseparable datasets. Therefore, a small neighbor size is desired when constructing a graph or hypergraph for these datasets.

4.3. Parameter Sensitivity Study

Two parameter sensitivity studies are carried out to further explore the robustness of the proposed method.

4.3.1. Impact of Hidden Neurons

Figure 5 shows the impact of different hidden neurons. We compare our method with five baselines that require setting hidden neurons. As seen from Figure 5a–d, all the compared methods can achieve better performance when using more hidden neurons on all the datasets. Nevertheless, our proposed HGCELM enjoys a more competitive edge than the other methods. In particular, when our method uses more than 20 hidden neurons. Notice that GCN can guarantee relatively better accuracy than other competitors when using small hidden neurons. This is because the filter parameters of GCN are fully optimized. By contrast, ELMs require larger hidden neurons, which is repeatedly demonstrated by many previous works. From Figure 5c,d, the classic ELM suffers from overfitting (accuracy declined with hidden neurons) caused by the inadequate training data. The problem is overcome in other semi-supervised ELMs including our HGCELM.

4.3.2. Impact of $\lambda$ and k

In this experiment, we further explore the impact of the other two important hyper-parameters, i.e., $\lambda$ and k. Figure 6a–d shows the results by grid searching. Here, we set $\lambda =\left\{{10}^{-7},{10}^{-6},\cdots ,{10}^2\right\}$ and $k=\left\{3,6,\cdots ,30\right\}$, respectively. We can observe two tendencies from the results. First, to obtain a better performance, the regularization coefficient $\lambda$ should be set to be less than 1. Theoretically, using a larger $\lambda$ will punish HGCELM to be a compact model but also resulting in an increased risk of under-fitting the training data. Second, a larger neighbor size k will guarantee a better classification accuracy in general. However, there will exist noise samples in hyperedges if k becomes too large. We provide the suitable ranges of the two parameters as $\lambda \in \left[{10}^{-3},{10}^{-1}\right]$ and $k\in \left[5,10\right]$.

5. Conclusions

We have proposed a novel ELM called HGCELM for semi-supervised classification. The idea behind HGCELM is to combine hypergraph convolution with ELM so that embedding the high-order relationship of data. The resulting model extends ELM into the non-Euclidian domain and endows ELM with the capability of modeling structure data. The proposed HGCELM is characterized by a very light computational burden and good generalization ability, making it easy to implement and apply in practice. Extensive experiments on 26 datasets demonstrate that HGCELM is superior to many existing methods. The successful attempt clues a promising avenue for designing randomized neural networks and graph neural networks.