EGNAS: Efficient Graph Neural Architecture Search Through Evolutionary Algorithm

Abstract

The primary objective of our research is to enhance the efficiency and effectiveness of Neural Architecture Search (NAS) with regard to Graph Neural Networks (GNNs). GNNs have emerged as powerful tools for learning from unstructured network data, compensating for several known limitations of Convolutional Neural Networks (CNNs). However, the automatic search for optimal GNN architectures has seen little progressive advancement so far. To address this gap, we introduce the Efficient Graph Neural Architecture Search (EGNAS), a method that leverages the advantages of evolutionary search strategies. EGNAS incorporates inherited parameter sharing, allowing offspring to inherit parameters from their parents, and utilizes half epochs to improve optimization stability. In addition, EGNAS employs a combined evolutionary search, which explores both the model structure and the hyperparameters within a large search space, resulting in improved performance. Our experimental results demonstrate that EGNAS outperforms state-of-the-art methods in node classification tasks on the Cora, Citeseer, and PubMed datasets while maintaining a high degree of computational efficiency. In particular, EGNAS is the fastest GNN architecture search method in terms of search time, particularly when compared to precedently suggested evolutionary search strategies, delivering performance up to 40 times faster.

1. Introduction

Graph Neural Networks (GNNs) have emerged as a crucial methodology, in areas such as social networks, transportation, and bioinformatics, where they tackle the intricacies of unstructured network data [1,2,3,4]. However, the manual construction of GNN models is a task that demands extensive expertise and time, presenting a significant challenge. Addressing this challenge, Neural Architecture Search (NAS) introduces an innovative approach, employing optimization algorithms to automate the identification of an optimal model [5,6,7]. The fusion of GNN with NAS has also led to state-of-the-art performance, surpassing manually designed architectures [8,9]. While NAS offers the convenience of automation, its significant search times can paradoxically lead to reduced efficiency. This paper aims to overcome the limitations of NAS, focusing on efficiently identifying high-performing GNN models.

The integration of model structure and hyperparameter searches, broadening the exploration scope, has been increasingly recognized as pivotal in enhancing Graph Neural Network performance. Furthermore, this method enhances the automation efficiency in NAS by obviating the necessity for an independent hyperparameter search following the determination of the optimal structure. This comprehensive approach, exemplified by Genetic-GNN, has ventured into this integrated search space, yielding outcomes that rival the capabilities of state-of-the-art models [10]. However, the method that employs separate evolutionary searches for structure and hyperparameters results in an extended duration of the search process. Consequently, a transformation in the search methodology is required, one that is tailored to the expanded scope of the search space, to ensure an efficient and effective exploration process.

Training models from scratch is a notable challenge in the process of GNN model discovery due to the extensive time required. The concept of parameter sharing emerges as a solution to this, allowing for the reuse of previously trained parameters, thereby obviating the need for ground-up training of each new model [11,12]. AGNN successfully reduces the number of epochs needed for training new models through parameter sharing, effectively cutting down the time cost [13]. However, this efficiency gain introduces a reduction in the accuracy of the final models. A primary cause of this limitation is weight coupling, a phenomenon that occurs when the same weights are frequently reused across different parts of a model. This coupling can lead to the repeated selection of high-performing structures, potentially reducing the diversity of representations. Furthermore, it poses a critical challenge in parameter sharing approaches, as it increases the risk of overfitting, particularly when excessive training epochs are applied. This over-reliance on shared weights can undermine the generalization capability of the model [14,15].

To this end, we introduce the Efficient Graph Neural Architecture Search (EGNAS) algorithm, featuring two novel methodologies: combined evolutionary search and inherited parameter sharing with half epochs. The combined evolutionary search integrates structure and hyperparameter searches, using crossover and mutation techniques to discover a broader range of diverse architectures and enhance the speed of search. Inherited parameter sharing, alongside the half epochs approach, addresses weight coupling issues by reducing redundant parameter usage. As a result, EGNAS not only achieved state-of-the-art accuracy in node classification tasks but also significantly reduced computational cost compared to existing methods. Furthermore, its efficiency and scalability make it a practical solution for real-world applications across domains such as social networks, transportation, and bioinformatics.

2. Related Works

2.1. Evolutionary Neural Architecture Search

Neural Architecture Search (NAS) is an automated method for designing Convolutional Neural Networks (CNNs) [16]. In NAS, Evolutionary Algorithm (EA), Reinforcement Learning (RL), and differentiable architecture search are commonly used as search strategies [17,18,19].

The Evolutionary Algorithm is an optimization technique inspired by natural selection and genetic principles. It iteratively evolves a population of candidate solutions to find the best possible answer for a given problem [20]. It is beneficial in automating architecture exploration and hyperparameter search by evaluating multiple candidates simultaneously [21]. Its ability to explore extensive search spaces allows it to effectively discover optimal solutions for deep learning tasks, making it applicable in various domains, including image classification, speech recognition, emotion recognition, and others [22,23,24]. Moreover, the evolutionary method is well-suited for addressing multi-objective optimization problems to find trade-off solutions and solve complex optimization challenges [25,26].

2.2. Graph Neural Network Architecture Search

Graph Neural Networks (GNNs) are a specialized type of neural network designed to operate on graph-structured data. They are adept at capturing and utilizing the relationships within complex interconnected systems. In our research, we focus on developing an attention-based graph model, drawing upon the foundational principles of Graph Convolutional Network (GCN) and Graph Attention Network (GAT). GCN performs localized convolutional operations on graph nodes [27]. This approach iteratively aggregates information from neighboring nodes, effectively capturing the local graph structure. GAT incorporates attention mechanisms [28]. These mechanisms allow the model to adaptively assign varying levels of importance to different neighboring nodes during the aggregation process, enhancing the ability of the model to focus on more relevant information within the graph.

Several attempts have been made to automate graph model design using Neural Architecture Search (NAS) techniques, eliminating the need for manual intervention. GraphNAS and AGNN employ Reinforcement Learning frameworks to explore task-specific GNN structures [8,13]. Both approaches require training separate recurrent networks as controllers. In contrast, Genetic-GNN utilizes Evolutionary Algorithms as the optimization method for architecture search [10]. Comparing it to reinforcement-based NAS methodologies, it shows comparable performance and lower computational costs. However, one drawback of this approach is the significant time required for the overall architecture search process. DSS and SANE utilize differentiable architecture search and one-shot search to discover GNN models, enabling the use of pre-trained weights and reducing search time [9,29]. Nevertheless, the limitation is the potential underestimation of performance due to evaluation without additional training.

3. Graph Neural Network Architecture Search

3.1. Problem Statement

The main goal of our paper is to leverage the Neural Architecture Search (NAS) in conjunction with the Evolutionary Algorithm (EA) to identify an optimal Graph Neural Network (GNN) structure for the task of semi-supervised node classification. Within our research, the objective function is defined as depicted in Equation (1). The aim is to discover a GNN model denoted as $g^{*}$, which belongs to the search space of GNN G, and maximizes the fitness value $f_D\left(g\right)$. D represents the validation dataset, and the fitness metric employed within the EGNAS framework is the validation accuracy of the GNN. To explore the expansive search space of GNNs, we adopt an evolutionary search methodology. By integrating NAS and our evolutionary search, we ascertain the optimal GNN structure that exhibits outstanding performance.

$$g^{*}=\underset{g\in G}{\arg \max }{f}_D\left(g\right)$$

(1)

The search space in the EGNAS framework comprises the structure space and the hyperparameter space. In the structure space, the GNN structure $\mathcal{S}$ is represented as a matrix with dimensions ${\mathbb{R}}^{\left|\mathcal{S}1\right|\times \left|\mathcal{S}2\right|\times \cdots \times |{\mathcal{S}}_m|}$. Each component ${\mathcal{S}}_i$ represents the set of choices for the ith structure component, such as the GNN aggregation function. Similarly, in the hyperparameter space, the hyperparameters $\mathcal{P}$ are represented as a matrix with dimensions ${\mathbb{R}}^{\left|\mathcal{P}1\right|\times \left|\mathcal{P}2\right|\times \cdots \times |{\mathcal{P}}_n|}$. Each component ${\mathcal{P}}_j$ represents the set of candidate choices for the jth hyperparameter, such as the dropout rate. The number of structure search space types is denoted by m, and the number of hyperparameter search space types is denoted by n. In the search space section, we provide a detailed explanation of the different search space types.

The primary objective of EGNAS is to discover the optimal combination of GNN structure components and hyperparameters from entire generations, denoted as ${\mathcal{SP}}^{best}$. This combination results in the Graph Neural Network $g^{*}$ that maximizes the fitness value f. It can be represented as ${\mathcal{SP}}^{best}={\{\mathcal{S}1,\cdots ,\mathcal{S}m,\mathcal{P}1,\cdots ,\mathcal{P}n\}}^{best}$.

Graph Neural Networks (GNNs) utilize the graph structure $G=(V,E)$, where V represents the set of nodes and E represents the set of edges. The input to GNNs consists of graph structure and node features $X_v$ for each node $v\in V$, and the output entails node embeddings $h_v$. In the $k^{th}$ layer of GNN, it can be represented as follows:

$$a_{u,v}^{\left(k\right)}={\mathrm{ATTENTION}}^{\left(k\right)}\left(h_u^{(k-1)},h_v^{(k-1)}\right)$$

(2)

$$h_v^{\left(k\right)}={\mathrm{ACT}}^{\left(k\right)}\left({\mathrm{AGG}}^{\left(k\right)}\left(\{a_{u,v}^{\left(k\right)}·h_u^{(k-1)}:u\in \mathcal{N}\left(v\right)\}\right)\right)$$

(3)

where $h_v^{\left(k\right)}$ is the embedding vector of node v and $a_{u,v}^{\left(k\right)}$ is the attention weight between the node u and v in the $k^{th}$ layer. The term $\mathcal{N}\left(v\right)$ represents the set of neighborhood nodes surrounding node v. $\mathrm{ATTENTION}$, $\mathrm{ACT}$, $\mathrm{AGG}$ refer to attention, activation, and aggregation functions, respectively.

In the context of developing GNN embedding vectors, we adopt a list of essential elements that delineate the range of exploration for potential model candidates. This list is referred to as the search space in Neural Architecture Search (NAS). In this study, we utilize two kinds of search space: the structure space and the hyperparameter space.

Table 1. The structure search space for GNN layers.

Structure Component	Search Space
Attention Function	const, gcn, gat, sym-gat, cos, linear, gene-linear
Aggregation Function	sum, mean-pooling, max-pooling, mlp
Activation Function	sigmoid, tanh, relu, linear, softplus, leaky_relu, relu6, elu
Attention Head	1, 2, 4, 6, 8, 16
Hidden Unit	4, 8, 16, 32, 64, 128, 256

provides components of the structure search space [8]. Attention Function determines the type of attention mechanism used in GNN. Each option represents a different way of weighing information from neighboring nodes. Aggregation Function determines how the information from neighboring nodes is combined in each graph convolution layer. Activation Function applies a non-linear transformation to the aggregated information in each layer. Attention Head specifies the number of attention heads used in the GNN. Multiple attention heads allow the model to capture different patterns and relationships in the graph. Hidden Unit determines the number of hidden units or output channels, represents the dimensionality of the node embeddings, and influences the expressive power and complexity of the model. When constructing a GNN with multiple layers, all five components need to be determined for each layer.

3.2. Search Space

Table 2. The hyperparameter search space for GNN layers.

Hyperparameter Component	Search Space
Dropout Rate	0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6
Learning Rate	$5\times {10}^{-4},1\times {10}^{-3},5\times {10}^{-3},1\times {10}^{-2}$
Weight Decay	$5\times {10}^{-4},8\times {10}^{-4},1\times {10}^{-3},4\times {10}^{-3}$

presents the search space for the hyperparameters of the GNN layers [10]. The three specific hyperparameters that need to be determined are the dropout rate of each layer, the learning rate, and the scale of weight decay for the training phase. We configure the dropout rate uniquely for each layer, ensuring its inclusion in every GNN layer. This setting originates from GraphNAS, a key reference study for our research. We use these values to enable a meaningful comparison between our study and previous works. By leveraging the search spaces in Table 1 and Table 2, we intend to explore various combinations to develop diverse GNN architectures with specific training configurations for optimal performance.

4. EGNAS

4.1. Overall Algorithm

The Efficient Graph Neural Architecture Search (EGNAS) algorithm aims to discover the best fitness Graph Neural Network (GNN) using an evolutionary approach. The overall algorithm is described in Algorithm 1. Algorithm 1 begins by initializing the individuals by randomly selecting elements from both the structure space and hyperparameter space. Each individual is then decoded into a GNN model based on its structure genes, and training is conducted according to its hyperparameter genes. The accuracy of trained model is evaluated using a validation set, enabling us to obtain the fitness values of the initial population.

Algorithm 1 EGNAS

Input: Population size K, the number of generations T, the number of parent candidates N, the number of parent pairs M, training set $D_{train}$, validation set $D_{valid}$ Output: Best fitness GNN   1: ${SP}_0\leftarrow$ Initialize the combined population with the size K   2: Training the individuals in ${SP}_0$ on $D_{train}$   3: Evaluate the fitness of individuals in ${SP}_0$ on $D_{valid}$   4: for $t:0$ to $T-1$ do   5:     $A_t\leftarrow$ Parent candidates selected from ${SP}_t$ through N-roulette wheel selection   6:     $O_t\leftarrow \varphi$   7:     for $m:1$ to M do   8:         $P_1,P_2\leftarrow$ Randomly selected parents from $A_t$   9:         $C_1^{\prime },C_2^{\prime }\leftarrow$Combined Evolutionary Search$\left(P_1,P_2\right)$ 10:         $O_t\leftarrow O_t\cup C_1^{\prime }\cup C_2^{\prime }$ 11:     end for 12:     Training the individuals in $O_t$ on $D_{train}$ 13:     Evaluate the fitness of individuals in $O_t$ on $D_{valid}$ 14:     ${SP}_{t+1}\leftarrow$ Select K individuals from ${SP}_t\cup O_t$ through elitist replacement selection 15: end for 16: return Best fitness individual from $\{{SP}_1,\cdots ,{SP}_T\}$

The algorithm operates through multiple generations. Each generation begins with the selection of parent candidates from the current population using a roulette wheel selection. Using $f_D\left(g\right)/{\sum }_{g\in {SP}_t}{f}_D\left(g\right)$ as the probability of selection, where $f_D\left(g\right)$ represents the fitness of the model, a predetermined number of parent candidates are selected based on this criterion. Parent pairs are randomly chosen from the selected candidates, ensuring that each pair is unique within the same generation. Crossover and mutation operations are employed to generate offspring individuals, mixing genetic information from the selected parent pair. After their creation, these offspring undergo training using the training set, and their fitness is subsequently evaluated on the validation set.

The population for the next generation is formed by applying elitist replacement selection. The elitist replacement selection is a method that replaces the worst individual of the current generation if the structure of a new individual is not found among the population. If the structure does exist, it is compared with that individual, and only the superior one is retained. This method ensures the preservation of the most superior solutions from the current generation, thereby maintaining the quality of the population in each successive generation. This iterative process continues for a specified number of generations, allowing the algorithm to refine and enhance the population over time. Finally, the best fitness individual from all populations is returned as the best-performing GNN model discovered by the algorithm.

4.2. Combined Evolutionary Search

The extension to include hyperparameter search in NAS enhances the potential for superior performance by broadening the exploration scope, which also facilitates obtaining a fully optimized model without necessitating further tuning. This advancement consequently raises the automation level within NAS. Nonetheless, the extended scope increases the required search time, overshadowing these benefits [10]. In response, we propose the combined evolutionary search, a strategy designed to leverage the advantages of expanded search scope while concurrently reducing the time expenditure. This method, comprising crossover and mutation phases as outlined in Algorithm 2, aims to generate offspring that inherit genetic information from their parent configurations.

Algorithm 2 Combined Evolutionary Search

Input: Parents $P_1,P_2$, the length of structure genes s, the length of hyperparameter genes h, mutation probability $\mu$ Output: Offsprings generated by crossover and mutation   1: Randomly generate a number $r_1$ in range of $[0,1)$   2: if  $r_1\le 0.5$ then   3:     $C_1,C_2\leftarrow$ Structure one-point crossover of $P_1,P_2$ in the range of integers $[0,s)$   4: else   5:     $C_1,C_2\leftarrow$ Hyperparameter one-point crossover of $P_1,P_2$ in the range of integers $[0,h)$   6: end if   7: for i to $1,2$ do   8:     Randomly generate a number $r_2$ in range of $[0,1)$   9:     if $r_2\le \mu$ then 10:         $C_i^{\prime }\leftarrow$ Structure and hyperparameter point replacement mutation of $C_i$ 11:     else 12:         $C_i^{\prime }\leftarrow C_i$ 13:     end if 14: end for 15: return Offsprings $C_1^{\prime },C_2^{\prime }$

In the EGNAS algorithm, we employ a combined chromosome structure, incorporating both structure genes and hyperparameter genes. We divide the crossover and mutation processes into two distinct types: structure and hyperparameter. Crossover is applied selectively, targeting either structure or hyperparameter for each model candidate. This selection is determined randomly with equal probability, ensuring controlled genetic modification. The purpose of this restriction is to reduce the risk of deviating too far from effective solutions identified in previous generations. This is crucial because crossover can induce extensive changes within the individual. On the other hand, mutations are applied concurrently to both structure and hyperparameter aspects. Through this simultaneous application, we increase variability and diversity within the expanded search space. This approach fosters a comprehensive exploration and aids in global optimization.

To perform the crossover, a random number $r_1$ is generated in the range of $[0,1)$. If r is less than or equal to 0.5, a structure one-point crossover is conducted between parents $P_1$ and $P_2$. This involves selecting a random point within the range of integers $[0,s)$, where s represents the length of the structure genes. On the other hand, if $r_1$ is greater than 0.5, a hyperparameter one-point crossover is performed. A random point within the range of integers $[0,h)$, where h denotes the length of the hyperparameter genes, is selected as the crossover point. For the mutation step, another random number $r_2$ is generated. If $r_2$ is less than or equal to the mutation probability $\mu$, a random point replacement mutation occurs for both the structure and hyperparameter, respectively. On the other hand, if $r_2$ is greater than $\mu$, the offspring remains unchanged. An example of a mutation in EGNAS is illustrated in Figure 1. In this example, the activation function of the first layer is mutated from ELU to Tanh, and the learning rate is mutated from $5\times {10}^{-3}$ to $1\times {10}^{-2}$. These $\mu$ were chosen to reflect the adoption of a relatively higher mutation probability, intended to introduce greater variation during the evolutionary process. This approach mitigates weight coupling and encourages the exploration of a more diverse range of architectural designs.

Finally, the algorithm outputs the two offspring individuals, denoted as $C_1^{\prime }$ and $C_2^{\prime }$, which have undergone the crossover and mutation operations. Throughout this step, new genetic combinations are explored, promoting diversity within the population and enabling the search for advanced solutions in subsequent generations.

4.3. Inherited Parameter Sharing

The main issue causing weight coupling when using parameter sharing techniques is that all trained models can potentially share their parameters. To address this, EGNAS leverages the characteristics of evolutionary search and introduces the concept of inherited parameter sharing. With this technique, offspring exclusively inherit parameters from their parents. By adopting inherited parameter sharing, EGNAS not only benefits from the accumulated knowledge of successful architectures but also avoids repeated selection and overfitting.

With parameter sharing, the necessity for further training diminishes. Accordingly, we introduce the half epochs strategy which reduces the number of training epochs by half. When all layers of an offspring have received parameters from the parents, the training epoch is halved. This not only mitigates overfitting but also reduces computational costs and enhances search efficiency.

Parameter sharing should be conducted after confirming whether specific elements of the GNN are the same. These key components include layer number, input channels, output channels (hidden units), attention type, aggregator type, and the number of heads. When the layer of the offspring matches the key components of the parent layer, the offspring inherits the parameters of the parents. This targeted parameter sharing ensures relevant and compatible information exchange.

During the crossover process, which combines genetic information from parents, parameter sharing plays a crucial role. Figure 2 illustrates a scenario where parameters of a single parent are inherited when the crossover point lies within a layer. Conversely, Figure 3 demonstrates an example where the crossover point is between layers, allowing the offspring to inherit parameters from both parents. In this case, the half epochs condition is met, and the offspring undergoes training for only half of the epochs compared to the original training procedure. This enables the offspring to take advantage of the pre-trained knowledge from both parents.

In the mutation stage, random modifications are introduced to the genetic information of offspring. Figure 4 showcases an example where a key component is altered during mutation, resulting in the loss of a specific shared parameter. When the second layer aggregator of $C_2$ is changed to mean, the key component of the second layer in both $C_2$ and $P_1$ becomes mismatched. As a result, the offspring cannot inherit weight 2. In such cases, the offspring is trained with the newly initialized parameters.

5. Experiments

5.1. Datasets

For our experiments, we utilized three representative node classification benchmark datasets: Cora, Citeseer, and PubMed [30]. These datasets represent citation networks, forming single graphs where each node corresponds to a document. The citation relationships between papers are depicted as undirected edges. Furthermore, each paper is composed of bag-of-words features, and the task is to classify the type of each paper.

Table 3. Descriptions of graph datasets.

Dataset	# Nodes	# Edges	# Features	# Classes
Cora	2708	5429	1433	7
Citeseer	3327	4732	3703	6
PubMed	19,717	44,338	500	3

presents the detailed statistics for each dataset.

For data splitting, we follow the semi-supervised learning approach introduced by Yang et al. [31], which utilizes all node features and edge information and employs only a subset of node labels for training. We designate the last 1000 data points in each dataset as the test set, while the preceding 500 data points are employed as the validation set. As the training data, we use the initial 140, 120, and 60 data points for Cora, Citeseer, and PubMed, respectively.

5.2. Experimental Settings

The two types of hyperparameters used for the experiments are as follows:

Hyperparameters of evolutionary search: The experimental settings for EGNAS include a population size of 20 individuals and a total of 50 generations of evolution. The number of parent candidates is set to 10, and 10 parent pairs are selected for reproduction. A mutation probability of 0.2 is applied during the evolutionary process.

Hyperparameters of GNNs: We adopt a GNN architecture consisting of two layers. As a result, the length of the structure genes is adjusted to 10, representing the two GNN layers. Additionally, since a dropout rate is specified for each of the two layers, the length of the hyperparameter genes is set to 4. Therefore, the total number of architectures in the search space is $5.4\times {10}^{13}$, which is the largest value among the GNN architecture search methods compared. The training duration for individuals that do not correspond to half epochs is set to 200 epochs, while individuals corresponding to half epochs are trained for 100 epochs. EGNAS-10 applies early stopping with a patience of 10, halting training if validation loss does not decrease for 10 consecutive epochs. We adopt Glorot parameter initialization [32] and utilize the Adam optimizer to minimize the cross-entropy loss during the training process.

5.3. Baseline Selection

To ensure a comprehensive comparison, we chose baselines that are not only a set of state-of-the-art approaches representative of the diverse field but also are practically implementable. Our selection includes two categories of baselines: hand-crafted GNN architectures and Graph Neural Network Architecture Search (GNN NAS) methods. The hand-crafted architectures comprise GCN (Graph Convolutional Network) and GAT (Graph Attention Network), which are representative structures in GNN research. GCN employs spectral convolutional filters to gather information from all adjacent nodes within a graph [27]. GAT utilizes an attention mechanism to weigh node-level features, thereby calculating the relationships between adjacent nodes [28]. In addition, we incorporate three GNN NAS approaches into our baselines. GraphNAS is a Reinforcement Learning-based method for optimizing GNN architectures, where a recurrent network is trained to maximize the metric of generated architectures [8]. SANE is a methodology that introduces differentiable NAS methods to improve search efficiency [29]. Genetic-GNN applies an evolutionary search to evolve model structures and hyperparameters over generations [10]. Notably, Genetic-GNN is one of the most recent studies and has recorded superior performance in the GNN NAS field. All algorithms are executed using the specified parameters outlined in the paper.

To ensure equitable evaluations, the handcrafted algorithm outcomes are generated through the training of 1000 models, mirroring the candidate count in EGNAS. The top five models with the highest validation accuracy are selected to calculate the average test accuracy for the handcrafted approach. In the GNN NAS baseline experiments, methods are executed in a manner consistent with their published methods. GraphNAS selects the top five models by validation accuracy, each retrained 20 times from scratch, with their best outcomes averaged for the final result. Similarly, Genetic-GNN utilizes its top five models but does not retrain them. SANE conducts five experiments, averaging the results of retraining the best validation model five times.

Contrasting with GraphNAS and SANE, EGNAS directly employs the top five validation models without retraining. The essence of NAS research lies in its capacity to automate processes, thereby reducing manual intervention and yielding immediately usable model parameters. EGNAS demonstrates this principle by incorporating hyperparameter search and presenting outcomes without additional retraining.

5.4. Results

Table 4. Test accuracy (%) comparison in the node classification task.

	Algorithm	Cora	Citeseer	PubMed
Handcrafted	GCN	$81.18\pm 0.49$	$72.04\pm 0.61$	$78.82\pm 0.47$
	GAT	$82.98\pm 0.44$	$71.40\pm 0.27$	$77.98\pm 0.24$
GNN NAS	GraphNAS	$82.26\pm 0.96$	$69.46\pm 0.99$	$76.94\pm 0.39$
	SANE	$79.12\pm 1.04$	$67.84\pm 0.99$	$76.29\pm 1.16$
	Genetic-GNN	$83.00\pm 0.44$	$73.12\pm 0.34$	$78.20\pm 0.00$
Ours	EGNAS	$83.70\pm 0.00$	$\mathbf{74}.\mathbf{10}\pm \mathbf{0}.\mathbf{00}$	$79.40\pm 0.00$
	EGNAS-10	$\mathbf{83}.\mathbf{80}\pm \mathbf{0}.\mathbf{00}$	$73.90\pm 0.00$	$\mathbf{79}.\mathbf{50}\pm \mathbf{0}.\mathbf{00}$

The bold formatting indicate the most optimal values.

compares test accuracies by various algorithms in node classification within semi-supervised learning. This table encompasses outcomes from conventional, handcrafted techniques (e.g., GCN, GAT) and cutting-edge GNN NAS methodologies, alongside our novel proposal, EGNAS. EGNAS-10 means integrating an early stopping mechanism into EGNAS. The most exceptional outcomes are highlighted using boldface type. The experimental findings demonstrate the notable efficacy of both EGNAS and EGNAS-10, which consistently surpass the baseline accuracy on the Cora, Citeseer, and PubMed datasets. EGNAS-10 achieves state-of-the-art performance in the Cora and PubMed datasets, while EGNAS emerges as the top-performing algorithm in the Citeseer dataset.

Table 5. Graph Neural Architecture Search time (m) in the node classification tasks.

Algorithm	Cora	Citeseer	PubMed
GraphNAS	119.5	153.3	256.4
SANE	39.1	61.7	40.2
Genetic-GNN	862.9	3937.8	4378.1
EGNAS	33.3	87.0	91.9
EGNAS-10	9.1	17.2	44.4

The bold formatting indicate the most optimal values.

presents the time duration, measured in minutes, required for the process of Graph Neural Architecture Search across diverse algorithms. The time measurements for the search process were uniformly conducted for each dataset. Specifically, for the Cora and Citeseer datasets, computations were performed on a GeForce RTX 2080 GPU, Intel Xeon Gold 6138 CPU, while the PubMed dataset leveraged an NVIDIA TITAN V with AMD Threadripper 3990 WX.

EGNAS surpasses both GraphNAS and Genetic-GNN in the comparison of search times across three datasets. Relative to GraphNAS, EGNAS requires only about 40% of the search time on average. Notably, against Genetic-GNN, which shows the highest accuracy apart from EGNAS in most datasets, EGNAS demonstrates a search speed that is on average about 40 times faster. EGNAS is capable of finding more accurate models than SANE with only a marginal difference in search time. SANE is on average 23.7 min faster than EGNAS, but it significantly lags behind EGNAS in accuracy, with differences of 4.58%, 6.26%, and 3.11% across the Cora, Citeseer, and PubMed datasets, respectively. Furthermore, the EGNAS-10 variant not only maintains a comparable level of accuracy to EGNAS but also enhances search time efficiency. Compared to SANE, the fastest among the baselines, EGNAS-10 demonstrates an average improvement of 23.4 min in search speed.

Table 6. Computational costs comparison in the node classification task.

Algorithm	Cora			Citeseer			PubMed
	Params (M)	Time (ms)	Latency (ms)	Params (M)	Time (ms)	Latency (ms)	Params (M)	Time (ms)	Latency (ms)
GCN	0.02	4.64	2.57	0.06	4.55	2.52	0.01	4.39	1.99
GAT	0.09	8.19	3.15	0.24	8.70	3.22	0.03	9.79	3.15
GraphNAS	1.36	9.95	3.82	7.53	22.01	9.28	0.10	7.32	2.99
SANE	0.73	8.94	3.07	1.31	10.66	4.53	0.04	8.99	3.14
Genetic-GNN	0.14	4.37	1.38	10.76	42.19	16.58	0.05	4.90	1.51
EGNAS	0.10	5.32	1.45	5.71	21.74	8.49	0.20	11.45	3.83
EGNAS-10	0.38	5.27	1.52	0.48	7.78	1.93	0.52	21.12	8.13

presents the computational costs associated with the results in Table 4, analyzing metrics that include the number of trainable parameters (Params), training time per epoch (Time), and inference latency per epoch (Latency). In the Cora dataset, EGNAS distinguished itself by not only achieving the highest accuracy among GNN NAS algorithms but also recording the smallest number of trainable parameters. Moreover, its training and inference durations are either comparable or faster than its counterparts. Notably, in the Citeseer dataset, the EGNAS-10 variant attains higher accuracy while demonstrating the least computational costs among the GNN NAS alternatives. On the PubMed dataset, EGNAS shows larger computational costs when compared to other GNN NAS methods. This might be due to the relatively subdued performance of these groups compared to manually constructed models (GCN, GAT).

In conclusion, EGNAS stands out as a remarkable solution in the realm of GNN NAS that not only achieves the highest performance with minimal search time but also maintains a balance between model complexity and computational efficiency. This indicates that the EGNAS methodology possesses both efficiency and effectiveness.

5.5. Ablation Study

To assess the effectiveness of the EGNAS framework, we conduct ablation studies on the combined evolutionary search, inherited parameter sharing, and half epochs. The results, shown in

Table 7. Comparison of test accuracy (%) in the node classification task for original algorithm and ablation studies.

Algorithm	Cora	Citeseer	PubMed
EGNAS	$\mathbf{83}.\mathbf{70}\pm \mathbf{0}.\mathbf{00}$	$\mathbf{74}.\mathbf{10}\pm \mathbf{0}.\mathbf{00}$	$\mathbf{79}.\mathbf{40}\pm \mathbf{0}.\mathbf{00}$
Separate	$83.08\pm 0.18$	OOM	$78.16\pm 0.14$
No PS	$82.70\pm 0.00$	$72.68\pm 0.65$	$78.00\pm 0.00$
Fully PS	$81.92\pm 0.35$	$72.90\pm 0.00$	$78.50\pm 0.00$

The bold formatting indicate the most optimal values.

and

Table 8. Comparison of Graph Neural Architecture Search time (m) in the node classification task for original algorithm and ablation studies.

Algorithm	Cora	Citeseer	PubMed
EGNAS	33.3	87.0	91.9
Separate	476.0	OOM	1299.9
No PS	50.2	67.5	149.0
Fully PS	28.0	33.7	90.8

, explore scenarios where we optimize the structure and hyperparameter space separately (‘separate’), do not employ parameter sharing (‘no PS’), and allow sharing from all trained parameters (‘fully PS’). In the ‘separate’ experiment, the Citeseseer trial was conducted under similar conditions as before. However, due to the increased number of model candidates that needed to be stored, we faced out-of-memory issues and then left that column blank. The population size and the number of generations for the hyperparameters were set at 4 and 5, respectively, identical to those used in Genetic-GNN [10]. Contrasting with EGNAS, the ‘separate’ requires running hyperparameter generations within structure generations, resulting in the formation of more candidates and extending the search time by approximately 14 times compared to EGNAS. Despite the increased time, the accuracy also falls by an average of 0.93%.

Under conditions without parameter sharing (’no PS’), except for Citeseer, more search time is required and accuracy also drops by an average of 1.27%. Conversely, when the scope of candidates for parameter sharing is expanded (’fully PS’), search time decreases compared to EGNAS, but performance decreases compared to the non-sharing scenario. This is because repeated selection of the same parameters speeds up the process by reducing training needs, but it often results in convergence to local optima.

Figure 5 illustrates the trajectory of performance for the best validation models across generations within the Cora dataset, contingent upon the employment of the half epochs. Each experiment was conducted 10 times. The solid lines represent the average experimental results, and the shaded areas indicate the variance. In EGNAS integrated with half epochs, accuracy consistently displays an upward trajectory with the passage of generations. In contrast, excluding the half epochs approach results in a relatively moderate upward curve recording lower values compared to EGNAS. This can be attributed to overfitting, which happens when excessive learning occurs from repeating the same epochs, despite using previously trained parameters through parameter sharing. Moreover, the absence of the half epochs strategy leads to an elongated training time, as evident from the increase in training duration to 41.2 min, in contrast to EGNAS’s 33.3 min.

6. Conclusions

We proposed EGNAS, a method for the efficient and effective discovery of optimal Graph Neural Networks. Through combined evolutionary search, we rapidly explored a large search space expanded into the hyperparameter space. The integration of inherited parameter sharing and half epochs strategy significantly enhanced the reusability of trained parameters and mitigated the risk of overfitting. As a result, EGNAS recorded the highest accuracy in node classification tasks than state-of-the-art hand-crafted models and other GNN NAS methods with less computational cost. Furthermore, the considerable reduction in search time suggests its immediate applicability in practical industrial settings. We anticipate EGNAS to be a valuable tool for rapidly identifying task-specific models across various GNN domains, like social networks, transportation, and bioinformatics.