Runtime Construction of Large-Scale Spiking Neuronal Network Models on GPU Devices

Abstract

Simulation speed matters for neuroscientific research: this includes not only how quickly the simulated model time of a large-scale spiking neuronal network progresses but also how long it takes to instantiate the network model in computer memory. On the hardware side, acceleration via highly parallel GPUs is being increasingly utilized. On the software side, code generation approaches ensure highly optimized code at the expense of repeated code regeneration and recompilation after modifications to the network model. Aiming for a greater flexibility with respect to iterative model changes, here we propose a new method for creating network connections interactively, dynamically, and directly in GPU memory through a set of commonly used high-level connection rules. We validate the simulation performance with both consumer and data center GPUs on two neuroscientifically relevant models: a cortical microcircuit of about 77,000 leaky-integrate-and-fire neuron models and 300 million static synapses, and a two-population network recurrently connected using a variety of connection rules. With our proposed ad hoc network instantiation, both network construction and simulation times are comparable or shorter than those obtained with other state-of-the-art simulation technologies while still meeting the flexibility demands of explorative network modeling.

1. Introduction

Spiking neuronal network models are widely used in the context of computational neuroscience to study the activity of the populations of neurons in the biological brain. Numerous software packages have been developed to simulate these models effectively. Some of these simulation engines offer the ability to accurately simulate a wide range of neuron models and their synaptic connections. Among the most popular codes are NEST [1], NEURON [2], and Brian 2 [3]. NENGO [4] and ANNarchy [5] should also be mentioned. In recent years, there has been a growing interest in GPU-based approaches, which can be particularly useful for simulating large-scale networks thanks to their high degree of parallelism. This interest is also fueled by the rapid technological development of this type of device and the availability of increasingly performant GPU cards both for the consumer and for high-performance computing (HPC) infrastructure. A main driving force behind this development is the demand from current artificial intelligence algorithms and similar applications for massively parallel processing of simple floating point operations, and a corresponding industry with huge financial resources. Present-day supercomputers are reaching for exascale by drawing their compute power from GPUs. For neuroscience to benefit from these systems, efficient algorithms for the simulation of spiking neuronal networks need to be developed. Simulation codes such as GeNN [6], CARLsim [7,8], and NEST GPU [9] have been primarily designed for GPUs, while, in recent times, popular CPU-based simulators have shown interest in integrating the more traditional CPU-based approach with libraries for GPU simulation [10,11,12,13,14,15]. Furthermore, the novel simulation library Arbor [16], which focuses on morphologically-detailed neural networks, takes GPUs into account.

In general, GPU-based simulators fall into one of three categories: those that allow the construction of network models at run time using scripting languages, those that require the network models to be fully specified in a compiled language, and hybrid ones that provide both options. The most extensively used compiled languages are C and C++ for host code and CUDA for device code (using NVIDIA GPUs), while the most widely used scripting language is Python. With scripting languages, simulations can be performed without the need to compile the code used to describe the model. Consequently, the time required for compilation is eliminated. Furthermore, in many cases, the use of a scripting language simplifies the implementation of the model, especially for users who do not have extensive programming language expertise. Approaches using compiled languages typically have much faster network construction times. To reconcile this benefit with the greater ease of model implementation using a scripting language, some simulators have shifted toward a code-generation approach [5,6]. In this approach, the model is implemented by the user via a brief high-level description, which the code generator then converts into the language or languages that must be compiled before being executed in the CPU and GPU. The main disadvantage of code-generation based simulators is the need for new code generation and compilation every time model modifications such as changes in network architecture are necessary. The times associated with code generation and compilation are typically much longer than network construction times [11].

Examples of the code-generation based approaches include GeNN [6] and ANNarchy [5]. In GeNN, neuron and synapse models are defined in C++ classes, and snippets of C-like code can be used to offload costly operations onto the GPU. The Python package PyGeNN [17] is built on top of an automatically generated wrapper for the C++ interface (using SWIG (https://www.swig.org, accessed on 14 August 2023)) and allows for the same low-level control. Further, Brian2GeNN [12] provides a code generation pipeline for defining models via the Python interface of Brian [3] and using GeNN as a simulator backend. Alternatively, Brian2CUDA [14] directly extends Brian with a GPU backend. The hybrid approach is exemplified in CARLsim [8], which has also developed its own Python interface to communicate with its C/C++ library named PyCARL [18]. Much like PyNEST [19], the Python interface of the NEST simulator, CARLsim exposes its C/C++ kernel with a dynamic library, which can then interact with Python. However, like GeNN, they make use of SWIG to automatically generate the binding between their library and their Python interface. PyCARL directly serves as a PyNN [20] interface. NEST GPU [13] is a software library for the simulation of spiking neural networks on GPUs. It originates from the prototype NeuronGPU library [9] and is now overseen by the NEST Initiative and integrated with the NEST development process. NEST GPU uses a hybrid approach and offers the possibility to implement models using either Python scripts or C++ code. The main commands of the Python interface, the use of dictionaries, and the names and parameters of the neuron and spike generator models are already aligned to those of the CPU-based NEST code. In a previous version of NEST GPU, connections were first created on the CPU side and then copied from the RAM to GPU memory. This approach benefited from the standard C++ libraries and, particularly, the dynamic allocation of container classes of the C++ Standard Template Library. However, it had the drawback of relatively long network construction times, not only due to the costly copying of connections and other CPU-side initializations, but also because the connection creation process was performed serially.

This work proposes a network construction method in which the connections are created directly in the GPU memory with a dynamic approach and then suitably organized in the same memory using algorithms that exploit GPU parallelism. This approach, so far applied to single-GPU simulations, enables much faster connection creation, initialization, and organization while preserving the advantages of dynamic connection building, particularly the ability to create and initialize the model at run-time without the need for compilation. Although this method was developed specifically in the framework of NEST GPU, the concepts are sufficiently general that they should be applicable with minimal adaptation to other GPU-based simulators as long as they are designed with a modular structure.

Section 2 of this manuscript first introduces the dynamic creation of connections and provides details on the used data structures and the spike buffer employed in the simulation algorithm (Section 2.1, Section 2.2 and Section 2.3); details on the employed block sorting algorithm are in Appendix A. The proposed dynamic approach for network construction is tested on the simulation of two complementary weighted network models across different hardware configurations; we then compare the performance to other simulation approaches. Details on the network models, the hardware and software, and time measurements for performance evaluations are given in Section 2.4, Section 2.5 and Section 2.6. The spiking activity of a network constructed with the dynamic approach is validated statistically in Section 2.7 and Appendix B. The performance results are shown in Section 3 (with additional data in Appendix C and Appendix D) and discussed in Section 4.

2. Materials and Methods

2.1. Creation of Connections Directly in GPU Memory

A network model is composed of nodes, which are uniquely identifiable by index and connections between them. In NEST and NEST GPU, a node can be either a neuron or a device for stimulation or recording. Neuron models can have multiple receptor ports to allow for receptor-specific parameterization of input connections. Connections are defined in NEST GPU (and similarly in other simulators) via high-level connection routines, e.g., ngpu.Connect(sources, targets, conn_dict, syn_dict), where the connection dictionary conn_dict specifies a connection rule, e.g., one_to_one, for establishing connections between source and target nodes. The successive creation of several individual sub-networks, according to deterministic or probabilistic rules, can then lead to a complex overall network. In the rules used here, we allow autapses (self-connections) and multapses (multiple connections between the same pair of nodes); see [21] for a summary of state-of-the-art connectivity concepts.

The basic structure of a NEST GPU connection includes the source node index, the target node index, the receptor port index, the weight, the delay, and the synaptic group index. The synaptic group index takes non-zero values only for non-static synapses (e.g., STDP) and refers to a structure used to store the synapse type and the parameters common to all connections of that group; it should not be confused with the connection group index, which groups connections with the same delay for spike delivery. Additional parameters may be present depending on the type of synapse, which is specified by the synaptic group. The delay must be a positive multiple of the simulation time resolution and can, therefore, be represented using time-step units as a positive integer. Connections are stored in GPU memory in dynamically-allocated blocks with a fixed number of connections per block, B, which can be specified by the user as a simulation kernel parameter before creating the connections. It should be chosen on the basis of a compromise. If it is chosen too small, then the total number of blocks would be high, resulting in longer execution times. Conversely, if it is chosen too large, a significant amount of memory could be wasted due to incomplete filling of the last allocated block. The default value for B used in all simulations of this study is ${10}^7$.

Each time a new connection–creation command is launched, if the last allocated block does not have sufficient free slots to store the new connections, an appropriate number of new blocks is allocated according to the following formula:

$$N_{\mathrm{newblocks}}=\lfloor \frac{N_{\mathrm{conns}}+N_{\mathrm{newconns}}+B-1}{B}\rfloor -N_{\mathrm{blocks}}$$

(1)

where $N_{\mathrm{newblocks}}$ is the number of new blocks that must be allocated, $N_{\mathrm{blocks}}$ is the old number of blocks, $N_{\mathrm{conns}}$ is the old number of connections, $N_{\mathrm{newconns}}$ is the number of connections that must be created, and $\lfloor x\rfloor$ denotes the integer part of x. The new connections are indexed contiguously as follows:

$$i_{\mathrm{newconns}}=0,\dots ,N_{\mathrm{newconns}}-1.$$

(2)

To understand our parallelization strategy we introduce two hardware-related terms, CUDA threads and CUDA kernels. CUDA threads are the smallest GPU computing units, these are grouped into blocks and several blocks are present in a multiprocessor unit. Thus, a thread block refers to a physical group of computing units, each unit having a unique index in the group. CUDA kernels are functions executed on the GPU device, these concurrently exploit multiple CUDA thread blocks. Henceforth, to avoid confusion with connection blocks, we will not mention CUDA thread blocks but refer to CUDA kernels, the functions employing said thread blocks, and CUDA threads as one of the computing units used by these kernels.

A loop is performed on the connection blocks, starting from the first block in which there are available slots up to the last allocated block, and the connections are created in each block by launching appropriate CUDA kernels to set the connection parameters described above. In each block b, the index of each of the new connections is calculated from the CUDA thread index k according to the formula:

$$i_{\mathrm{newconns},\mathrm{b}}=N_{\mathrm{prevconns},\mathrm{b}}+k\mathrm{with}k=0,\dots ,N_{\mathrm{thr}}-1$$

(3)

where $i_{\mathrm{newconns},\mathrm{b}}$ refers to the subset of $i_{\mathrm{newconns}}$ on the current block, and $N_{\mathrm{prevconns},\mathrm{b}}$ is the number of new connections created in the previous blocks. The number of connections to be created in the current block, which corresponds to the number of required threads $N_{\mathrm{thr}}$, is computed before launching the kernel; if the block will be completely filled, the number of threads equals the block size, $N_{\mathrm{thr}}=B$. See Figure 1 for an example of how the connections are numbered and assigned to the blocks.

The indexes of a source node s and a target node t are calculated from $i_{\mathrm{newconns}}$ using expressions that depend on the connection rule. Here, we provide both the name of the rules as defined in [21] and their corresponding parameter of the NEST interface. In case both the source-node group and the target-node group contain nodes with consecutive indexes, starting from $s_0$ and from $t_0$, respectively, the node indexes are as follows:

• One-to-one (one_to_one):

  $$\begin{array}{cc}\hfill s& \hfill =s_0+i_{\mathrm{newconns}}\hfill \end{array}$$

  (4)

  $$\begin{array}{cc}\hfill t& \hfill =t_0+i_{\mathrm{newconns}}\hfill \end{array}$$

  (5)

  with $N_{\mathrm{newconns}}=N_{\mathrm{sources}}=N_{\mathrm{targets}}$.
• All-to-all (all_to_all):

  $$\begin{array}{cc}\hfill s& \hfill =s_0+\lfloor \frac{i_{\mathrm{newconns}}}{N_{\mathrm{targets}}}\rfloor \hfill \end{array}$$

  (6)

  $$\begin{array}{cc}\hfill t& \hfill =t_0+\mathrm{mod}(i_{\mathrm{newconns}},N_{\mathrm{targets}})\hfill \end{array}$$

  (7)

  with $N_{\mathrm{newconns}}=N_{\mathrm{sources}}\times N_{\mathrm{targets}}$.
• Random, fixed out-degree with multapses (fixed_outdegree):

  $$\begin{array}{cc}\hfill s& \hfill =s_0+\lfloor \frac{i_{\mathrm{newconns}}}{K}\rfloor \hfill \end{array}$$

  (8)

  $$\begin{array}{cc}\hfill t& \hfill =t_0+\mathrm{rand}\left(N_{\mathrm{targets}}\right)\hfill \end{array}$$

  (9)

  where K is the out-degree, i.e., the number of output connections per source node, $\mathbf{rand}\left(N_{\mathrm{targets}}\right)$ is a random integer between 0 and $N_{\mathrm{targets}}-1$ sampled from a uniform distribution, and $N_{\mathrm{newconns}}=N_{\mathrm{sources}}\times K$.
• Random, fixed in-degree with multapses (fixed_indegree):

  $$\begin{array}{cc}\hfill s& \hfill =s_0+\mathrm{rand}\left(N_{\mathrm{sources}}\right)\hfill \end{array}$$

  (10)

  $$\begin{array}{cc}\hfill t& \hfill =t_0+\lfloor \frac{i_{\mathrm{newconns}}}{K}\rfloor \hfill \end{array}$$

  (11)

  where K is the in-degree, i.e., the number of input connections per target node, and $N_{\mathrm{newconns}}=n_{\mathrm{targets}}\times K$.
• Random, fixed total number with multapses (fixed_total_number):

  $$\begin{array}{cc}\hfill s& \hfill =s_0+\mathrm{rand}\left(N_{\mathrm{sources}}\right)\hfill \end{array}$$

  (12)

  $$\begin{array}{cc}\hfill t& \hfill =t_0+\mathrm{rand}\left(N_{\mathrm{targets}}\right)\hfill \end{array}$$

  (13)

  where pairs of sources and targets are sampled until the specified total number of connections $N_{\mathrm{newconns}}$ is reached.

If the indexes of source or target nodes are not consecutive but are explicitly given by an array, the above formulas are used to derive the indexes of the array elements from which to extract the node indexes. Weights and delays can have identical values for all connections, or be specified for each connection by an array having a size equal to the number of connections or be randomly distributed according to a given probability distribution. In the latter case, the pseudo-random numbers are generated using the cuRAND library (https://developer.nvidia.com/curand, accessed on 14 August 2023). The delays are then converted to integer numbers expressed in units of the computation time step by dividing their values, expressed in milliseconds, by the duration of the computation time step, and rounding the result to an integer. The minimal delay that is permitted is one computation time step [22]. Thus, if the result is less than 1, the delay is set to 1 in time step units.

2.2. Data Structures Used for Connections

In order to efficiently manage the spike transmission in the presence of delays, the connections must be organized in an appropriate way. To this end, the algorithm divides the connections into groups so that connections from the same group share the same source node and the same delay. This arrangement is needed for the spike delivery algorithm, which is described in the next section. The algorithm achieves this by hierarchically using two sorting keys: the index of the source node as the first key and the delay as the second. Since the connections are created dynamically, their initial order is arbitrary. Therefore, we order connections in a stage that follows network construction and that precedes the simulation called calibration phase (for a definition of the simulation phases, see Section 2.6). The sorting algorithm is an extension of radix sort [23] applied to an array organized in blocks based on the implementation available in the CUB library (https://nvlabs.github.io/cub, accessed on 14 August 2023). Once the connections are sorted, their groups must be adequately indexed so that when a neuron emits a spike, the code has quick access to the groups of connections outgoing from this neuron and to their delays. This indexing is carried out in parallel using CUDA kernels on connection blocks with one CUDA thread for each connection. The connection index extracts the source node index and the connection delay. If one of these two values differs from those of the previous connection, it means that the current connection is the first of a connection group. We use this criterion to count the number of connection groups per source node, $G_i$, and to find the position of each connection group in the connection blocks. The next step constructs for each source node an array of size equal to the number of groups of outgoing connections containing the global indexes of the first connections of each group. Since allocating a separate array for each node would be a time-consuming operation, we concatenate all arrays into a single one-dimensional array. The starting position $p_i$ of the sub-array corresponding to a given source node i can be evaluated using the cumulative sum of $G_i$ as follows:

$$p_i=\sum _{j=0}^{i-1}{G}_{j}i=1,\dots ,N_{\mathrm{nodes}}\mathrm{and}{p}_0=0$$

(14)

where $N_{\mathbf{nodes}}$ is the total number of nodes in the network.

2.3. The Spike Buffer

The simulation algorithm employs a buffer of outgoing spikes for each neuron in the network to manage connection delays [9,13]. Each spike object is composed of three parameters: a time index, a connection group index, and a multiplicity (i.e., the number of physical spikes emitted by a network node in a single time step). The spike buffer has the structure of a queue into which the spikes emitted by the neuron are inserted. Whenever a spike is emitted from the neuron, it is buffered, and both its time index and its connection-group index are initialized to zero. At each simulation time step, the time indexes of all the spikes are increased by one unit. When the time index of a spike matches the delay of the connection group indicated by its connection group index, the spike is fed into a global array called spike array, and its connection group index is incremented by one unit to point to the next connection group in terms of delay. In the spike array, each spike is represented by the source node index, the connection group index, and the multiplicity. The spikes are delivered in parallel to the target nodes using a CUDA kernel with one CUDA thread for each connection of each connection group inserted in the spike array.

2.4. Models Used for Performance Evaluation

This present work evaluates the performance of the proposed approach on two network models: a cortical microcircuit and a simple network model of two neuron populations. The models are depicted schematically in Figure 2. The microcircuit model of Potjans and Diesmann [24] represents a 1 mm${}^2$ patch of early sensory cortex at the biological plausible density of neurons and synapses. The full-scale model comprises four cortical layers (L2/3, L4, L5, and L6) and consists of about 77,000 current-based leaky integrate-and-fire model neurons, which are organized into one excitatory and one inhibitory population per layer. These eight neuron populations are recurrently connected by about 300 million synapses with exponentially decaying postsynaptic currents; the connection probabilities are derived from anatomical and electrophysiological measurements. The connection rule used is fixed_total_number with autapses and multapses allowed. The dynamics of the membrane potentials and synaptic currents are integrated using the exact integration method proposed by Rotter and Diesmann [25], and the membrane potential of the neurons of every population are initialized from a normal distribution with mean and standard deviation optimized from the neuron population as in [26]. This approach avoids transients at the beginning of the simulation. Signals originating from outside of the local circuitry, i.e., from other cortical areas and the thalamus, can be approximated with Poisson-distributed spike input or DC current input. Tables 1–4 of [27] (see fixed total number models) contain a detailed model description and report the values of the parameters. The model explains the experimentally observed cell-type and layer-specific firing statistics, and it has been used in the past both as a building block for larger models (e.g., [28]) and as a benchmark for several validation studies [9,17,26,29,30,31,32].

The second model is designed for testing the scaling performance of the network construction by changing the number of neurons and the number of connections in the network across biologically relevant ranges for different connection rules (see Section 2.1; autapses and multapses allowed). The model consists of two equally sized neuron populations, which are recurrently connected to themselves and to each other in four nestgpu.Connect() calls. The total number of neurons in the network is N (i.e., $N/2$ per population), and the target total number of connections is $N\times K$ connections, where K is the target number of connections per neuron. Dependent on the connection rule used, the instantiated networks may exhibit small deviations from the following target values:

• fixed_total_number:
  The total number of connections used in each connect call is set to $\lfloor N\times K/4\rfloor$.
• fixed_indegree:
  The in-degree used in each connect call is set to $\lfloor K/2\rfloor$.
• fixed_outdegree:
  The out-degree used in each connect call is set to $\lfloor K/2\rfloor$.

The network uses Izhikevich neurons [33], but the studied scaling behavior is independent of the neuron model as well as the neuron, connection, and simulation parameters. Indeed, the only parameters that have an impact on this scaling experiment are the total number of neurons and the number of connections per neuron (i.e., N and K).

2.5. Hardware and Software of Performance Evaluation

As a reference, we implement the proposed method for generating connections directly in GPU memory in the GPU version of the simulation code NEST. In the following, NEST GPU (onboard) refers to the new algorithm in which the connections are created directly in GPU memory, while NEST GPU (offboard) indicates the previous algorithm, which first generates the network in CPU memory and subsequently copies the network structure into the GPU as demonstrated in [9,13]. For a quantitative comparison to other established codes, we use the CPU version of NEST [1] (version 3.3 [34]) and the GPU code generator GeNN [6] (version 4.8.0 (https://github.com/genn-team/genn/releases/tag/4.8.0, accessed on 14 August 2023)).

We evaluate the performance of the alternative codes on four systems equipped with NVIDIA GPUs of different generations and main application areas: two compute clusters, JUSUF [35] and JURECA-DC [36], both using CUDA version 11.3 and equipped with the data center GPUs V100 and A100, respectively, and two workstations with the consumer GPUs RTX 2080 Ti, with CUDA version 11.7 and RTX 4090 with CUDA version 11.4. The NEST GPU and GeNN simulations discussed in this work each employ a single GPU card, both because the novel network construction method developed for NEST GPU is limited to single-GPU simulations and also because all the simulation systems employed have enough GPU memory to simulate the models previously described using a single GPU card. The CPU simulations use a single compute node of the HPC cluster JURECA-DC and exploit its 128 cores via 8 MPI processes each running 16 threads.

Table 1. Hardware configuration of the different systems used to measure the performance of the simulators. Cluster information is given on a per node basis.

System	CPU	GPU
JUSUF cluster	2× AMD EPYC 7742, 2× 64 cores, 2.25 GHz	NVIDIA V100 1, 1530 MHz, 16 GB HBM2e, 5120 CUDA cores
JURECA-DC cluster	2× AMD EPYC 7742, 2× 64 cores, 2.25 GHz	NVIDIA A100 2, 1410 MHz, 40 GB HBM2e, 6912 CUDA cores
Workstation 1	Intel Core i9-9900K, 8 cores, 3.60 GHz	NVIDIA RTX 2080 Ti 3, 1545 MHz, 11 GB GDDR6, 4352 CUDA cores
Workstation 2	Intel Core i9-10940X, 14 cores, 3.30 GHz	NVIDIA RTX 4090 4, 2520 MHz, 24 GB GDDR6X, 16384 CUDA cores

¹ Volta architecture: https://developer.nvidia.com/blog/inside-volta, accessed on 14 August 2023. ² Ampere architecture: https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth, accessed on 14 August 2023. ³ Turing architecture: https://developer.nvidia.com/blog/nvidia-turing-architecture-in-depth, accessed on 14 August 2023. ⁴ Ada Lovelace architecture: https://www.nvidia.com/en-us/geforce/ada-lovelace-architecture, accessed on 14 August 2023.

shows the specifications of these three systems.

For the network models, their specific implementations were taken from the original source for each simulator in the case of the cortical microcircuit model. In particular, both NEST (https://github.com/nest/nest-simulator, accessed on 14 August 2023) and NEST GPU (https://github.com/nest/nest-gpu, accessed on 14 August 2023) provide example implementations of the cortical microcircuit model inside their respective source code repositories. Additionally, GeNN provides their own implementation of the microcircuit along with the data used for their PyGeNN publication [17] in the corresponding publicly available GitHub repository (https://github.com/BrainsOnBoard/pygenn_paper, accessed on 14 August 2023). Furthermore, to correctly compare the performance of the simulation, we adapt the existing scripts so that the overall behavior remains the same. In particular, we disabled spike recordings and we enabled optimized initialization of membrane potentials as in [26].

The second model presented in Section 2.4 is implemented for NEST GPU (onboard), and different connection rules can be chosen for the simulation. See the Data Availability Statement for further details on how to access the specific model versions used in this publication.

2.6. Simulation Phases

On the coarse level, we divide a network simulation into two successive phases: network construction and simulation. The network construction phase encompasses all steps until the actual simulation loop starts. To assess different contributions to the network construction, we further divide this phase into stages. The consecutively executed stages in the NEST implementations (both CPU and GPU versions) follow the same pattern as follows:

• Initialization is a setup phase in the Python script for preparing both model and simulator by importing modules, instantiating a class, or setting parameters, etc.
  •    import nestgpu
• Node creation instantiates all the neurons and devices of the model.
  •    nestgpu.Create()
• Node connection instantiates the connections among network nodes.
  •    nestgpu.Connect()
• Calibration is a preparation phase that orders the connections and initializes data structures for the spike buffers and the spike arrays just before the state propagation begins. In the CPU code, the pre-synaptic connection infrastructure is set up here. This stage can be triggered by simulating just one time step h.
  •    nestgpu.Simulate(h)
  Previously, the calibration phase of NEST GPU was used to finish moving data to the GPU memory and instantiate additional data structures like the spike buffer (cf. Section 2.3). Now, as no data transfer is needed and connection sorting is carried out instead (cf. Section 2.2), the calibration phase is now conceptually closer to the operations carried out in the CPU version of NEST [37].

In GeNN, the network construction is decomposed as follows:

• Model definition defines neurons and devices and synapses of the network model.
  •    from pygenn import genn_model
  •    model = genn_model.GeNNModel()
  •    model.add_neuron_population()
• Building generates and compiles the simulation code.
  •    model.build()
• Loading allocates memory and instantiates the network on the GPU.
  •    model.load()

Timers at the level of the Python interface assess the performance of the three different simulation engines. This has the advantage of being agnostic to the implementation details of each stage at the kernel level, including any overhead of data conversion required by the C++ API, and close to the actual time perceived by the user.

2.7. Validation of the Proposed Network Construction Method

The generation of random numbers for the probabilistic connection rules differs between the previous and the novel approach for network construction in NEST GPU. This means that the connectivity resulting from the same rule with the same parameters is not identical but only matches on a statistical level. It is, therefore, necessary to determine that the network dynamics is qualitatively preserved.

Using the cortical microcircuit model, we validate the novel method against the previous one using a statistical analysis of the simulated spiking activity data. To this end, we apply a similar validation procedure to that proposed in [9,13], where the GPU version of NEST was compared to the CPU version as a reference. We follow the example of [26,27,29,32] and compute for each of the eight neuron populations three statistical distributions to characterize the spiking activity as follows:

• Time-averaged firing rate for each neuron;
• Coefficient of variation of inter-spike intervals (CV ISI);
• Pairwise Pearson correlation of the spike trains obtained from a subset of 200 neurons for each population.

These distributions are then compared for the two different approaches for network construction, as detailed in Appendix B.

3. Results

This section evaluates the performance of the proposed method for generating connections directly in GPU memory using the reference implementation NEST GPU (onboard). For the cortical microcircuit model, we compare the network construction time and the real-time factor of the simulations obtained with the novel method to NEST GPU (offboard) (i.e., the simulator version employing the previous algorithm of instantiating the connections first on the CPU), the CPU version of the simulator NEST [1] and the code-generation based simulator GeNN [6].

With the two-population network model, we assess the network construction time upon scaling the number of neurons and the number of connections per neuron. Refer to Section 2.4 for details on the network models.

3.1. Cortical Microcircuit Model

Figure 3 directly compares the two approaches for network construction implemented in NEST GPU, i.e., onboard and offboard, in terms of the network construction time (panel A) and the real-time factor obtained using a simulation of the network dynamics (panel B). Panel A shows that the novel method for network construction enables a speed up by two orders of magnitude with respect to the previous network construction algorithm. The overhead of the offboard method (used in [9,13]) transferring the network from CPU to GPU becomes obsolete with the proposed approach to generate the connections directly on the GPU. Moreover, the onboard version shows lower network construction times across all hardware configurations without compromising the simulation times (panel B). An additional detail to take note of with the novel algorithm is that the calibration phase is now by far the longest compared to the node creation and node connection (3–5 times longer, depending on the hardware used). However, this is only due to the fact that both the creation and connection phases are now only used to instantiate data structures in GPU memory, whereas the calibration phase takes charge of the connection sorting as described in Section 2.2. Both versions have real-time factors of less than one second (sub-real-time simulation), thus showing also an improvement in the simulation time compared to the results of [9] obtained with the prototype NeuronGPU library. Additionally, in some cases, it is possible to see a small improvement when simulating using the novel network construction approach due to some code optimization related to the simulation phase. While the network construction times are independent of the choice of external drive, the DC input as expected leads to faster simulations of the network dynamics compared to the Poisson generators. Comparing the different hardware configuration, the smallest real-time factor obtained with NEST GPU (onboard) is achieved with DC input on the latest consumer GPU RTX 4090, $0.386\left(0.001\right)$ (mean (standard deviation)). The respective result for Poisson input is $0.4707\left(0.0008\right)$. For completeness, we also measure the real-time factor of NEST and GeNN simulations using the same framework used for Figure 3B. These results are shown in

Table 2. Performance metrics of NEST and NEST GPU when using Poisson spike generators to drive external stimulation to the neurons of the model. All times are in seconds with notation (mean (standard deviation)). Simulation time is calculated for a simulation of 10 s of biological time.

Metrics	NEST GPU (onboard)				NEST GPU (offboard)				NEST 3.3 (CPU)
	V100	A100	2080Ti	4090	V100	A100	2080Ti	4090	2 $\times$ 64 Cores
Initialization	$5.08\left(0.15\right)$ × 10−4	$1.44\left(0.15\right)$ × 10−3	$1.71\left(0.09\right)$ × 10−4	$1.91\left(0.04\right)$ × 10−4	$4.99\left(0.08\right)$ × 10−4	$1.44\left(0.15\right)$ × 10−3	$1.66\left(0.12\right)$ × 10−4	$1.84\left(0.05\right)$ × 10−4	$0.02\left(0.01\right)$
Node creation	$0.02\left(0.004\right)$	$0.03\left(0.007\right)$	$1.63\left(0.09\right)$ × 10−3	$1.94\left(0.02\right)$ × 10−3	$3.02\left(0.02\right)$	$3.32\left(0.05\right)$	$1.93\left(0.04\right)$	$1.781$(0.018)	$0.39\left(0.02\right)$
Node connection	$0.105$(0.0003)	$0.08\left(0.002\right)$	$0.308$(0.009)	$0.1600$(0.0005)	$54.65\left(0.11\right)$	$56.02\left(0.27\right)$	$41.16\left(0.28\right)$	$44.2\left(0.7\right)$	$1.72\left(0.17\right)$
Calibration	$0.57$(0.001)	$0.408$(0.005)	$0.602$(0.0006)	$0.3638$(0.0004)	$1.99\left(0.01\right)$	$2.06\left(0.01\right)$	$2.202\left(0.01\right)$	$2.183$(0.014)	$2.39\left(0.01\right)$
Network construction	$0.708$(0.001)	$\mathbf{0}.\mathbf{52}(\mathbf{0}.\mathbf{08})$	$0.91\left(0.09\right)$	$\mathbf{0}.\mathbf{5259}$(0.0008)	$59.67\left(0.13\right)$	$61.41\left(0.27\right)$	$45.29\left(0.32\right)$	$48.2\left(0.7\right)$	$4.54\left(0.18\right)$
Simulation (10 s)	$8.82\left(0.09\right)$	$8.54\left(0.03\right)$	$8.504\left(0.02\right)$	$4.707$(0.008)	$9.28\left(0.04\right)$	$8.94\left(0.02\right)$	$8.64\left(0.01\right)$	$5.219$(0.018)	$12.66\left(0.08\right)$

,

Table 3. Performance metrics of NEST and NEST GPU when using DC input to drive external stimulation to the neurons of the model. All times are in seconds with notation (mean (standard deviation)). Simulation time is calculated for a simulation of 10 s of biological time.

Metrics	NEST GPU (onboard)				NEST GPU (offboard)				NEST 3.3 (CPU)
	V100	A100	2080Ti	4090	V100	A100	2080Ti	4090	2 $\times$ 64 Cores
Initialization	$5.04\left(0.13\right)$ × 10−4	$1.44\left(0.08\right)$ × 10−3	$1.75\left(0.16\right)$ × 10−4	$1.97\left(0.09\right)$ × 10−4	$5.1\left(0.4\right)$ × 10−4	$1.5\left(0.4\right)$ × 10−3	$1.62\left(0.04\right)$ × 10−4	$1.86\left(0.04\right)$ × 10−4	$0.018$(0.003)
Node creation	$7.0\left(0.5\right)$ × 10−3	$6.6\left(0.3\right)$ × 10−3	$1.43\left(0.13\right)$ × 10−3	$1.64\left(0.04\right)$ × 10−3	$3.01\left(0.02\right)$	$3.28\left(0.03\right)$	$1.91\left(0.02\right)$	$1.79\left(0.03\right)$	$0.392$(0.003)
Node connection	$0.1028$(0.0004)	$0.0790$(0.0013)	$0.31\left(0.02\right)$(0.009)	$0.1538$(0.0005)	$54.65\left(0.17\right)$	$55.89\left(0.19\right)$	$40.8\left(0.5\right)$	$44.2\left(0.7\right)$	$1.53\left(0.07\right)$
Calibration	$0.5785$(0.0013)	$0.412$(0.008)	$0.6011$(0.0006)	$0.3632$(0.0003)	$1.993$(0.012)	$2.059$(0.016)	$2.194$(0.015)	$2.181$(0.015)	$2.352$(0.005)
Network construction	$0.6888$(0.0018)	$\mathbf{0}.\mathbf{499}(\mathbf{0}.\mathbf{10})$	$0.91\left(0.02\right)$	$\mathbf{0}.\mathbf{5189}$(0.0005)	$59.65\left(0.19\right)$	$61.23\left(0.19\right)$	$44.9\left(0.5\right)$	$48.1\left(0.7\right)$	$4.30\left(0.07\right)$
Simulation (10 s)	$6.36\left(0.02\right)$	$7.32\left(0.05\right)$	$5.61\left(0.03\right)$	$3.86\left(0.01\right)$	$6.530$(0.012)	$7.43\left(0.02\right)$	$5.604$(0.016)	$3.953$(0.013)	$7.77\left(0.15\right)$

and

Table 4. Performance metrics of GeNN. All times are in seconds with notation (mean (standard deviation)). Simulation time is calculated for a simulation of 10 s of biological time.

Metrics	GeNN
	V100	A100	2080Ti	4090
Model definition	$1.704\left(0.008\right)$ × 10−2	$1.75\left(0.01\right)$ × 10−2	$1.07\left(0.01\right)$ × 10−2	$1.094\left(0.007\right)$ × 10−2
Building	$13.87\left(0.36\right)$	$14.301\left(0.72\right)$	$7.25\left(0.04\right)$	$8.15\left(0.04\right)$
Loading	$0.77\left(0.02\right)$	$0.85\left(0.006\right)$	$0.51\left(0.01\right)$	$0.445\left(0.015\right)$
Network construction (no building)	$0.79\left(0.02\right)$	$0.85\left(0.006\right)$	$0.52\left(0.01\right)$	$0.456\left(0.015\right)$
Network construction	$14.67\left(0.35\right)$	$15.15\left(0.72\right)$	$7.78\left(0.04\right)$	$8.61\left(0.04\right)$
Simulation (10 s)	$6.48\left(0.01\right)$	$5.39\left(0.01\right)$	$7.007\left(0.01\right)$	$2.719\left(0.006\right)$

and depicted in Appendix C.

Figure 4 compares the network construction times of the full-scale cortical microcircuit model obtained using NEST GPU (onboard), NEST 3.3, and GeNN 4.8.0 on different hardware configurations; for details on the hardware and software configurations, see Section 2.5. The settings are the same as in Figure 3. While panel A resolves the contributions of the different stages (defined in Section 2.6) using a linear y-axis, panel B shows the same data with logarithmic y-axis to facilitate a comparison of the absolute numbers. As mentioned in Section 2.4, the external input to the cortical microcircuit implementations in both the CPU and GPU versions of NEST can be provided via either generators of Poisson signals or DC input. We run simulations comparing both approaches. However, Figure 4 only shows the results for the case of Poisson generators because the network construction times with DC input are similar. GeNN, in contrast, mimics the incoming Poisson spike trains via a current directly applied to each neuron. Both NEST GPU (onboard) and GeNN (without the building phase) achieve fast network construction times of less than a second. The fastest overall network construction takes $0.499\left(0.10\right)$ s, as measured with NEST GPU (onboard) using DC input on the A100 GPU, the data center GPU of the latest architecture tested. The time measured using the RTX 4090 is also compatible with the A100 result; the measured times with the V100 and the consumer GPU RTX 2080 Ti are also close. Table 2, Table 3 and Table 4 provide the measured values for reference.

Hitherto, we discussed the performance for both network construction and simulation of NEST GPU (onboard) compared to NEST GPU (offboard), NEST, and GeNN. Turning on the statistical analysis of the simulated activity, data show good agreement between NEST GPU (offboard) and NEST GPU (onboard) as well as between NEST GPU (onboard) and NEST 3.3. That means that differences between the compared simulator versions are of the same order as the fluctuations due to the choice of different seeds in either of the codes (see Section 2.7 and Appendix B). The novel approach constructs the network model in around half a second, whereas the same task using the previous network construction method took around a minute. In this validation process, we, therefore, experience a 15% reduction in the overall time to solution when obtaining the simulated activity of $600\mathrm{s}$ of biological time from NEST GPU (onboard) with respect to NEST GPU (offboard). More details on the time needed to perform this study are reported in the Appendix B.

3.2. Two-Population Network

The two-population network described in Section 2.4 is designed to evaluate the scaling performance of the proposed network construction method. To this end, we perform simulations on NEST GPU (onboard) varying the number of neurons and the number of connections per neuron. The scaling performance of NEST GPU (offboard) has been evaluated on [9] for a balanced network model. We opted for a total number of neurons in the network (N) ranging from 1000 to 1,000,000 and a target number of connections per neuron (K) ranging from 100 to 10,000.

To enable the largest networks, benchmarks are performed on the JURECA-DC cluster, which is equipped with the GPUs with the largest GPU memory (i.e., the NVIDIA A100 with 40 GB) among the systems described in Table 1. Figure 5 shows the network construction times using the fixed_total_number connection rule and ranging the number of neurons and connections per neuron. The performance obtained using the fixed_indegree and fixed_outdegree connection rules are totally compatible with the ones shown in this figure, and the respective plots are available in Appendix D for completeness.

As can be seen, the value of network construction time for the network with ${10}^6$ neurons and ${10}^4$ connections per neuron is not shown because of the lack of GPU memory. Using an NVIDIA A100 GPU, we can, thus, say that this method enables the construction of networks with up to an order of magnitude of ${10}^9$ connections. The largest network tested on this GPU comprised $3\times {10}^5$ neurons with $K=10,000$ connections per neuron (data not shown).

4. Discussion

It takes less than a second to generate the network of the cortical microcircuit model [24] with the GPU version of NEST using our proposed dynamic approach for creating connections directly in GPU memory on any GPU device tested. That is two orders of magnitude faster than the previous algorithm, which instantiates the connections first on the CPU and copies them from RAM to GPU memory just before the simulation starts (Figure 3). The reported network construction times are also shorter compared to the CPU version of NEST and the code generation framework GeNN (Figure 4); if code generation and compilation are not required in GeNN, the results of NEST GPU and GeNN are compatible. The time to simulate the network dynamics after network construction is not compromised by the novel approach.

The latest data center and consumer GPUs (i.e., A100 and RTX 4090, respectively) show the fastest network constructions as expected, approximately $0.5$ s. We observe the shortest simulation times on the RTX 4090 and attribute this result to the fact that the kernel design of NEST GPU particularly benefits from the high clock speeds of this device (cf. Section 2.5). Contrary to expectation, our simulations with DC input on the A100 are slower compared to the V100 although the former has higher clock speeds; an investigation of this observation is left for future work.

For models of the size of the cortical microcircuit, the novel approach renders the contribution of the network construction phase to the absolute wall-clock time negligible, even for short simulation durations. Further performance optimizations should preferentially target the simulation phase. Our result that GeNN currently simulates faster than NEST GPU indicates that there is room for improvement, which could possibly be exploited via further parallelizations of the simulation kernel.

The evaluation of the scaling performance with the two-population network on the A100 shows that the network construction time is dominated by the total number of connections (i.e., $N\times K$, Figure 5) and mostly independent of the connection rule used. The maximum network size that can be simulated depends on the GPU memory of the card employed for the simulation. Future generation GPU cards with more memory available will enable the construction of larger or denser networks of spiking neurons and, at the same time, give reasons to expect further performance improvements with novel architectures and the possibility of an even higher degree of parallelism. Nonetheless, this conclusion is affected by the structure of the connection objects, as each object contains multiple data like the weight and the delay (cf. Section 2.2); when using natural connection density, GPU memory would be consequently filled up faster. However, if one considers simple connection objects like the ones used in weightless neural networks such as the Ring Probabilistic Logic Neural Networks discussed in [38], the maximum network size would naturally increase. The novel approach is currently limited to simulations on a single GPU, and future work is required to extend the algorithm to employ multiple GPUs as achieved with the previous algorithm [13].

Further improvements to the library may also expand upon the available connection rules and more flexible control via the user interface. At present, the pairwise Bernoulli connection routine [21] is not available; this is because the onboard construction method requires a precise number of connections that must be allocated at once in order to not waste any GPU memory. The pairwise Bernoulli connection routine implies that this number is not known; hence, additional heuristics would be required to optimize memory usage. Autapses and multapses are currently always allowed in NEST GPU; therefore, another useful addition would be the possibility to prohibit them (for example, using a flag as in the CPU version of NEST).

In conclusion, we propose a novel algorithm for network construction, which dynamically creates the network exploiting the high degree of parallelism of GPU devices. It enables short network construction times comparable to code generation methods and the advantageous flexibility of run-time instantiation of the network. This optimized method makes the contribution of the network construction phase in network simulations marginal, even when simulating highly-connected large-scale networks. As discussed in [39], this is especially interesting for parameter scan applications, where a high volume of simulations needs to be tested, and any additional contribution to the overall execution time of each test aggregates considerably and slows down the exploration process. Future work can investigate the extension of this algorithm for multi-GPU simulations, incorporate further connection rules, and optimize the simulation kernel to enable the fast network construction and simulation of large networks approaching the size of the human brain.