Activeness: A Novel Neural Coding Scheme Integrating the Spike Rate and Temporal Information in the Spiking Neural Network

Abstract

In neuromorphic computing, the coding method of spiking neurons serves as the foundation and is crucial for various aspects of network operation. Existing mainstream coding methods, such as rate coding and temporal coding, have different focuses, and each has its own advantages and limitations. This paper proposes a novel coding scheme called activeness coding that integrates the strengths of both rate and temporal coding methods. It encompasses precise timing information of the most recent neuronal spike as well as the historical firing rate information. The results of basic characteristic tests demonstrate that this encoding method accurately expresses input information and exhibits robustness. Furthermore, an unsupervised learning method based on activeness-coding triplet spike-timing dependent plasticity (STDP) is introduced, with the MNIST classification task used as an example to assess the performance of this encoding method in solving cognitive tasks. Test results show an improvement in accuracy of approximately 4.5%. Additionally, activeness coding also exhibits potential advantages in terms of resource conservation. Overall, activeness offers a promising approach for spiking neural network encoding with implications for various applications in the field of neural computation.

1. Introduction

In the field of neuromorphic computing, spiking neural networks have garnered significant attention as a biologically inspired computational model and are regarded as the fourth generation of neural networks [1]. This model is based on the spiking behavior of biomimetic neurons, allowing for the processing of spatiotemporal information while efficiently conserving energy. The representation of information by spiking neurons, known as neural coding [2], is closely associated with the input encoding, information transmission, learning algorithms, and output readout of spiking neural networks.

Currently, one of the predominant approaches for encoding the activity of spiking neurons in spiking neural networks is rate coding [3]. This method is simple and robust, as it considers the firing rate of neurons. However, rate coding suffers from slow information transmission and lower processing efficiency due to the need for statistical calculations over longer time windows. Additionally, rate coding only considers the firing rate of neurons, disregarding the crucial impact of precise spike timing on network activity, which contradicts relevant physiological findings [4].

Another widely adopted encoding approach is temporal coding, which utilizes precise spike timing to represent information. Examples include time-to-first-spike (TTFS) coding [5] and relative spike latency (RST) coding [6]. These methods can speed information transmission and offer energy-efficient single-spike requirements. However, they exhibit poorer robustness against interference, and the sparse firing rates make it challenging to directly extend such networks to multiple layers. Moreover, other encoding methods, such as phase coding [7] and burst coding [8], have been proposed to seek simple, efficient, and robust encoding strategies for spiking neurons.

This paper proposes a novel spike-based encoding method called activeness, which integrates the historical activity information of spiking neurons. Activeness quantifies the discharge of spiking neurons using a single scalar value, effectively integrating the historical firing rate and precise spike timing information of the neurons while maintaining a limited resource cost. This encoding method can be directly applied to input encoding, learning algorithms, and output decoding in spiking neural networks. It enables resource savings, complexity reduction, and improved learning efficiency in network operations.

In the following sections, we begin by providing an exposition and analysis of the definition and computation method of activeness. Subsequently, we explore the representation capability and robustness of Activeness through experimental investigations, as well as its performance in classical classification tasks. Finally, we summarize and discuss the potential applications of activeness.

2. Proposed Methods

In the biological brain, neurons process and transmit information through seemingly instantaneous, stochastic, and disordered firing activities. Understanding the mechanisms of neuronal firing in the biological brain is of great significance for cognitive neuroscience and deciphering brain function. In spiking neural networks inspired by the brain, one of the primary tasks is to represent information at the level of neurons, which is known as neural coding.

A good neural coding scheme should accurately represent the input information of neurons (accuracy), yield consistent coding outcomes for different signal frequencies or intensities (robustness), maximize the inclusion of information to enhance coding efficiency (efficiency), and provide an explanation for the encoding mechanisms that align with the knowledge of neuroscience (interpretability).

Neuroscientific research has revealed that neuronal firing is a complex series of electrochemical reactions triggered by excitation. Sustained neuronal firing leads to the accumulation of calcium ions within the cell [9,10], and calcium ions participate in various processes of signal transmission and activity modulation [11]. Neuronal firing activity also continuously regulates the synthesis and degradation of proteins within the cell [12], and these proteins are involved in nearly all aspects of neuronal activity [13]. The changes in calcium ions and proteins within the neuron have a significant impact on synaptic plasticity and neuronal function. However, these changes are not instantaneous. They occur gradually through the accumulation or degradation process, which is the result of long-term neuronal activity. This is different from the rapid processes occurring in synapses, such as the release of neurotransmitters controlled by the activity states of pre and postsynaptic neurons, and the switching of ion channels. These rapid processes are the transient activities of neurons. Nevertheless, there is a close relationship between these two processes. The transient activities cause changes in regulatory substances (such as continuous stimulation leading to the synthesis of transcription factors and new proteins, which may result in the formation of new synapses [14]), and these changes in substances then affect the intensity of each transient activity (such as the concentration of calcium ions affecting the movement of synaptic vesicles and the release of neurotransmitters [15], which directly influences the action of postsynaptic neurons).

The previous encoding schemes, including rate coding, temporal coding, phase coding, and sequence coding, mainly reflect the transient activity of neurons and lack emphasis on the historical context. To achieve an encoding scheme that captures both transient changes and historical activity, this paper proposes a novel neural encoding scheme called activeness coding. The specific definition is as follows.

Activeness is a metric used to quantify the activity level of a neuron. When an action potential arrives, the activeness, denoted as A, increases by R and subsequently decays to zero with a time constant ${\tau }_{\mathrm{A}}$. The step size of activeness, R, is determined by the time interval since the last discharge. When a neuron fires, R is updated to 1 and then decays to zero with a time constant ${\tau }_{\mathrm{R}}$. The computation method is

$$\left\{\begin{array}{c}\frac{dA\left(t\right)}{dt}=-\frac{A\left(t\right)}{{\tau }_{\mathrm{A}}},\hfill \\ \frac{dR\left(t\right)}{dt}=-\frac{R\left(t\right)}{{\tau }_{\mathrm{R}}}.\hfill \end{array}\right.$$

(1)

$$\left\{\begin{array}{c}A\left(t\right)=A\left(t\right)+R(t-ϵ),\hfill \\ R\left(t\right)=1,\hfill \end{array}\right.t=t^{\mathrm{f}}.$$

(2)

where $t^{\mathrm{f}}$ represents the moment of neuronal firing. The small positive constant $ϵ$ in Equation (2) is used to ensure that the activeness A of the neuron is updated before R is set to 1. This guarantees that the increase in A is correlated with the time interval between neuronal firings. In other words, as the time interval between two firings becomes shorter, R approaches 1, whereas for longer intervals, R approaches 0. The relationships among the membrane potential V, the step size R, and the activity level A of the neuron are illustrated in Figure 1.

Activeness is a scalar value that is dimensionless. It is not only influenced by the most recent discharge of the neuron but also takes into account the impact of all previous discharges. The step size R reflects the precise timing of the most recent spike. When the input remains constant, the cumulative sum of step sizes R can reflect the average number of spike occurrences within a certain time period, which is consistent with the spike rate. Therefore, we believe that activeness encompasses both spike rate information and precise spike timing information. Detailed experimental validation and analysis are presented in Section 3.

In the activeness formula (see Equation (1)), ${\tau }_{\mathrm{R}}$ determines the influence of the most recent spike on the activeness, while ${\tau }_{\mathrm{A}}$ determines the influence of the historical spike activity on the activeness. Typically, ${\tau }_{\mathrm{R}}$ is not greater than ${\tau }_{\mathrm{A}}$, as otherwise, the activeness encoding results would exhibit significant fluctuations.

It is worth noting that the activeness calculation model proposed in this paper exhibits similarities with the output voltage waveform of an LIF (leaky integrate-and-fire) neuron’s RC integrator in some intervals. However, their fundamental natures are entirely different and they are not interchangeable. First and foremost, they address different problems. LIF neurons (with synaptic models included) deal with the dynamic relationship between input spikes and the membrane potential of neurons. In contrast, activeness coding quantifies the degree of neural activity. Furthermore, they correspond to different neurophysiological processes. LIF neurons primarily simulate changes in the neuronal membrane potential, whereas, as mentioned earlier, activeness coding abstracts the processes related to internal substances like calcium ions and proteins within neurons. Moreover, they operate on different time scales. The membrane time constant of LIF neurons typically falls in the range of tens of milliseconds, while activeness coding integrates information over longer time scales, typically in the hundreds of milliseconds, to more accurately represent the activity characteristics of neurons. Finally, there are differences in computational details. Although both LIF neurons and activeness involve nonlinear integrators, unlike the fixed time constant of LIF’s RC circuit, the integration time constant in activeness is not constant but varies depending on the input spike pattern. Activeness coding also does not require a comparator similar to the membrane potential threshold used in LIF neurons.

3. Experiments and Evaluation

3.1. Basic Characteristics

As described in the previous section, activeness coding possesses a certain degree of biological interpretability. The calculation method of activeness allows it to naturally incorporate both the temporal information of the nearest spike and the activity information from all previous spikes, thus achieving high encoding efficiency. However, the accuracy and robustness of its encoding need to be examined through experiments.

To visualize the basic performance of activeness coding more directly, we conducted tests using the encoding of input neuron activity as an example. The data processing procedure for the testing system is illustrated in Figure 2.

In Figure 2, the input to the neuron, denoted as $D_{\mathrm{in}}$, is a real number, such as the pixel value of a gray-scale image in image classification tasks. Firstly, based on the specified peak firing rate $f_{\mathrm{P}}$, we calculate the firing rate of the neuron for the current input, which determines whether the membrane potential V at the current moment is at the resting state or action potential state. This enables the conversion of real-valued input into spiking output. Subsequently, we calculate the activeness A using Equations (1) and (2). Finally, we visualize the recorded membrane potential V and activeness A over the testing period. The test results are shown in Figure 3 and Figure 4.

Accuracy. When the input real numbers $D_{\mathrm{in}}$ are set to 63, 127, 191, and 255, with $f_{\mathrm{P}}=63.75$ Hz, ${\tau }_{\mathrm{R}}={\tau }_{\mathrm{A}}$ = 100 ms, the neuron is configured to discharge at an ideal constant rate. The membrane potential variations of the input neuron within 1000 ms are shown in the four waveform segments in Figure 3a. Meanwhile, the activeness gradually increases nonlinearly from an initial value of 0 over time, and after approximately 300 ms, it stabilizes. The growth rate of activeness is positively correlated with the firing rate of the neuron, i.e., with the input real values, as depicted by the four curves in Figure 3b, from bottom to top representing the four increasing input values. On a microscopic level, influenced by the stimulatory pulses, the activeness dynamically fluctuates between step increments and decays, with the amplitude of the fluctuations related to the time constants ${\tau }_{\mathrm{R}}$ and ${\tau }_{\mathrm{A}}$. However, the overall trend of the activeness corresponds to the input values. In other words, the activeness can accurately encode the input information. Even during the rising period of activeness (0–300 ms), the relative magnitudes of activeness reflect the relative sizes of the inputs. Although activeness is not as precise as temporal coding, it overcomes the dependency of rate coding on large time windows, thereby improving the information transmission rate.

Robustness. The actual firing patterns of biological neurons differ from the aforementioned ideal state and are characterized by inherent randomness. In neural computational models, the firing process is often treated as a series of stochastic events related to the input, which are commonly assumed to follow a Poisson distribution. Therefore, in this experiment, Poisson random numbers were used to generate firing pulses. When the inputs $D_{\mathrm{in}}$ are set to 63, 127, 191, and 255, with $f_{\mathrm{P}}=63.75$ Hz, ${\tau }_{\mathrm{R}}={\tau }_{\mathrm{A}}$ = 100 ms, the membrane potential variations of the neuron after Poisson coding are shown in Figure 4a. It can be observed that the generated discrete pulses have varying time intervals between them. Consequently, the activeness also exhibits significant fluctuations. By averaging the results of 10 experiments and calculating the standard deviation, the curve representing the changes in activeness is obtained, as depicted in Figure 4b. The solid lines in the figure represent the mean activeness curves for different inputs, while the translucent areas of the same color surrounding the lines indicate the range of data fluctuations. From the figure, it can be seen that the activeness exhibits a substantial amount of random fluctuations, but its mean values still demonstrate a strong correlation with the inputs. The four curves are clearly distinct, without overlap or intertwining, indicating the robustness of the activeness coding.

3.2. Classification Performance

To evaluate the performance of the spiking neural networks utilizing the activeness coding in cognitive tasks, we used handwritten digit recognition as an example. Introducing activeness coding into the three-layer spiking neural network established by Diehl and Cook [16], we investigated the changes in network performance. The architecture outlined by Diehl and Cook [16] encompasses an input layer, where input pixel values are encoded using Poisson-distributed spike trains. This is followed by an excitatory neuron layer, fully connected to the input layer, utilizing leaky integrate-and-fire (LIF) neurons along with conductance-based synaptic models. Subsequently, there is an inhibitory neuron layer implementing lateral inhibition connections to establish competition among excitatory neurons. The synaptic connection weights between the input layer and the excitatory neuron layer are adjusted using unsupervised exponential-weight-dependence spike-timing-dependent plasticity (STDP), coupled with an adaptive membrane potential threshold regulation mechanism to achieve network homeostasis. They implemented this network using the Python-based spiking neural network simulator Brian [17] and trained the network with the Modified National Institute of Standards and Technology (MNIST) [18] handwritten digit dataset, eventually achieving the best unsupervised SNN test accuracy on MNIST at that time.

Based on the network architecture proposed by Diehl and Cook, the structure of the network developed for this test is illustrated in Figure 5.

The MNIST dataset, used as the input for the network, consists of 60,000 hand-written digit images with a resolution of $28\times 28$, as well as 10,000 test images with the same resolution. The pixel values range from 0 to 255. Each input neuron receives the input from one pixel and generates output spikes using Poisson random numbers, which are then encoded using the activeness-based encoding scheme. Therefore, there are a total of 784 input neurons, encoding the 784 pixels of the input. The input layer neurons are excitatory and fully connected to 400 excitatory neurons. Each excitatory neuron drives one inhibitory neuron, which in turn is recurrently connected to the other 399 excitatory neurons, excluding the excitatory neuron that excites it, implementing a competitive mechanism. The entire network was built and operated using the upgraded neural simulator Brian2 [19].

The recurrent connection weights between the excitatory and inhibitory neuron layers are fixed, while the excitatory synaptic connections between the input layer and the excitatory neuron layer are plastic. In contrast to the approach in reference [16], this paper introduced an improvement to triplet-STDP [20] by proposing an activeness-based triplet-STDP rule for synaptic weight modification. The implementation method involves replacing the synaptic trace in the triplet-STDP calculation formula with the activeness of the presynaptic neuron and replacing the postsynaptic trace with the activeness of the postsynaptic neuron. Consequently, when a presynaptic neuron fires at time $t=t_{\mathrm{pre}}^{\mathrm{f}}$, the weight modification is given by $\mathrm{\Delta }\omega ={\gamma }_{\mathrm{pre}}{A}_{\mathrm{post}}\left(t_{\mathrm{pre}}^{\mathrm{f}}\right)$. Similarly, when a postsynaptic neuron fires at time $t=t_{\mathrm{post}}^{\mathrm{f}}$, the weight update is $\mathrm{\Delta }\omega ={\gamma }_{\mathrm{post}}{A}_{\mathrm{pre}}\left(t_{\mathrm{post}}^{\mathrm{f}}\right)A_{\mathrm{post}}\left(t_{\mathrm{post}}^{\mathrm{f}}\right)$. By integrating the above formulas, the synaptic weight update formulas are obtained as shown in Equations (3) and (4).

$$\omega \left(t\right)=\omega (t-1)+\mathrm{\Delta }\omega \left(t\right)$$

(3)

$$\mathrm{\Delta }\omega \left(t\right)=\left\{\begin{array}{cc}{\gamma }_{\mathrm{pre}}{A}_{\mathrm{post}}\left(t\right),\hfill & \hfill t=t_{\mathrm{pre}}^{\mathrm{f}},\\ {\gamma }_{\mathrm{post}}{A}_{\mathrm{pre}}\left(t\right)A_{\mathrm{post}}\left(t\right),\hfill & \hfill t=t_{\mathrm{post}}^{\mathrm{f}},\\ 0,\hfill & \hfill \mathrm{others}.\end{array}\right.$$

(4)

where ${\gamma }_{\mathrm{pre}}$ represents the learning rate of the presynaptic neuron, and ${\gamma }_{\mathrm{post}}$ represents the learning rate of the postsynaptic neuron. $A_{\mathrm{pre}}\left(t\right)$ denotes the activeness of the presynaptic neuron at the time t, and $A_{\mathrm{post}}\left(t\right)$ represents the activeness of the postsynaptic neuron at the time t.

Based on the above approach, the synaptic dynamics formula is defined in Brian2 as shown in Figure 6.

The code executed when the presynaptic neuron fires is shown in Figure 7.

The code executed when the postsynaptic neuron fires is shown in Figure 8.

The labeling assignment method during the model training phase and the classification result determination during the testing and inference phase are consistent with the method described in [16]. Therefore, a detailed description of these methods will not be reiterated here.

In this test, $f_{\mathrm{P}}$ was still set to 63.75 Hz. The determination of other parameters involved multiple repeated experiments, searching for a set of parameter combinations that achieved a relatively good classification performance. These obtained parameter values are shown in

Table 1. Parameters for the classification performance testing network.

Parameters	Values
${\tau }_{\mathrm{R}}$	20 ms
${\tau }_{\mathrm{A}}$	100 ms
$f_{\mathrm{P}}$	63.75 Hz
${\gamma }_{\mathrm{pre}}$	0.0001
${\gamma }_{\mathrm{post}}$	0.01

. It is important to note that this set of values may not necessarily represent the optimal parameter combination.

After three rounds of training, the synaptic weight matrix of size $784\times 400$ connecting the input layer to the excitatory neuron layer is reshaped into a matrix of size $(28\times 20)\times (28\times 20)$. The distribution of synaptic weights obtained from this rearrangement is shown in Figure 9.

At this stage, a testing accuracy of 91.50% was achieved on the test dataset. The comparison between these test results and the results from [16] are shown in

Table 2. Comparison of MNIST classification performance test results.

Works	Learning Rule	Accuracy
Diehl et al., 2015 [16]	Spike-based triplet STDP	87%
This work	Activeness-based triplet STDP	91.5%

. The experimental findings clearly demonstrate that by solely replacing the synaptic trace in STDP with activeness, while maintaining all other conditions identical, the MNIST classification accuracy is improved by 4.5%. This improvement validates the superiority of activeness coding in the context of the MNIST task.

When the number of training samples varies, the curve depicting the change in test accuracy is shown in Figure 10.

As shown in Figure 10, the test accuracy steadily increases with the number of training iterations. It is observed that even after a single training epoch, the test accuracy achieved is comparable to that of the reference experiment conducted for three epochs. This finding further demonstrates the efficiency of activeness coding.

4. Discussion

4.1. Comparison

The activeness coding method is inspired by synaptic traces. Synaptic traces are a classic mathematical model used in spiking neural networks to characterize the impact of synaptic history on synaptic plasticity [21]. However, the two are fundamentally different, primarily in the following aspects:

Structural differences. Synaptic traces record the impact of neural activity on synapses and are typically defined as variables within the synapse. They include both pre-synaptic and post-synaptic traces. In contrast, activeness coding mainly considers the potential influence of a neuron’s own historical activity on itself and is only relevant to the neuron (soma). It is simpler and more straightforward.

Computational differences. Synaptic traces can be either all-to-all or only consider the nearest spiking event. Typically, constant values are accumulated in their calculations. On the other hand, the dynamic changes in the accumulation of activeness amplify the impact of precise discharge timing, thereby improving the efficiency of network operation.

Resource requirements. Learning based on synaptic traces requires maintaining two to three trace variables per synapse. In activeness encoding, only two relevant variables need to be maintained per neuron. Taking the classification performance test network in Section 3 as an example, using triplet STDP, the network would require a total of $784\times 400\times 3=940,800$ trace variables. However, when using activeness coding, the network would only require $(784+400)\times 2=2368$ relevant variables, which is approximately 1/400th of the previous approach. As the network size increases, the resource savings from this improvement become even more significant.

4.2. Potential Future Research

As a concise, macroscopic, efficient, and robust encoding method, activeness coding holds promise beyond input encoding and can be applied to the learning algorithm and network output inference.

Taking learning algorithms as an example, with the expressive power of activeness coding in a macroscopic and robust manner, it is possible to improve weight update strategies based on the precise discharge timing of pre- and post-synaptic neurons in unsupervised learning, supervised learning, and reward-based reinforcement learning algorithms. This improvement can be achieved by adopting rules based on activeness difference, leading to more stable network activity and faster learning convergence.

In terms of network output inference, many networks require additional classifiers or statistical computations to obtain human-understandable operational results. However, activity encoding provides a single quantized value that directly reflects the activity of output layer neurons, making the network’s operational results readily interpretable.

In addition to using spiking neural networks with activeness coding for image classification tasks, we are also exploring its potential in cross-modal information perception and integration. In the future, the proposed method could be further investigated to tackle various cognitive tasks.

5. Conclusions

This paper introduces a novel encoding method for spiking neurons, termed activeness encoding. This method is inspired by the cumulative changes in calcium ions and proteins within neurons that influence neural activity. Activeness combines precise timing information of the most recent spike with the entire history of spike events, demonstrating good encoding accuracy and robustness in basic characteristic tests. The results of image classification tasks show that activeness-based learning improves the testing accuracy on the MNIST dataset by approximately 4.5% compared to trace-based learning. Furthermore, activeness exhibits potential advantages in terms of computational power and storage capacity requirements.