IC-SNN: Optimal ANN2SNN Conversion at Low Latency

Abstract

The spiking neural network (SNN) has attracted the attention of many researchers because of its low energy consumption and strong bionics. However, when the network conversion method is used to solve the difficulty of network training caused by its discrete, too-long inference time, it may hinder the practical application of SNN. This paper proposes a novel model named the SNN with Initialized Membrane Potential and Coding Compensation (IC-SNN) to solve this problem. The model focuses on the effect of residual membrane potential and rate encoding on the target SNN. After analyzing the conversion error and the information loss caused by the encoding method under the low time step, we propose a new initial membrane potential setting method and coding compensation scheme. The model can enable the network to still achieve high accuracy under a low number of time steps by eliminating residual membrane potential and encoding errors in the SNN. Finally, experimental results based on public datasets CIFAR10 and CIFAR100 also demonstrate that the model can still achieve competitive classification accuracy in 32 time steps.

1. Introduction

Artificial neural networks (ANNs) have achieved great success in the past few years, although they differ structurally from the brain in terms of calculation methods and learning rules. Although some technologies similar to pruning, quantization, and compression have been proposed, ANN still faces problems of high power consumption and complex calculations [1,2,3]. The spiking neural network (SNN) is a third-generation neural network, and it has gained widespread attention due to its low power consumption. Benefiting from its biological inspiration, SNNs use spike-based binary values to pass information between neurons [4]. Figure 1 represents the overall structure of SNN and the workflow on spiking neurons. Davd et al. [5] carried out detailed calculations on the energy consumption of a single neuron, proving that the brain’s energy consumption is low. Maass et al. [6] analyzed the capabilities of cortical microcircuits for real-time computing and demonstrated that SNNs have powerful computing capabilities. The binary nature of SNN enables it to use accumulated operations instead of expensive multiply accumulated operations during network propagation, thereby reducing the amount of network computation. In the application of edge devices, the low calculation load of SNN has a great advantage. In addition, in some specific neuromorphic chips, SNN can run in an event-driven manner. When not receiving spikes, neurons will remain dormant, saving a lot of power consumption. However, SNNs struggle to achieve similar accuracy in traditional machine learning tasks. On the one hand, there is currently no conclusive understanding about the learning process in the brain, and it is far from enough to simply simulate brain neurons. On the other hand, the learning algorithms of SNNs are still in the developmental stage and lack effective learning algorithms.

Research on learning algorithms for SNNs can be roughly divided into three categories: learning algorithms based on synaptic plasticity, surrogate gradient-based backpropagation, and ANN-to-SNN conversion. As SNN is biologically credible, some researchers believe that it should follow biological mechanisms to train SNNs. They tend to apply the neuronal mechanism in the biological brain to SNNs. The network is trained by using Hebbian, Spike-Timing-Dependent Plasticity (STDP), Self-backpropagation (SBP), and other neuron mechanisms [7,8,9]. Inspired by backpropagation (BP), which excels in ANNs, researchers try to use BP in the training of SNNs [10]. They convert non-differentiable pulse trains into differentiable values, and build surrogate gradient functions to achieve backpropagation in space and time [11,12]. Furthermore, inspired by transfer learning, the researchers trained a topological and homogeneous non-spiking neural network in advance and transferred the weights to the spiking neural network to achieve approximate accuracy. There are differences between different networks and neurons, so it is necessary to weigh thresholds and weights in the conversion process, as well as eliminate errors such as clipping and quantization [13,14,15].

In this paper, we analyze the error of the network in the conversion process and propose an algorithm to initialize the membrane potential. This algorithm can sufficiently reduce the error in the conversion process. To cope with neural networks at low time steps, we use the ANN-SNN transformation as an initialization step and initialize the membrane potentials of neurons in each layer based on the transformed network and the training set. In addition, we propose a compensation tactic for the adopted rate coding. The tactic is based on the time step and combines the compensation part of each layer with the initialized membrane potential, which reduces the information loss to a certain extent.

The main contributions of this work are as follows:

• We theoretically analyze the error of ANN2SNN, which explains why information is lost at low time steps to a certain extent. We propose an algorithm to initialize membrane potential to eliminate the error term at low latency.
• We analyze the coding methods and propose a coding compensation method combined with the membrane potential. The novel coding compensation method can reduce information loss.
• Experiments are conducted on datasets based on two traditional network models. Experimental results demonstrate the validity of the new algorithm. The results show that our approach can effectively reduce the conversion error and obtain a high accuracy at low latency.

2. Related Work

The learning algorithm of SNN is one factor that restricts its network performance (e.g., accuracy, time step). Current learning algorithms can roughly be divided into three categories. We mainly introduce the related research work in recent years.

2.1. Learning Algorithms Based on Synaptic Plasticity

Based on the biological credibility of SNNs, it is feasible to transfer the neuron learning mechanism in biology to the network. Masquelier et al. [16] used STDP rules in feedforward spiking neural networks, which showed that temporal encoding may be the key to understanding the fantastic processing speed achieved by the visual system and that STDP can lead to fast and selective responses. By analyzing the learning effect of STDP, Legenstein et al. [17] realized that neurons that have learned through STDP can classify the firing patterns in space and time. Ponulak et al. [18] proposed a remote supervised method (ReSuMe). They demonstrated that the technique could train neurons to reproduce spike trains with controllable temporal offsets relative to a target template. Furthermore, they noted that spiking neurons may predict behavior through firing timing. Wade et al. [19] combined the Bienenstock Cooper Munro (BCM) learning rule with STDP and proposed the Synaptic Weight Association Training (SWAT) algorithm. Diehl et al. [20] used excitatory and inhibitory neurons and opted to use a lateral inhibition mechanism to reduce computational complexity. Tavanaei et al. [21] trained a biologically plausible convolutional filter and learned detector. The detector employs probabilistic neurons to extract independent features. Zhang et al. [22] first trained the network to a balanced membrane potential state, then maintained the network’s balance based on excitatory and inhibitory neurons. The neural networks adopted short-term plasticity (STP) to ensure overall balance to speed up convergence. Influenced by the self-organized propagation mechanism of hippocampal neurons, Zhang et al. [9] introduced a non-local synaptic modification method termed “self-BP” (SBP) for synaptic potentiation and depression. However, the various improvements described above for synaptic plasticity are often applied to shallow networks with five, four, or even three layers. To a certain extent, this indicates that the improvement based on synaptic plasticity pays too much attention to local information. It lacks the acquisition of global information, and is difficult to apply in deeper networks.

2.2. Surrogate Gradient-Based Backpropagation

The BP and the optimization algorithm of gradient descent commonly used in ANN are a good choice. Bohte et al. [11] tried to apply the BP algorithm to SNNs for the first time and proposed the Spike Propagation (SpikeProp) algorithm, which is suitable for SNN. The proposed of SpikeProp has led some researchers to focus on converting discrete, non-differentiable spike trains into continuous values and using backpropagation to train the network. Wu et al. [12,23] proposed space-time-based backpropagation. The method employs gradient approximation to avoid discontinuities in spiking activation functions and applies backpropagation through time (BPTT) on SNNs. Zhang et al. [24] used Temporal Spike Sequence Learning via Backpropagation(TSSL-BP) to bypass the non-differentiability of the spike activation function and improve the temporal learning accuracy. Fang et al. [25] improved the neuron model and added hyper-parameters to the backpropagation learning to enhance the expressive power of neurons. Xiao et al. [26] proposed that the average firing rate of SNN evolves to an equilibrium state over time and follows a fixed point equation. Equation-based implicit differentiation, computing the parameters’ gradient, avoids the non-differentiable spike function problem. The current research on such algorithms generally focuses on adjusting hyperparameters to make the gradient function more suitable for the spike firing rate. The study focuses on constructing the gradient function rather than the SNN itself. In addition, the gradient function also accumulates approximation errors. Thus, it is difficult to obtain a high-performance SNN.

2.3. ANN-to-SNN Conversion

Transfer learning can not only achieve higher initial performance, but it can also reduce training time significantly. Inspired by transfer learning, the researchers share the weight parameters of the source neural network to the target the neural network, thus saving the training time on the target neural network. Cao et al. [13] proposed an indirect training approach, which trained a homogeneous topological neural network and transferred trained neural network parameters to the spiking neuron network. Diehl et al. [27] explained that the conversion principle is a correlation between the value of the activation function of traditional neurons and the firing rate of spiking neurons, and proposed weight normalization for the difference in the output range of ANN and SNN. Rueckauer et al. [28] proposed a weight standardization based on the P quantile considering the influence of the singular point in the weight normalization process. Sengupta et al. [29] further analyzed the conversion principle. They pointed out that threshold balance and weight normalization can be regarded as the same operation, that is, to adjust the relationship between threshold and weight. Then, the existing standardization is improved by the standardization technology Spike Normalization based on SNN. For the linear regression problem, Kim et al. [30] proposed a more fine-grained channel normalization and neurons with negative thresholds. Park et al. [31,32] proposed time-based coding of Time-To-First-Spike (TTFS) and Temporal-Switch-Coding (TSC). Stöckl et al. [33] adopted dynamic neurons to fit the activation function used by the ANN network, which relieved the constraints of the activation function. Han et al. [34] analyzed the reset mechanism of neurons and refined the threshold balance. Deng et al. [35] analyzed the transformation error by recursively reducing it to a hierarchical sum. Ding et al. [36] proposed a best-fit curve to quantify the error of the conversion process. Rathi et al. [37] combined ANN2SNN with spike-based backpropagation and used the spike time for secondary training based on the transformed SNN to reduce inference time. Bu et al. [14] realized that the transformed SNN suffers from severe performance degradation with shorter time steps. By analyzing and deriving the estimated SNN activation function, a quantization clip-floor-shift activation function is proposed to replace the ReLU activation function in the source ANN. Bu et al. [38] set a trainable clipping upper bound in the ANN and obtained the optimal initial membrane potential based on the derivation results and clipping upper-bound. Wang et al. [15] employed quantization noise to train thresholds in networks with quantized clipping functions and employed BPTT in SNNs to calibrate weights, biases, and initial membrane potential to minimize residual potential. Early switching research focused on how to switch two different types of neurons, and each proposed a switching strategy. As this area of research progresses, most researchers are focusing on how to reduce the conversion loss and achieve accuracy as close to the ANN as possible. At present, researchers are gradually realizing that the long inference time brings a huge obstacle to the application of SNN. They have begun to analyze the error term at low latency and proposed various strategies to eliminate the error, thereby improving the network accuracy of SNN at low latency.

3. Method

In this section, we introduce the activation function and network output of the source ANN, as well as the neuron working mechanism and network output of the target SNN. After analyzing the network output of the target SNN, we obtain the error term produced during the transformation. For this conversion error, an algorithm for initializing the membrane potential is proposed. Furthermore, coding compensation is proposed to eliminate the errors caused by the coding method. Combining coding compensation with initial membrane potential, a neuron with initial membrane potential is obtained.

3.1. Neuron Model

This paper uses ReLU as the activation function in the source neural network. During the conversion, the threshold balancing technique is used to solve the problem of inconsistent input and output ranges of different neurons [27,28]. The meanings of symbols that are used in the following paper are provided in

Table 1. The definition of the symbols in this paper.

Symbol	Definition
l	Layer index
$a^l$	Input in ANN
$W^l$	Weight in ANN and SNN
$b^l$	Bias in ANN and SNN
$max(a,0)$	Activation function in ANN
t	Time step
T	Simulation time step
$x^l\left(t\right)$	The spike sequence, consisting of 0 and 1
$v_{temp}^l\left(t\right)$	Temporary membrane potential at time t
$v^l\left(t\right)$	Membrane potential at time t
$V_{th}^l$	Threshold
${\theta }^l\left(t\right)$	The spiking state of neurons at time t
$r^l$	Spike fire rate

.

The output of ReLU at l-th layer can be described by:

$$a^{l+1}=max\left(W^l{a}^l+b^l,0\right)$$

(1)

where $max(x,0)$ refers to the ReLU activation function. The symbols $a^l$ and $a^{l+1}$ denote the input of the l-th layer and (l + 1)-th layer, respectively.

SNN employs Integrate-And-Fire (IF) neurons, which integrate stimuli from presynaptic neurons and fire spikes at postsynaptic neurons [39]. The IF model has a better fit for the output value of the activation function of the ANN than the Leaky Integrate-And-Fire (LIF) model [27]. This neuron can be described as

$$V_{temp}\left(t\right)=V(t-1)+X\left(t\right)$$

(2)

$$V\left(t\right)=\left\{\begin{array}{c}{V}_{temp}\left(t\right)-V_{th},\mathrm{if}V\left(t\right)>V_{th}\hfill \\ V_{temp}\left(t\right),\mathrm{otherwise}\hfill \end{array}\right.$$

(3)

where $V(t-1)$ is the temporary membrane potential at the previous moment, $X\left(t\right)$ is the input at the current moment, and $V_{temp}\left(t\right)$ is the temporary membrane potential at time t. The neuron fires and produces a spike when $V_{temp}\left(t\right)$ exceeds its threshold $V_{th}$. Then, $V\left(t\right)$ is obtained by performing “soft reset” operations on $V_{temp}\left(t\right)$. Otherwise, no spike is fired, and $V\left(t\right)$ is equal to $V_{temp}\left(t\right)$. Here, “soft reset” can reduce the loss in the conversion process [28,34].

Figure 2 depicts the differences between ANN and SNN concerning neuron models. Unlike the direct input of actual values in ANN, SNN takes spikes as input. After integration, SNN makes the membrane potential change and emits spikes. Then, the emitted spike sequence is used as the output of this layer. The values in the spike sequence are only 0 and 1, where 0 indicates no spikes are fired, and 1 indicates spikes are fired.

3.2. Error Analysis

From Equations (1) and (2), the change of membrane potential in SNN can be obtained as follows:

$$v_{temp}^l\left(t\right)=v^l(t-1)+W^l{x}^l\left(t\right)+b^l$$

(4)

where $v_{temp}^l\left(t\right)$ is the temporary membrane potential at time t of the l-th layer, which is the sum of the membrane potential at time $t-1$ and the membrane potential received at time t. Based on the spike fire mechanism in Equation (3), the final membrane potential change is described as:

$$v^l\left(t\right)=v^l(t-1)+W^l{x}^l\left(t\right)+b^l-{\theta }^l\left(t\right)V_{th}^l$$

(5)

The temporary membrane potential needs to be judged to determine whether it emits a spike, and then the membrane potential is reset by a “soft reset”. The ${\theta }^l\left(t\right)$ is used to determine whether to emit spikes and is described as:

$${\theta }^l\left(t\right)=\left\{\begin{array}{c}1,\mathrm{if}{v}_{\mathrm{temp}}^l\left(t\right)>V_{th}^l\hfill \\ 0,\mathrm{otherwise}\hfill \end{array}\right.$$

(6)

where $\theta \left(t\right)$ is a piece-wise function, such that if the value of $v_{temp}^l\left(t\right)$ is greater than the threshold $V_{th}^l$, it is regarded as a spike and has a value of 1, and otherwise the value is 0. ${\theta }^l\left(t\right)$ indicates whether a spike is sent at time t. Therefore, ${\theta }^l$ can be regarded as the output spike sequence of the l-th layer and the input of the $(l+1)-th$ layer.

Based on the explicit iterative Equation (5), the values are given by integrating the input sequence and the spike fire sequence during $t\in (0,T)$. The equation of membrane potential change at time T can be described as follows:

$$v^l\left(T\right)=v^l\left(0\right)+W^l\sum _{t=1}^T{x}^l\left(t\right)+T{b}^l-\sum _{t=1}^T{\theta }^l\left(t\right)V_{th}^l$$

(7)

where $v^l\left(0\right)$ is the membrane potential of the l-th layer at time $t=0$, and $v^l\left(T\right)$ is the membrane potential at time $t=T$. Of course, ${\theta }^l$ is the output of the l-th layer and also the input of the $(l+1)$-th layer. Here, the spike fire rate $r^l$ be led into:

$$r^l=\frac{{\sum }_{t=1}^T{\theta }^l\left(t\right)}{T}$$

(8)

Since we use the threshold balance mechanism in the conversion process, here is the equation: $x^{l+1}\left(t\right)={\theta }^l\left(t\right)V_{th}^l$. Therefore, the new spike fire rate $r^{l-1}$ can be described as:

$$r^{l-1}=\frac{{\sum }_{t=1}^T{x}^l\left(t\right)}{T{V}_{th}^{l-1}}$$

(9)

Substituting (8) and Equation (9) into Equation (7), we obtain:

$$v^l\left(T\right)=v^l\left(0\right)+W^{l}T{r}^{l-1}{V}_{th}^{l-1}+T{b}^l-T{r}^l{V}_{th}^l$$

(10)

Arranging Equation (10) and extracting $T{V}_{th}^l$, we can obtain the following equation:

$$r^l=\frac{W^l{r}^{l-1}{V}_{th}^{l-1}+b^l}{V_{th}^l}+\frac{v^l\left(0\right)-v^l\left(T\right)}{T{V}_{th}^l}$$

(11)

The right-hand side of Equation (11) has two subterms. The first sub-item $\frac{W^l{r}^{l-1}{V}_{th}^{l-1}+b^l}{V_{th}^l}$ is the correspondence between the spike firing rate $r^l$ at the l-th layer and the spike firing rate $r^{l-1}$ at the $(l-1)$-th layer. The second sub-term $\frac{v^l\left(0\right)-v^l\left(T\right)}{T{V}_{th}^l}$ is the error term obtained during the derivation process.

3.3. Initialize Membrane Potential

From Equation (11), it can be determined that there will be a error term in the process of conversion: $\frac{v^l\left(0\right)-v^l\left(T\right)}{T{V}_{th}^l}$. The error term $\frac{v^l\left(0\right)-v^l\left(T\right)}{T{V}_{th}^l}$ can be seen as $\frac{v^l\left(0\right)-v^l\left(T\right)}{V_{th}^l}\frac{1}{T}$, where $\frac{v^l\left(0\right)-v^l\left(T\right)}{V_{th}^l}$ is strictly limited to (−1, 0), and $\frac{1}{T}$ is used as a scaling factor of $\frac{v^l\left(0\right)-v^l\left(T\right)}{V_{th}^l}$. The common processing method is to set the initial membrane potential $v^l\left(0\right)$ to 0 and discard the other part of $v^l\left(T\right)$. Traditional conversion scenarios generally use a larger simulation duration T, such as 1024, 2048, or even 4096. In this case, the scaling factor $\frac{1}{T}$ becomes so small that the entire error term shrinks to be negligible. Therefore, setting $v^l\left(0\right)$ to 0 and discarding $v^l\left(T\right)$ brings almost no loss of information. Although long simulation time can bring the accuracy closer to the original network, it also leads to the problem of too-long inference time.

Large time steps could become a huge obstacle to the practical application of SNN. When considering small time steps, as T gradually decreases, the scaling factor gradually increases, so that the error term can no longer be ignored. At this time, the common processing method is no longer applicable. The analysis of Equation (11) helps explain why the accuracy of the SNN obtained by the conversion is low in the case of low latency.

In the process of analyzing the conversion error, Bu et al. [38] regarded $v^l\left(T\right)$ in it as being caused by the uneven distribution of the output pulse, and pointed out that assuming the potential $v^l\left(t\right)$ falls into [0, $V_{th}^l$] will be able to estimate the activation function of SNNs, ignoring the effect of unevenness error. Unlike Bu, who abandons $v^l\left(T\right)$, in this paper, we regard $v^l\left(T\right)$ as a non-negligible error, which is the residual membrane potential because the membrane potential does not reach the neuron threshold at t = T. Wang et al. [15] also considers the residual membrane potential, which cannot be ignored, but the overall training steps use secondary training, more training steps may have a certain impact on the training time and training volume. Here, we propose a strategy that only requires forward propagation on the transformed SNNs to obtain the initial membrane potential, thereby eliminating this part of the error. Unlike the first two, which require various changes and adaptations on ANN, this strategy only relies on SNN. In the face of other types of ANNs, this strategy does not need to consider the adaptability of ANN to the strategy and only focuses on the target SNN.

This paper, the values $v^l\left(0\right)$ and $v^l\left(T\right)$ are processed together. If the value can be given of $v^l\left(T\right)$ and used as the initial membrane potential, these two terms can be canceled simultaneously. From Equation (7), $v^l\left(T\right)$ is the membrane potential remaining at the last moment. If all membrane potentials from $t=0$ to $t=T$ are accumulated and regarded as the dividend, then the threshold of the trigger spike can be regarded as the divisor. Therefore, $v^l\left(T\right)$ is equivalent to the remainder obtained by dividing the two. Therefore, a method is proposed to obtain the residual membrane potential $v^l\left(0\right)$ based on the trained ANN.

In Algorithm 1, the fire distribution of the spike sequence is more extreme. That is, neurons emit spikes at every moment. In this case, the sum of the membrane potentials of the $t=1$ to $t=T$ time steps in each neural network layer are approximately the accumulation of the weights for the entire T period. Then, the sum of the obtained membrane potentials is taken as the dividend; the threshold after equilibrium is taken as the divisor. Finally, the remainder is given to obtain the approximate residual membrane potential. This algorithm reduces the conversion loss in the ANN2SNN process and improves the network’s accuracy.

Algorithm 1 Initialized membrane potential based on a trained ANN.

Input:  ANN that has been trained; Simulation duration T; threshold $V_{th}^l$ that has been threshold-balanced; Output:  Initialized membrane potential $init\text{\_}mem$; 1: The threshold is modulated based on the weights in the trained ANN; 2: $init\text{\_}mem=\left[\right]$ 3: for$linlayers$do 4:     $sum\text{\_}mem=ANN.weight\left[l\right]\ast T+ANN.bias\left[l\right]$ 5:     $mem=sum\text{\_}mem\%V_{th}^l$ 6:     $init\text{\_}mem\left[l\right]=mem.mean\left(\right)$ 7: end for 8: return$init\text{\_}mem$;

Data often play an essential role in the training process of neural networks. Therefore, the training set is introduced in the algorithm. The initial membrane potential of neurons is obtained through the training set, and the extreme firing of neurons is discarded. Dataset-based initialized membrane potentials are not only not limited by spiked discharge conditions and more versatile, but also perform better in practical applications, allowing the network to achieve higher accuracy.

Algorithm 2 firstly transfers the weight parameters of the trained ANN to the SNN and adopts the threshold balancing operation. Then, the training set is used for training on the SNN. The forwarding propagation process and the record of remaining membrane potential at time $t=T$ in each neural network layer must be paid attention to in the training process. Then, based on the record, the initial membrane potential is approximately given. While this is not a strong guarantee that the network can maintain performance on the test set, the training set should represent the test set. Results show this approach to be highly effective.

Algorithm 2 Initialized membrane potential based on training set.

Input:  ANN that has been trained; SNN ready for transformation; Simulation duration T; training datasets; Output:  Initialized membrane potential $init\text{\_}mem$; 1: The weights of the trained ANN are converted into the SNN; then the SNN is used to train on the training set, and the value of the remaining membrane potential is recorded. 2: $init\text{\_}mem=\left[\right]$ 3: $mem\text{\_}sum=\left[\right]$ 4: for$linlayers$do 5:     $SNN.weight\left[l\right]=ANN.weight\left[l\right]$ 6:     $SNN.bias\left[l\right]=ANN.bias\left[l\right]$ 7: end for 8: for$datasetindatasets$do 9:     for $tinT$ do 10:         for $linLayer$ do 11:            if $t=T$ then 12:                $mem=l.mem.mean\left(\right)$ 13:                $mem\text{\_}sum\left[l\right]=mem\text{\_}sum\left[l\right]+mem$ 14:            end if 15:         end for 16:     end for 17: end for 18: for$linLayer$do 19:     $init\text{\_}mem\left[l\right]=mem\text{\_}sum\left[l\right]/len\left(datasets\right)$ 20: end for 21: return$init\text{\_}mem$;

3.4. Coding Compensation

A very important point in brain research at the moment is how the brain encodes incoming information. SNN can simulate the use of spikes to transmit information between neurons in the brain. However, encoding the original information into a spike sequence is still an ongoing research problem. This is one of the reasons why an SNN is challenging to train.

In the choice of encoding, most researchers will choose rate encoding, which is also the most commonly used encoding method. The rate at which spikes are fired over time can be used to replace the actual values passed into the neural network. In the case of a relatively large time step, the network can achieve almost lossless conversion. If the time step is shortened, the information loss during the conversion process will be greatly increased. In Figure 3, rate-encoded pictures at different time steps are shown. Among them, 2.a is the original image from the dataset CIFAR10, and the image size is 32 × 32. Furthermore, 2.b, 2.c, and 2.d are pictures encoded with 32-time steps, 128 time steps, and 1024 time steps, respectively. Although the difference between the image obtained under the 1024 time steps encoding and the source image is minimal, there is still a visible difference at low latency.

In Section 3.3, an algorithm is given about the initial membrane potential for SNN. So here, the second half of Equation (11) can be approximately zero. Then, simplify Equation (11), which can be described as:

$$r^l=\frac{W^l{r}^{l-1}{V}_{th}^{l-1}+b^l}{V_{th}^l}$$

(12)

Moving the threshold of the l-th layer to the left side of the equation, we can obtain

$$r^l{V}_{th}^l=W^l{r}^{l-1}{V}_{th}^{l-1}+b^l$$

(13)

where $x^l=r^l{V}_{th}^l$ is introduced, representing the average membrane potential of the postsynaptic and also the decoding of the spike train. Substituting this into Equation (13), we can obtain:

$$x^l=W^l{x}^{l-1}+b^l$$

(14)

In addition, with Equation (1), we can obtain the error of the output of the two neural networks:

$$\begin{array}{cc}\hfill \Delta a^l& =a^l-x^l\hfill \\ \hfill & =W^l{a}^{l-1}+b^l-\left(W^l{x}^{l-1}+b^l\right)\hfill \\ \hfill & =W^l\left(a^{l-1}-x^{l-1}\right)\hfill \end{array}$$

(15)

From Equation (15), it is known that, from the first layer’s encoding, each layer accumulates the information loss generated during the encoding process. Consider the following equation when using rate coding: $a^l\approx \frac{V_{th}^l}{T}{\sum }_{t=1}^{T}x\left(t\right)=x^l$. Some information is discarded during this coding process. Furthermore, unlike traditional networks, SNNs acquire less information about each layer. After multiple layers of accumulation, the information loss caused by encoding will accumulate rapidly.

From the equation $a^l\approx \frac{V_{th}^l}{T}{\sum }_{t=1}^{T}x\left(t\right)=x^l$, it is clear that there are potential information loss items in the process of encoding: $\Delta a^l\in (0,\frac{V_{th}^l}{T})$. Of course, the distribution of this part of the coding error is unknown; if we assume that its distribution is uniform, then the expectation of $\Delta a^l$ is $\frac{V_{th}^l}{2T}$. Bu et al. [14] also assumes that the distribution of the input $z^l$ is uniform, and the expectation of the conversion error is minimum, and zero when $\phi$ is 0.5. The final result is also $\frac{V_{th}^l}{2T}$, which is called a shift operation. The traditional way of processing can be regarded as a lack of firing of a spike. Then, the activation of a spike is increased by adjusting the bias of each layer of the network. Here, combined with the content of Section 3.3, a method for handling this situation is proposed.

The Algorithm 3 uses SNN as the source network, and the SNN does not perform the initializing membrane potential operation. First, the compensation value is used as the initial membrane potential in each layer of the network, and then the network uses the algorithm of Section 3.3 to initialize the membrane potential. Finally, the obtained initialized membrane potential is combined with the compensation value to obtain the final membrane potential. Compared with directly combining the compensation value with the initialized membrane potential, this algorithm can improve the effect of the algorithm in Section 3.3 while performing the compensation operation, thereby improving the overall accuracy of the network.

Algorithm 3 Coding compensation.

Input:  SNN before initializing membrane potential; Simulation duration T; Initializing membrane potential function $init\text{\_}mem\left(\right)$; Output:  Initialized membrane potential $final\text{\_}mem$; 1: A compensation value is added to each layer of the SNN, and then the initial membrane potential is obtained by training on the SNN after coding compensation, and finally, the initial membrane potential is added to the coding compensation to obtain the final membrane potential: 2: $init\text{\_}mem=\left[\right]$ 3: $final\text{\_}mem=\left[\right]$ 4: for$linlayers$do 5:     $SNN.init\text{\_}mem\left[l\right]=V_{th}^l/2T$ 6: end for 7: $init\text{\_}mem=init\text{\_}mem\left(SNN\right)$ 8: for$linlayers$do 9:     $final\text{\_}mem=init\text{\_}mem\left[l\right]+V_{th}^l/2T$ 10: end for 11: return$final\text{\_}mem$;

To sum up, the overall architecture of the network is shown in Figure 4. First, the initially trained network is an ANN, and then the weights after training are transferred to the SNN, and the threshold-balancing operation is used in this process. Subsequently, the suitable initial membrane potential is obtained based on the remaining membrane potential training at time T. Considering the error of the data brought about by the encoding process, the encoding compensation is combined with the initial membrane potential to complete the training. A neuron with a suitable initial membrane potential is finally obtained.

4. Experiments

In this section, we use different datasets to conduct multiple experiments and analyze the improved network type in detail from various point of views. We also compare the experimental results with other networks to demonstrate the characteristics of our model and verify the accuracy of the model.

4.1. Experiment Preparation

Before conducting experiments, we must preprocess the dataset and initialize the hyperparameters using related techniques to ensure the best experimental results.

4.1.1. Experiment Environment

Two public image classification datasets, CIFAR10 and CIFAR100, were used in our experiments. For the source ANN for training, VGG16 and ResNet20, which are more common in ANNs, were chosen to be used, with a slight modification [40,41]. Batch normalization and dropout in neural networks were discarded, AvgPool replaced MaxPool, and the ReLU function was used as an activation function. The experiment was run using Pytorch as the deep learning framework, which runs on RTX2060 (12 GB). During the whole conversion process, only the threshold balancing technique was used [27,28].

4.1.2. Pre-Processing

The dataset used for training needed to be preprocessed. CIFAR10 and CIFAR100 were flipped horizontally and randomly cropped to prevent overfitting. The image size was set to 32 × 32 to fit the input size of the neural network. The final image was normalized to ensure a mean of 0 and a variance of 1.

4.1.3. Hyper-Parameters

The ANN training method generally does not affect the error term of the network conversion, so in our experiments, we adopted a common method to train ANN. During the training of the ANN, the initial learning rate was set to 0.1, the momentum was set to 0.9, the weight decay was set to 5 $\times {10}^{-4}$, and the optimizer we chose was SGD. The batch size was set to 128, and the epoch was set to 300. At epochs 70, 130, 190, and 240, the learning rate decayed by 5. In the ANN-to-SNN conversion threshold balance, the p-quantile was set to [0.9–0.99].

4.2. Validity Verification

We will verify the effectiveness of our improvements from multiple perspectives and analyze the specific reasons for the improvement.

4.2.1. Accuracy

We experimentally verified the two improvement strategies mentioned in Section 3.3 and Section 3.4. We used VGG16 as the source neural network and CIFAR10 as the dataset and divided the experiments into five groups: the source neural network (ANN), the non-improved SNN (SNN), the SNN with initialized membrane potential (I-SNN), the SNN with coding compensation(C-SNN), and the SNN with initialized membrane potential and coding compensation(IC-SNN). The results are shown in Figure 5.

Compared with the conventional ANN2SNN, the proposed two improved strategies improve the accuracy. The SNN can achieve higher accuracy at lower time steps with both improvements. It can be seen in the figure that the conventional ANN2SNN has inferior performance at low latency, with an accuracy rate of only 20% to 30%. Adopting the improved strategy can easily achieve an accuracy rate of more than 70%. As the time step increases gradually, although the magnitude of the improvement will become progressively smaller, the overall progress will still be ongoing.

4.2.2. Fire Time

The spiking neuron needs to accept the previous layer’s output for a long time and accumulates until it reaches the threshold, and then the spike is fired. We analyzed the timing of spiking neurons firing and explored the impact of our improvements on the firing timing. At low time steps, neurons often do not receive enough output from the previous layer to fire spikes and remain inactive, which seriously hinders the firing of spiking neurons. A large amount of information is lost in the transmission process, resulting in a significant drop in the accuracy of the neural network. The two improved strategies in this paper essentially give the neuron an initial membrane potential, which enables the neuron to activate faster and fire spikes to transmit information to the next layer. Then, to a certain extent, this avoids situations where neurons are not activated or under-activated for a sustained period.

Figure 6 shows the time of the first burst of each layer of the IC-SNN after introducing the improved strategy compared to the traditional transformation network SNN. The abscissa is the corresponding network layer number, and the ordinate is the average first firing time of neurons. It can find that after the introduction of the initial membrane potential, the average firing time of neurons in each layer of the IC-SNN is earlier than that in the SNN. Especially in deeper networks, IC-SNN can significantly speed up the time of firing spikes, which is conducive to transmitting more information and improving the accuracy of the network.

4.2.3. Output

Figure 7 shows the output of the intermediate process of the neural network, which explains to a certain extent why the improved strategy leads to an increase in accuracy. The images on the left are the output of the first layer of the neural network. From top to bottom, the network output is under eight time steps, 16-time steps, and 32-time steps. The images on the right are the output of the third layer network.

From Figure 7, at 32 time steps, the IC-SNN with the improved strategy almost coincides with the ANN, while there are still some large differences between the conventional SNN and the ANN. This situation is more pronounced at 16 time steps and especially at eight time steps. As the network deepens, conversion errors begin to accumulate. At this point, IC-SNN and ANN begin to separate gradually, but are still closer to the original value than SNN. Therefore, adopting the initialization membrane potential and coding compensation (IC) strategy can effectively reduce the conversion error and make the output of each layer in the SNN closer to the source neural network, thereby improving the accuracy of the SNN.

4.3. Contrast Experiment

We compare the proposed improvements with other networks, and the specific comparison results are shown in

Table 2. Results of the test on the CIFAR10 dataset using VGG16.

$\mathit{Method}$	${\mathit{ANN}}_{\mathit{ACC}}$	${\mathit{T}}_8\text{\_}\mathcal{L}$	${\mathit{T}}_{16}\text{\_}\mathcal{L}$	${\mathit{T}}_{32}\text{\_}\mathcal{L}$	${\mathit{T}}_{64}\text{\_}\mathcal{L}$	${\mathit{T}}_{128}\text{\_}\mathcal{L}$	${\mathit{T}}_{256+}\text{\_}\mathcal{L}$
$RobustNorm$ [28]	$92.82$	−	$82.71$	$49.79$	$11.30$	$2.02$	$0.07$
$SpikeNorm$ [29]	$91.70$	−	−	−	−	−	$0.15$
$HybridTrain$ [37]	$92.81$	−	−	−	−	$1.68$	$0.33$
$RMP$ [34]	$93.63$	−	−	$33.33$	$3.28$	$1.22$	$\mathbf{0.01}$
$TSC$ [32]	$93.63$	−	−	−	$0.84$	$0.36$	$0.01$
$Opt.$ [35]	$\mathbf{95.72}$	−	−	$19.48$	$5.08$	$1.61$	$0.01$
$RNL$ [36]	$92.86$	−	$34.96$	$7.46$	$1.71$	$0.35$	$0.01$
$I-SNN\left(Ourwork\right)$	$92.09$	$22.28$	$8.14$	$1.72$	$0.91$	$0.75$	$0.08$
$C-SNN\left(Ourwork\right)$	$92.09$	$28.52$	$12.06$	$5.25$	$2.84$	$1.37$	$0.13$
$\mathit{IC}-\mathit{SNN}\left(\mathit{Our}\mathit{work}\right)$	$92.09$	$\mathbf{9.95}$	$\mathbf{2.97}$	$\mathbf{1.03}$	$\mathbf{0.61}$	$\mathbf{0.29}$	$0.05$

,

Table 3. Results of the test on the CIFAR100 dataset using VGG16.

$\mathit{Method}$	${\mathit{ANN}}_{\mathit{ACC}}$	${\mathit{T}}_8\text{\_}\mathcal{L}$	${\mathit{T}}_{16}\text{\_}\mathcal{L}$	${\mathit{T}}_{32}\text{\_}\mathcal{L}$	${\mathit{T}}_{64}\text{\_}\mathcal{L}$	${\mathit{T}}_{128}\text{\_}\mathcal{L}$	${\mathit{T}}_{256+}\text{\_}\mathcal{L}$
$SpikeNorm$ [29]	$71.22$	−	−	−	−	−	$0.45$
$RMP$ [34]	$71.22$	−	−	−	−	$7.46$	$0.29$
$TSC$ [32]	$71.22$	−	−	−	−	$1.36$	$0.25$
$Opt.$ [35]	$\mathbf{77.89}$	−	−	$70.25$	$50.05$	$16.85$	$\mathbf{0.18}$
$I-SNN\left(Ourwork\right)$	$70.59$	$33.79$	$10.22$	$4.03$	$1.37$	$0.83$	$0.31$
$C-SNN\left(Ourwork\right)$	$70.59$	$47.38$	$20.42$	$8.31$	$2.85$	$1.89$	$0.41$
$\mathit{IC}-\mathit{SNN}\left(\mathit{Our}\mathit{work}\right)$	$70.59$	$\mathbf{15.72}$	$\mathbf{7.11}$	$\mathbf{3.74}$	$\mathbf{1.24}$	$\mathbf{0.77}$	$0.20$

,

Table 4. Results of the test on the CIFAR10 dataset using ResNet20.

$\mathit{Method}$	${\mathit{ANN}}_{\mathit{ACC}}$	${\mathit{T}}_8\text{\_}\mathcal{L}$	${\mathit{T}}_{16}\text{\_}\mathcal{L}$	${\mathit{T}}_{32}\text{\_}\mathcal{L}$	${\mathit{T}}_{64}\text{\_}\mathcal{L}$	${\mathit{T}}_{128}\text{\_}\mathcal{L}$	${\mathit{T}}_{256+}\text{\_}\mathcal{L}$
$SpikeNorm$ [29]	$89.10$	−	−	−	−	−	$1.64$
$HybridTrain$ [37]	$\mathbf{93.15}$	−	−	−	−	−	$0.21$
$RMP$ [34]	$91.47$	−	−	−	−	$3.87$	$0.11$
$TSC$ [32]	$91.47$	−	−	−	$22.09$	$2.90$	$\mathbf{0.05}$
$I-SNN\left(Ourwork\right)$	$91.46$	$43.21$	$21.75$	$4.72$	$3.87$	$1.06$	$0.21$
$C-SNN\left(Ourwork\right)$	$91.46$	$57.18$	$30.39$	$8.25$	$5.34$	$2.39$	$0.27$
$\mathit{IC}-\mathit{SNN}\left(\mathit{Our}\mathit{work}\right)$	$91.46$	$\mathbf{37.31}$	$\mathbf{10.23}$	$\mathbf{3.64}$	$\mathbf{2.14}$	$\mathbf{0.81}$	$0.09$

and

Table 5. Results of the test on the CIFAR100 dataset using ResNet20.

$\mathit{Method}$	${\mathit{ANN}}_{\mathit{ACC}}$	${\mathit{T}}_8\text{\_}\mathcal{L}$	${\mathit{T}}_{16}\text{\_}\mathcal{L}$	${\mathit{T}}_{32}\text{\_}\mathcal{L}$	${\mathit{T}}_{64}\text{\_}\mathcal{L}$	${\mathit{T}}_{128}\text{\_}\mathcal{L}$	${\mathit{T}}_{256+}\text{\_}\mathcal{L}$
$SpikeNorm$ [29]	$69.72$	−	−	−	−	−	$5.63$
$RMP$ [34]	$68.72$	−	−	$41.08$	$21.89$	$11.03$	$0.90$
$TSC$ [32]	$\mathbf{71.22}$	−	−	−	−	$10.30$	$0.54$
$I-SNN\left(Ourwork\right)$	$66.05$	$39.68$	$22.41$	$15.77$	$4.62$	$2.03$	$0.59$
$C-SNN\left(Ourwork\right)$	$66.05$	$46.71$	$27.19$	$21.03$	$7.93$	$3.31$	$0.63$
$\mathit{IC}-\mathit{SNN}\left(\mathit{Our}\mathit{work}\right)$	$66.05$	$\mathbf{35.59}$	$\mathbf{17.70}$	$\mathbf{9.28}$	$\mathbf{3.13}$	$\mathbf{1.61}$	$\mathbf{0.45}$

. This paper only compares the experimental results using the same network model and dataset in the original literature. The first column lists previous research and improvements; $AN{N}_{ACC}$ in the second column represents the accuracy of the source neural network. Each subsequent column represents the accuracy loss of ANN and the converted SNN in different time steps. We define accuracy loss as $T_t\text{\_}\mathcal{L}=AN{N}_{ACC}-SN{N}_{ACC}$, where t represents the time step. For example, the column of $T_8\text{\_}\mathcal{L}$ represents the value of the accuracy loss when converting ANN to SNN at eight time steps. The smaller value indicates a more effective algorithm. In the comparison item, the table sets the SNN that only uses the initialized membrane potential (I-SNN), the SNN that only uses the coding compensation (C-SNN), and the SNN that uses both the initialized membrane potential and the coding compensation (IC-SNN).

As shown in Table 2 and Table 3, we used the VGG16 model and the CIFAR10 and CIFAR100 datasets to compare with other researchers’ experiments. In the CIFAR10 dataset, IC-SNN achieved more than 80% accuracy at eight time steps, whereas other networks require about 32 time steps to achieve similar accuracy. At 16 time steps, the network achieves close to 90% accuracy, but other networks require at least 64 time steps. The results of VGG16 running on the CIFAR100 dataset further demonstrate the effectiveness of our improvements. The accuracy rate has exceeded 60% at 16-time steps, reaching the mark that other networks require 64 time steps. The conversion error loss is reduced to 1.24% at 64-time steps and to within 1% at 128-time steps.

Thanks to the proposed initializing membrane potential strategy and coding compensation strategy, our network has fewer conversion errors than other networks at low time steps, especially at eight time steps and 16 time steps. From the error analysis in Section 3.2, it can be found that the error equation we want to eliminate is negatively related to the time step. Therefore, the lower the time step, the more noticeable our improvements. In the process of gradually increasing the time step, the performance improvement of our network starts to decrease slowly. However, it can still achieve relatively high accuracy at the time step of 128.

We also used the ResNet20 model to conduct experiments on CIFAR10 and CIFAR20. The structure of ResNet is more complicated than that of VGG. Here, we adopt the conventional processing for the residual block, regarding the residual block as a whole layer, and provide a corresponding balance threshold for one residual block. Similarly, when initializing membrane potential and coding compensation, we adopt a similar method and only introduce the related improvement strategy for the first layer in the residual block.

Table 4 and Table 5 show the comparative results of our networks. Even on the more complex ResNet, our improvements still lead to huge improvements. IC-SNN achieves 80% accuracy on CIFAR10 at 16 time steps, and the error of the network transformation shrank to 1% at 128 time steps. The performance on CIFAR100 is also excellent. At 32 time steps, the accuracy error with the source network was reduced to less than 10%, and the error was only 1.77% at 128 time steps.

During the experiment, we tried 16 time steps and even eight time steps, which hardly anyone other than us has used. This is because, in such a short time step, the firing of spikes is seriously affected, which will cause a large amount of information loss, resulting in a significant drop in the accuracy of the neural network. After the improved strategy was introduced, our network still showed excellent performance at such a low time step. It also confirms that our improvement can effectively speed up the firing of spikes, thereby reducing inference time.

5. Conclusions

This paper analyzes errors in the conversion process to help explain the loss of information at low latency. For the case of low delay, a method to initialize the membrane potential was proposed to eliminate the error. Weight-based initialization takes into account more intense spike sequences; data-based initialization is more general and better improves the accuracy of the network. In further analysis after removing errors, a rate-coding-based compensation method was proposed, which, combined with the initialized membrane potential, enabled the network to perform better. This paper conducts experiments on the IC-SNN on different network models and datasets. The results show higher accuracy and minor transformation error at low latency.

In future work, we will focus on the source neural network to improve its performance using batch normalization and dropout techniques. In addition, we will also try to combine biological mechanisms, which further enhance the accuracy at low time steps.