Optimizing Artificial Neural Networks to Minimize Arithmetic Errors in Stochastic Computing Implementations

Abstract

Deploying modern neural networks on resource-constrained edge devices necessitates a series of optimizations to ready them for production. These optimizations typically involve pruning, quantization, and fixed-point conversion to compress the model size and enhance energy efficiency. While these optimizations are generally adequate for most edge devices, there exists potential for further improving the energy efficiency by leveraging special-purpose hardware and unconventional computing paradigms. In this study, we explore stochastic computing neural networks and their impact on quantization and overall performance concerning weight distributions. When arithmetic operations such as addition and multiplication are executed by stochastic computing hardware, the arithmetic error may significantly increase, leading to a diminished overall accuracy. To bridge the accuracy gap between a fixed-point model and its stochastic computing implementation, we propose a novel approximate arithmetic-aware training method. We validate the efficacy of our approach by implementing the LeNet-5 convolutional neural network on an FPGA. Our experimental results reveal a negligible accuracy degradation of merely 0.01% compared with the floating-point counterpart, while achieving a substantial 27× speedup and 33× enhancement in energy efficiency compared with other FPGA implementations. Additionally, the proposed method enhances the likelihood of selecting optimal LFSR seeds for stochastic computing systems.

1. Introduction

The increasing use of intelligent edge devices for inference tasks utilizing deep learning (DL) models is largely due to their advantages in privacy, latency, bandwidth, energy efficiency, and cost. This contrasts with the conventional method of transmitting sensor data to the cloud for analysis [1]. In this scenario, real-time environmental monitoring minimizes the need to send data to the cloud, thus reducing the burden on data centers.

In response to the growing technological trend, numerous companies are developing specialized CPUs, DSPs, GPUs, and other processors to support general-purpose machine learning (ML) and neural network (NN) capabilities at the edge [2,3]. These processors are compatible with popular ML frameworks [4,5]. Meanwhile, some silicon companies are exploring unconventional processing methods, like approximate computations, to surpass the inherent limitations of silicon technology and achieve greater computational efficiency per unit of power [6,7]. This study examines the digital implementation of DL models using stochastic computing (SC), a unary computing method based on random and non-weighted bitstreams [8].

The accuracy of NNs in SC models is significantly affected by the stochastic nature of bitstreams and the quantization of parameters and activations. To mitigate the accuracy decline in SC models, we analyze the error introduced by SC multiplications and integrate this analysis with the optimal selection of Linear Feedback Shift Registers (LFSRs) [9]. Additionally, we propose a novel training method that takes into account the approximate arithmetic operations in SC NNs.

This approach produces an SC inference model with minimal accuracy loss compared with its fixed-point counterpart. We demonstrate this method using the LeNet-5 model for classifying the MNIST dataset [10,11], providing results that include both accuracy and FPGA implementation metrics [12].

2. Stochastic Computing

Stochastic computing is a numerical representation methodology that deviates from conventional representation in the digital domain. Instead of using binary digits (bits) to represent numbers, SC utilizes a bitstream where the value is conveyed by the probability of encountering a logic 1. This approach translates real numbers into probability distributions, where the number is embedded within the stochastic fluctuations of the bitstream. For instance, the value 0.5 (50% probability of observing a 1) can be represented in SC using various bitstrings, such as 01, 1100, 1001100110, or 01010101010101010101. These examples demonstrate that the ordering of 1’s is inconsequential for representation; rather, their proportion within the bitstream holds significance.

This unconventional numerical representation presents both advantages and limitations. One drawback is the computational overhead associated with operations, as well as the inherent loss of precision due to the stochastic nature of the bitstream. However, SC also offers compelling benefits, including reduced area and energy consumption in implementing computationally intensive tasks like multiplication and non-linear functions.

Two primary codifications exist within the SC framework: unipolarand bipolar. Unipolar encoding represents numbers within the interval [0, 1], while bipolar encoding spans the range to [−1, 1]. The conversion between these codifications can be accomplished using the following equation: $p^{*}=2p-1$, where p represents the unipolar representation of the number and $p^{*}$ signifies the bipolar counterpart. As shown, employing the same bitstream length to represent a single number in both codifications, bipolar exhibits half the precision of unipolar coding. For instance, consider the scenario where a probability value isrepresented using a bitstream with a length of 4. The attainable values for both codifications are presented in

Table 1. Comparison of unipolar and bipolar coding.

Pulses	Unipolar	Bipolar
0/4	0	−1
1/4	0.25	−0.5
2/4	0.50	0
3/4	0.75	0.5
4/4	1	1

. As observed, the unipolar representation exhibits twice the precision of the bipolar encoding. This disparity in precision can lead to elevated errors when employing bipolar coding, as elaborated in the following section.

3. Multiplication Error

Recent interest in the scientific and industrial realms has been directed towards SC circuits due to their efficient execution of computationally intensive operations, notably multiplication, while requiring minimal resources. In fact, SC multiplication requires a single logic gate, either an AND gate for unipolar encoding or an XNOR gate for bipolar encoding, which facilitates a substantial reduction in both area and power consumption. However, this comes at the expense of increased execution time and diminished arithmetic accuracy.

In order to analytically derive an expression for the mean squared error (MSE), consider the i-th bit of two different bitstreams with activation probabilities of x and y that are i.i.d. Bernoulli random variables denoted by $X_i$ and $Y_i$, respectively. These bitstreams represent bipolar variables $x^{*}$ and $y^{*}$, so that the exact product is $z^{*}=x^{*}{y}^{*}$, where $z^{*}$ is a bipolar variable represented by a bitstream with activation probability z. This bitstream with an activation probability z is obtained via a XNOR logic gate, i.e., the i-th bitstream bit is given by (1).

$$Z_i=X_i{Y}_i+(1-X_i)(1-Y_i)\sim \mathrm{Bernoulli}\left({\displaystyle \frac{x^{*}{y}^{*}+1}{2}}\right)$$

(1)

The activation probability of the resulting bitstream is approximated by counting the number of activations. In general, accumulating the Bernoulli experiment result for N iterations yields the binomial random variable Z, given by (2), so that the activation probability is approximated as $Z/N$.

$$Z=\sum _{i=1}^N{Z}_i\sim \mathrm{Bin}\left(N,{\displaystyle \frac{x^{*}{y}^{*}+1}{2}}\right)$$

(2)

Therefore, MSE associated to the bipolar variable ${\widehat{z}}^{*}={(Z/N)}^{*}$ w.r.t. the exact bipolar product $z^{*}={(\mathrm{E}\left[Z\right]/N)}^{*}$ is given by (3) and represented in Figure 1.

$$\begin{array}{cc}\hfill \mathrm{MSE}\left[{\widehat{z}}^{*}\right]& =\mathrm{MSE}\left[2{\displaystyle \frac{Z}{N}}-1\right]=\mathrm{E}\left[{\left(\left(2{\displaystyle \frac{Z}{N}}-1\right)-\left(2{\displaystyle \frac{\mathrm{E}\left[Z\right]}{N}}-1\right)\right)}^2\right]\hfill \\ \hfill & ={\displaystyle \frac{4\mathrm{Var}\left[Z\right]}{N^2}}={\displaystyle \frac{4z(1-z)}{N}}={\displaystyle \frac{1-{\left(x^{*}{y}^{*}\right)}^2}{N}}\hfill \end{array}$$

(3)

Moreover, during neural network (NN) inference, many multiplications are carried out. So, rather than considering the MSE associated to a single bipolar multiplication, we focus on the average MSE (AMSE), which depends on how inputs/activations and weights are distributed. In general, the AMSE of the bipolar multiplication error is given by (4), where f and g are the probability density functions of the inputs and weights, respectively.

$$\mathrm{AMSE}\left[{\widehat{z}}^{*}\right]={\displaystyle \frac{1}{N}}{\int }_{-1}^{+1}{\int }_{-1}^{+1}\left(1-x^2{y}^2\right)f\left(x\right)g\left(y\right)dxdy$$

(4)

In addition, for simplicity, it is considered that the inputs are uniformly distributed, i.e., f is given by (5), which yields (6).

$$f\left(x\right)=\left\{\begin{array}{cc}{\displaystyle \frac{1}{2}}\hfill & -1\le x\le +1,\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(5)

$${\mathrm{AMSE}}_u\left[{\widehat{z}}^{*}\right]={\displaystyle \frac{1}{2N}}{\int }_{-1}^{+1}{\int }_{-1}^{+1}\left(1-x^2{y}^2\right)g\left(y\right)dxdy$$

(6)

Taking (6) as the starting point, three different scenarios are covered:

• Normal–uniform: Weights are normally distributed with zero mean $\mu =0$ and standard deviation $\sigma$, i.e., g is given by (7), which results in the AMSE given by (8), where erf is the error function. Notice this is a good approximation for $\sigma ⪅1/3$, and AMSE can be approximated to ${\displaystyle \frac{1}{N}}$ if $\sigma$ is small.

  $$g\left(y\right)={\displaystyle \frac{1}{\sigma \sqrt{2\pi }}}{e}^{-{\displaystyle \frac{1}{2}}{\left({\displaystyle \frac{y-\mu }{\sigma }}\right)}^2}$$

  (7)

  $$\begin{array}{cc}\hfill {\mathrm{AMSE}}_{nu}\left[{\widehat{z}}^{*}\right]& \approx {\displaystyle \frac{1}{4N\sigma \sqrt{2\pi }}}{\int }_{-1}^{+1}{\int }_{-1}^{+1}\left(1-x^2{y}^2\right)e^{-{\displaystyle \frac{1}{2}}{\left({\displaystyle \frac{y-\mu }{\sigma }}\right)}^2}dxdy\hfill \\ \hfill & ={\displaystyle \frac{1}{3N}}\left(\left(3-{\sigma }^2\right)\mathrm{erf}\left({\displaystyle \frac{1}{\sigma \sqrt{2}}}\right)-\sigma \sqrt{{\displaystyle \frac{2}{\pi }}}{e}^{-{\displaystyle \frac{1}{2{\sigma }^2}}}\right)\hfill \end{array}$$

  (8)
• Uniform–uniform: Weights are uniformly distributed between −1 and +1, i.e., $f=g$, which results in the AMSE given by (9).

  $${\mathrm{AMSE}}_{uu}\left[{\widehat{z}}^{*}\right]={\displaystyle \frac{1}{4N}}{\int }_{-1}^{+1}{\int }_{-1}^{+1}\left(1-x^2{y}^2\right)dxdy={\displaystyle \frac{8}{9N}}$$

  (9)
• Custom–uniform: Weights follow a custom probability density function given by (10), which results in the AMSE given by (11). Notice this is a good approximation for $\sigma ⪅1/3$, and the AMSE can be approximated to ${\displaystyle \frac{8}{9N}}-\alpha {\displaystyle \frac{2}{9N}}$ if $\sigma$ is small.

  $$g\left(y\right)\approx \left\{\begin{array}{cc}{\displaystyle \frac{1-\alpha }{2}}+{\displaystyle \frac{\alpha }{\sigma \sqrt{2\pi }}}\left(e^{-{\displaystyle \frac{1}{2}}{\left({\displaystyle \frac{y+1}{\sigma }}\right)}^2}+e^{-{\displaystyle \frac{1}{2}}{\left({\displaystyle \frac{y-1}{\sigma }}\right)}^2}\right)\hfill & -1\le y\le +1,\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

  (10)

  $${\mathrm{AMSE}}_{cu}\left[{\widehat{z}}^{*}\right]\approx (1-\alpha ){\displaystyle \frac{8}{9N}}+\alpha {\displaystyle \frac{1}{3N}}\left(\left(2+{\sigma }^2\right)\mathrm{erf}\left({\displaystyle \frac{\sqrt{2}}{\sigma }}\right)+{\displaystyle \frac{4}{\sqrt{2\pi }}}\sigma \right)$$

  (11)

These AMSE expressions and the corresponding weight distributions are represented in Figure 2 using $\alpha =0.5$ and $\sigma =1/12$ in (11). Using these numbers and $N=255$ (8-bit signal) results in an AMSE reduction of $19.7\%$ for the custom–uniform case w.r.t. the normal–uniform case with $\sigma =1/12$. Moreover, under the small $\sigma$ approximation, the AMSE reduction is up to $22.\overline{2}\%$. This multiplication error is further reduced using random maximum length sequences (m-sequences) to minimize the representation error of the bitstream w.r.t. the complement number representation of the two. So, using m-sequences leads to exact multiplications by $\pm 1$. In addition, in the experiments, random sequences are generated by LFSRs, which are m-sequences, and also allow for fine tuning the final neural network accuracy by trying different LFSR seed pairs, as described in Section 4.

As illustrated in the histogram of Figure 3 for the LeNet-5 case, weights typically follow a normal or similar distribution. Therefore, in order to reduce the arithmetic error associated with stochastic multiplications, it is proposed to modify the probability density function of the weights during the training stage of a neural network. In particular, it is proposed to modify it so that it is similar to (11), represented in Figure 2 (green dotted line), with many more extreme values.

4. LFSR Seed Selection Error

The most cost-effective approach for generating stochastic bitstreams in a real SC implementation involves utilizing a Linear-Feedback-Shift-Register (LFSR) [9,13,14,15]. Although there are alternatives, such as random number generators (RNGs), LFSRs have some crucial advantages: area saving (which is relevant in SC hardware, given the large area overhead due to RNGs) and the fact of generating a uniformly distributed signal $R\left(t\right)$ in the interval of all possible values of X, the digital number which, after being compared with $R\left(t\right)$, will be converted into a stochastic sequence $x\left(t\right)$ with an associated probability x. An LFSR is a circuit based on a shift register and a linear function of its previous state connected to its input. This linear function is implemented by connecting exclusive OR gates to different taps in the state registers. The pseudorandom sequence is repeated periodically, starting from the seed value. As outlined in the existing literature, it is crucial to identify an appropriate LFSR seed pair capable of producing stochastic bitstreams with minimal correlation. This is necessary to mitigate the correlation phenomenon in the multiplication operation [9,16]. Despite the significance of this requirement, devising an effective methodology for identifying optimal LFSR seeds, especially in the case of complex systems like neural networks (NNs), remains a formidable challenge. The current state of the art has yet to offer a reliable solution, and the predominant approach typically involves trial and error. This can be a time-consuming process, given the various scenarios encountered. For instance, consider Figure 4, illustrating diverse accuracies achieved when employing all possible LFSR seeds for the 8-bit convolutional neural network (CNN) presented by C. Frasser et al. [12]. The results reveal that more than 80% of the seeds yielded inaccurate results (Acc < 0.95) due to accumulation errors in the zero region of multiplication (see Figure 1) and the correlation effects of the seeds. This phenomenon poses a significant challenge that necessitates community attention. As systems grow in complexity, the difficulty identifying LFSR seeds results in minimal AD increases.

5. Stochastic Computing Aware Training

To enhance the precision of SC hardware, several strategies are adopted. These considerations are integral to refining the accuracy of SC-NN hardware:

• Weight clamping: Analogous to fixed-point representation, the technique of clamping weights within a specific range $[-n_{lim},+n_{lim}]$ is employed. This method prevents the emergence of extreme weight values in the distribution, which can otherwise escalate linear quantization errors. By restricting the variance of the weight distribution, smaller quantization errors are achieved.
• Weight distribution uniformization: Adjusting the weight distribution to be more uniform can mitigate the relative errors associated with smaller quantized values. Additionally, dispersing weights that are proximate to zero can diminish the incidence of large relative errors during bipolar SC multiplications.
• Weight distribution binarization: To enhance the accuracy of multiplication outcomes in stochastic circuits, increasing the proportion of weights represented by bipolar quantities set to $-1$ and $+1$ is beneficial. In the bipolar encoding framework, these values are correlated with activation rates of 0 and 1, respectively, thereby not contributing to the arithmetic error.

Grounded in the aforementioned strategies, we introduce a methodology as delineated in Algorithm 1, which incorporates an additional quantization phase. Typically, neural network (NN) training employs iterative gradient descent methodologies, as outlined in [17], consisting of three principal phases. The initial phase involves forward propagation to ascertain intermediate and predicted outputs of the network. This is followed by the backward propagation phase, wherein weight discrepancies are calculated based on a predetermined cost function, with errors being propagated in reverse through the network. Subsequently, the weights are updated employing either basic gradient descent or more advanced algorithms such as Adam [18]. The proposed framework introduces a novel, independent fourth step that manipulates weights to enforce a pseudo-uniform distribution of quantized values upon the culmination of the training regime, as depicted in Figure 5.

Algorithm 1 Stochastic computing aware weight distribution modification and quantization steps

Require: Weights $\mathit{Ｗ}$, width factor $n_{\sigma }$, rounding rate $\epsilon$, number of integer values $n_{values}$     1: # calculate the limit for weight values based on the width factor and standard deviation     2: $n_{lim}\leftarrow n_{\sigma }\mathrm{std}\left(\mathit{Ｗ}\right)$     3: # restrict the range of weight values     4: $\mathit{Ｗ}\leftarrow \mathrm{clamp}\left(\mathit{Ｗ},-n_{lim},+n_{lim}\right)$     5: # quantized weights     6: ${\mathit{Ｗ}}_q\leftarrow ⌊{\displaystyle \frac{\mathit{Ｗ}{n}_{values}}{n_{lim}}}⌉{\displaystyle \frac{n_{lim}}{n_{values}}}$     7: # step towards quantization     8: if this is the last quantization step then     9:      # training is done, keep the quantized weights     10:    $\mathit{Ｗ}\leftarrow {\mathit{Ｗ}}_q$     11: else    12:     # put weights closer to its quantized version based on the rounding rate     13:    $\mathit{Ｗ}\leftarrow \mathit{Ｗ}+\epsilon \left({\mathit{Ｗ}}_q-\mathit{Ｗ}\right)$     14: end if     15: return $\mathit{Ｗ}$

After each optimizer step, we calculate new limits for the weight values. To do this, $n_{\sigma }$ is defined as a new hyperparameter corresponding to the number of standard deviations allowed in the previous weight distribution (width factor). Therefore, the actual absolute limit value is given by (12) and the weight values are restricted using a simple clamping operation given by (13) for every weight value in a given weight tensor $\mathit{Ｗ}$, as denoted by line 4 in Algorithm 1.

$$n_{lim}=n_{\sigma }\mathrm{std}\left(\mathit{Ｗ}\right)$$

(12)

$$\mathrm{clamp}\left(\omega ,-n_{lim},n_{lim}\right)\equiv \left\{\begin{array}{cc}{n}_{lim}\hfill & \mathrm{if}\omega >n_{lim}\hfill \\ -n_{lim}\hfill & \mathrm{if}\omega <-n_{lim}\hfill \\ \omega \hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(13)

For illustration purposes, Figure 6 demonstrates the evolution of the weight distribution from a Gaussian distribution through three sequential iterations of the clamping process across various $n_{\sigma }$ settings. Post clamping, the weight values achieve a more uniform distribution, with the exception of the extremities at $-1$ and $+1$ (it is noteworthy that these extreme values are devoid of associated SC multiplication errors, rendering the resultant weight distribution more advantageous than an ideally uniform distribution). This uniformity enhances the accuracy of uniform quantization, particularly in comparison with the normal distributions. Accordingly, we have implemented a straightforward uniform symmetric quantization approach, given by (14).

$${\mathit{Ｗ}}_q=⌊{\displaystyle \frac{\mathit{Ｗ}{n}_{values}}{n_{lim}}}⌉{\displaystyle \frac{n_{lim}}{n_{values}}}$$

(14)

In the above expression, the operation $⌊·⌉$ signifies rounding to the nearest integer. For radix-2 data representation, it is pragmatic to select $n_{values}$ as a power of two minus one; for instance, in the case of 8-bit weights, we set $n_{values}=255$. To facilitate a controlled transition from continuous weight values to their quantized counterparts, we introduce the rounding rate $\epsilon$ as a hyperparameter. This hyperparameter governs the incremental adjustment of the weights $\mathit{Ｗ}$ towards their quantized form ${\mathit{Ｗ}}_q$ with each iteration, as detailed in (15). The hyperparameter is adjusted as a function that increases monotonically with the number of training epochs, starting at $\epsilon =0$ and ending at $\epsilon =1$ towards the conclusion of the training period.

$${\mathit{Ｗ}}_{next}=\mathit{Ｗ}+\epsilon \left({\mathit{Ｗ}}_q-\mathit{Ｗ}\right)$$

(15)

Although this adaptive training strategy has proven to increase the accuracy for SC-NN implementations, it requieres the careful management of two additional hyperparameters: the width factor $n_{\sigma }$ and the rounding rate $\epsilon$.

6. Experimental Results

Experimental outcomes detailed in this section stem from the dynamic adjustments made to the additional hyperparameters $n_{\sigma }$ and $\epsilon$, as visualized in Figure 7. Employing a batch size of 64, we adhered to default Adam optimizer parameters ($\lambda =0.001$, ${\beta }_1=0.9$, ${\beta }_2=0.999$), and utilized a cross-entropy cost function. The width factor, initially set to $n_{\sigma }=6$ with no constraints, underwent exponential reduction until approximately $n_{\sigma }\approx 2$ across 60 epochs. Meanwhile, the rounding rate followed a logistic function shape, commencing near 0 with negligible quantization and progressively ascending to 1 ($\mathit{Ｗ}={\mathit{Ｗ}}_q$) upon training completion.

To validate the efficacy of our training methodology, we executed a hardware (FPGA) implementation [12] of the renowned CNN architecture, LeNet-5 [10], utilizing stochastic computing. The network underwent training utilizing the MNIST handwriting dataset, comprising 60 k training images and 10 k testing images [11]. Our implementation adhered to the CNN architecture originally proposed by J. Lecun et al. [10], which includes two convolutional layers, two max-pooling layers, and three fully connected layers. A comprehensive outline of the hardware design employed in this endeavor is delineated in Figure 8, with further details provided in reference [12]. The implementation was executed on a GIDEL PROC10A board [19], equipped with an Intel 10AX115H3F34I2SG FPGA operating at 150 MHz.

The software model achieved an accuracy of 98.98% after training using the SC-aware methodology. Meanwhile, the hardware accuracy with 8-bit precision stood at 98.97%, and at 97.64% for 6-bit precision. The associated weight distribution following the proposed training method is depicted in Figure 9.

Table 2. Enhancements in FPGA implementation compared to prior work (SCFPGA22 [12]).

Metric	SCFPGA22 [12]	This Work
Year	2022	2024
Architecture parallelism	Parallel	Parallel
Computing paradigm	SC	SC
Activation/Weight bits	8/8	6/6
FPGA platform	Arria10 GX1150	Arria10 GX1150
Frequency (MHz)	150	150
Software Acc (%)	98.60	98.98
Hardware Acc (%)	97.58	97.64
Acc Degradation (%)	1.02	1.34
Throughput (Images/s)	294,118	1,190,476
Performance (Images/s/MHz)	1961	7937
Power (W)	21.0	4.9
Energy efficiency (Images/J)	14,006	243,171
Logic used K (LUT or ALM)	343	318
DSP (blocks)	0	0
Memory (Mbits)	0.00	0.00

illustrates the advancements achieved through the adoption of the training methodology advocated in this research compared with utilizing the same hardware architecture without such training. It is apparent that the implementation of this approach facilitated a transition from 8 bits to 6 bits, resulting in a 4× increase in throughput and performance, accompanied by a 17.4× improvement in energy efficiency and a 0.06% enhancement in accuracy. It is noteworthy that despite only a 1.08× reduction in area, there is a 4.3× decrease in power consumption. This phenomenon is attributed to the proposed training method, which induces a significant reduction in signal switching, given that the weights tend to approximate to 1 or −1 in SC.

Table 3 presents a performance comparison with other FPGA implementations, including SC, BNN, and TC approaches. The table emphasizes the two most favorable outcomes for the pertinent metrics. In comparison with the most accurate study (BNFPGA18 [20]), our approach yields a four-fold increase in throughput, a four-fold improvement in performance, and a 21.6× enhancement in energy efficiency, albeit with a marginal 0.88% reduction in accuracy degradation (AD). Relative to a more recent investigation (SCFPGA24 [21]), our results exhibit a slightly inferior performance, characterized by a mere 0.61% reduction in accuracy degradation, a 1.44× slower processing speed, and a 1.97× decrease in performance. However, our approach showcases a mere 1.05× reduction in energy efficiency.

When exclusively considering SC implementations of the LeNet-5 architecture documented in the literature, we juxtapose the accuracy degradation among them when using the 8-bit approach (see Table 4). The row designated as “Test Platform” delineates the methodology employed for accuracy assessment, with “Sim” indicating simulation and “FPGA” indicating FPGA implementation. It is evident that accuracy degradation poses a significant challenge for SC-CNN designers, with the most favorable scenario achieving an AD of 0.14%. Conversely, in our study, we attained an AD of merely 0.01%, representing a 14-fold enhancement over the best-performing scenario (SCCNN24 [21]), while concurrently achieving the highest accuracy (98.97%). These findings underscore the efficacy of the proposed training approach, which systematically addresses SC-associated limitations while preserving hardware integrity [22].

Table 3. FPGA LeNet-5 implementations comparison. Notice the results for SCFPGA24 are for the 64 bitstream length case.

Metric	TCFPGA17 [23]	BNFPGA18 [20]	SCFPGA20 [24]	SCFPGA24 [21]	This Work
Year	2017	2018	2020	2024	2024
Architecture parallelism	Sequential	Sequential	Semi-Parallel	Parallel	Parallel
Computing paradigm	TC	BNN	SC	SC	SC
Activation/Weight bits	16/8	8/1	9/9	6/6	6/6
FPGA family	Virtex7	Stratix V	Zynq	Kintex7	Arria 10
FPGA name	VX690T	5SFSD8	XC7Z020	XC7K325T	GX1150
Frequency (MHz)	100	150	60	110	150
Software Acc (%)	99.17	98.70	98.67	98.36	98.98
Hardware Acc (%)	98.16	98.24	98.13	97.63	97.64
Acc Degradation (%)	1.01	0.46	0.54	0.73	1.34
Throughput (Images/s)	10,617	294,118	170	1,718,800	1,190,476
Performance (Images/s/MHz)	106	1961	3	15,626	7937
Power (W)	25.2	26.2	3.7	6.8	4.9
Energy efficiency (Images/J)	421	11,226	46	254,373	243,171
Logic used K (LUT or ALM)	233	0.182	28	153	318
DSP (blocks)	2907	20	0	0	0
Memory (Mbits)	17.2	44.2	1.7	0.0	0.0

Table 3. FPGA LeNet-5 implementations comparison. Notice the results for SCFPGA24 are for the 64 bitstream length case.

Metric	TCFPGA17 [23]	BNFPGA18 [20]	SCFPGA20 [24]	SCFPGA24 [21]	This Work
Year	2017	2018	2020	2024	2024
Architecture parallelism	Sequential	Sequential	Semi-Parallel	Parallel	Parallel
Computing paradigm	TC	BNN	SC	SC	SC
Activation/Weight bits	16/8	8/1	9/9	6/6	6/6
FPGA family	Virtex7	Stratix V	Zynq	Kintex7	Arria 10
FPGA name	VX690T	5SFSD8	XC7Z020	XC7K325T	GX1150
Frequency (MHz)	100	150	60	110	150
Software Acc (%)	99.17	98.70	98.67	98.36	98.98
Hardware Acc (%)	98.16	98.24	98.13	97.63	97.64
Acc Degradation (%)	1.01	0.46	0.54	0.73	1.34
Throughput (Images/s)	10,617	294,118	170	1,718,800	1,190,476
Performance (Images/s/MHz)	106	1961	3	15,626	7937
Power (W)	25.2	26.2	3.7	6.8	4.9
Energy efficiency (Images/J)	421	11,226	46	254,373	243,171
Logic used K (LUT or ALM)	233	0.182	28	153	318
DSP (blocks)	2907	20	0	0	0
Memory (Mbits)	17.2	44.2	1.7	0.0	0.0

Table 4. Comparison with other SC LeNet-5 implementations.

Metric	Year	Software Acc (%)	Hardware Acc (%)	Acc Degradation (%)	Test Platform
SCCNN19 [25]	2019	98.47	97.94	−0.53	Sim
SCFPGA20 [24]	2020	98.67	98.13	−0.54	FPGA
SCCNN21 [26]	2021	98.75	97.50	−1.25	Sim
SCFPGA22 [12]	2022	98.60	97.58	−1.02	FPGA
SCFPGA24 [21] (8-bits)	2024	98.36	98.22	−0.14	FPGA
This work (8-bits)	2024	98.98	98.97	−0.01	FPGA

Table 4. Comparison with other SC LeNet-5 implementations.

Metric	Year	Software Acc (%)	Hardware Acc (%)	Acc Degradation (%)	Test Platform
SCCNN19 [25]	2019	98.47	97.94	−0.53	Sim
SCFPGA20 [24]	2020	98.67	98.13	−0.54	FPGA
SCCNN21 [26]	2021	98.75	97.50	−1.25	Sim
SCFPGA22 [12]	2022	98.60	97.58	−1.02	FPGA
SCFPGA24 [21] (8-bits)	2024	98.36	98.22	−0.14	FPGA
This work (8-bits)	2024	98.98	98.97	−0.01	FPGA

Figure 10 depicts a comparison of the number of LFSR seeds producing various accuracies across different bit precisions for two academic endeavors: the SCFPGA22 study [12] and the present work under discussion. Remarkably, the training methodology introduced in this study facilitates the identification of optimal seeds for LFSR implementation. As the bit precision decreases, the extent of improvement becomes more pronounced. For the 8-bit precision scenario, it is noted that 60% of the seeds in the present study yield accuracies surpassing 98.5% (with an AD of merely 0.5%), while no seeds have been identified to achieve a comparable absolute difference value in the SCFPGA22 study [12]. Furthermore, at 6-bit precision, the maximum accuracy achieved in the [12] study is 17.7%, whereas this study achieves accuracies exceeding 95% with five different seeds.

7. Conclusions

Stochastic computing (SC) emerges as a viable approach for overcoming the difficulties of deploying neural networks (NNs) on hardware platforms. This research has concentrated on resolving the accuracy degradation (AD) challenges inherent in SC when applied to actual hardware systems by employing an arithmetic-aware training strategy. The outcomes have shown considerable advancements, achieving a reduction in AD by a factor of 100 without necessitating any modifications to the hardware setup. In comparison with similar studies, our method attains an exceptionally low AD rate of only 0.01%, which represents a 54-fold improvement over the best comparison case, while also delivering an enhanced performance in terms of speed (27 times faster) and energy efficiency (33 times more efficient). Furthermore, our training technique has aided in selecting LFSR seeds suitable for SC hardware applications, where 60% of the seeds show an AD below 0.5% and 83% maintain an AD under 5%. These results indicate that our method is more effective in securing precise outcomes with minimal variations from the desired accuracy, all while preserving the integrity of the hardware configuration. Finally, it is worth highlighting that the proposed method has been tested with the hardware proposed in [27], with a baseline MNIST test accuracy degradation of 1.02%, which is improved to 0.01%. Nevertheless, the proposed technique is not limited to this particular accelerator. In fact, implementations resulting in more precise SC multipliers like SCFPGA24 [21] are expected to improve in terms of accuracy degradation for 8-bit and lower weight and activation precision, just by modifying the way the NN is trained.