AERO: A 1.28 MOP/s/LUT Reconfigurable Inference Processor for Recurrent Neural Networks in a Resource-Limited FPGA
Abstract
:1. Introduction
2. Dataflow of RNN Inference
3. Proposed Processor: AERO
3.1. RNN-Specific Instruction-Set Architecture
3.2. Microarchitecture
3.2.1. Processing Pipeline
3.2.2. Vector Processing Unit Based on the Approximate Multipliers
- The truncation is performed by dropping the upper eight bits of an operand in the proposed multiplication approximation scheme. The truncation is performed in a consistent manner without regard to the RNN models and, thus, can be fulfilled by a simple logic circuitry picking the sub bit-vector at the fixed position as shown in Figure 3.
- A different truncation size might be considered in applying the proposed multiplication approximation scheme. When the truncation size is , 16-bit × 16-bit multiplication is carried out by the 16-bit ×-bit multiplier by dropping out the upper bits in one of the multiplication operands. With a larger , the multiplier becomes simpler so that its resource usage can become less. However, this may affect the inference results more severely because the probability that both of the two operands are not truncatable, which correspond to the last two cases in Table 4 bringing about approximation errors, may become larger. was determined to be 8 so that the proposed multiplication approximation scheme does not have a noticeable effect on the inference results, which were validated extensively based on the experimental results.
- The proposed scheme exploits the truncatability of the multiplication operands, which is highly probable in the inference based on the RNN models (e.g., vanilla RNN, GRU, and LSTM) that are already trained. Therefore, it does not entail any training issues necessarily addressed by a special methodology, such as the retraining [6]. It does not require any model modifications, either.
3.2.3. Activation Coefficient Unit Based on the Reduced Tables
3.3. Prototype Inference System
4. Results and Evaluation
5. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Abiodun, O.I.; Jantan, A.; Omolara, A.E.; Dada, K.V.; Mohamed, N.A.; Arshad, H. State-of-the-art in artificial neural network applications: A survey. Heliyon 2018, 4, e00938. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Athiwaratkun, B.; Stokes, J.W. Malware classification with LSTM and GRU language models and a character-level CNN. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2017), New Orleans, LA, USA, 5–9 March 2017; pp. 2482–2486. [Google Scholar]
- Jurgovsky, J.; Granitzer, M.; Ziegler, K.; Calabretto, S.; Portier, P.E.; He-Guelton, L.; Caelen, O. Sequence classification for credit-card fraud detection. Expert Syst. Appl. 2018, 100, 234–245. [Google Scholar] [CrossRef]
- Graves, A.; Liwicki, M.; Fernández, S.; Bertolami, R.; Bunke, H.; Schmidhuber, J. A novel connectionist system for unconstrained handwriting recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2008, 31, 855–868. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Stepp, H.; Jurgen, S. Long short-term memory. Neural Comput. 1997, 9, 1735–1780. [Google Scholar]
- Han, S.; Kang, J.; Mao, H.; Hu, Y.; Li, X.; Li, Y.; Xie, D.; Luo, H.; Yao, S.; Wang, Y.; et al. ESE: Efficient speech recognition engine with sparse LSTM on FPGA. In Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, Monterey, CA, USA, 22–24 February 2017; pp. 75–84. [Google Scholar]
- Azari, E.; Vrudhula, S. An Energy-Efficient Reconfigurable LSTM Accelerator for Natural Language Processing. In Proceedings of the 2019 IEEE International Conference on Big Data (IEEE BigData 2019), Los Angeles, CA, USA, 9–12 December 2019; pp. 4450–4459. [Google Scholar]
- Chang, A.X.M.; Martini, B.; Culurciello, E. Recurrent neural networks hardware implementation on FPGA. arXiv 2015, arXiv:1511.05552. [Google Scholar]
- Guan, Y.; Yuan, Z.; Sun, G.; Cong, J. FPGA-based accelerator for long short-term memory recurrent neural networks. In Proceedings of the 2017 22nd Asia and South Pacific Design Automation Conference (ASP-DAC), Tokyo, Japan, 16–19 January 2017; pp. 629–634. [Google Scholar]
- Wang, S.; Li, Z.; Ding, C.; Yuan, B.; Qiu, Q.; Wang, Y.; Liang, Y. C-LSTM: Enabling efficient LSTM using structured compression techniques on FPGAs. In Proceedings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, Monterey, CA, USA, 25–27 February 2018; pp. 11–20. [Google Scholar]
- Cao, S.; Zhang, C.; Yao, Z.; Xiao, W.; Nie, L.; Zhan, D.; Liu, Y.; Wu, M.; Zhang, L. Efficient and effective sparse LSTM on FPGA with bank-balanced sparsity. In Proceedings of the ACM/SIGDA Int’l Symp. Field-Programmable Gate Arrays, ACM, Seaside, CA, USA, 24–26 February 2019; pp. 63–72. [Google Scholar]
- Cho, K.; Van Merriënboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; Bengio, Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv 2014, arXiv:1406.1078. [Google Scholar]
- Nurvitadhi, E.; Sim, J.; Sheffield, D.; Mishra, A.; Krishnan, S.; Marr, D. Accelerating recurrent neural networks in analytics servers: Comparison of FPGA, CPU, GPU, and ASIC. In Proceedings of the 2016 26th International Conference on Field Programmable Logic and Applications (FPL), Lausanne, Switzerland, 29 August–2 September 2016; pp. 1–4. [Google Scholar]
- Chen, C.; Ding, H.; Peng, H.; Zhu, H.; Ma, R.; Zhang, P.; Yan, X.; Wang, Y.; Wang, M.; Min, H.; et al. OCEAN: An on-chip incremental-learning enhanced processor with gated recurrent neural network accelerators. In Proceedings of the ESSCIRC 2017 43rd IEEE European Solid State Circuits Conference, Leuven, Belgium, 11–14 September 2017; pp. 259–262. [Google Scholar]
- Li, S.; Wu, C.; Li, H.; Li, B.; Wang, Y.; Qiu, Q. FPGA acceleration of recurrent neural network based language model. In Proceedings of the 2015 IEEE 23rd Annual International Symposium on Field-Programmable Custom Computing Machines, Vancouver, BC, Canada, 2–6 May 2015; pp. 111–118. [Google Scholar]
- Li, Z.; Ding, C.; Wang, S.; Wen, W.; Zhuo, Y.; Liu, C.; Qiu, Q.; Xu, W.; Lin, X.; Qian, X.; et al. E-RNN: Design optimization for efficient recurrent neural networks in FPGAs. In Proceedings of the 2019 IEEE International Symposium on High Performance Computer Architecture (HPCA), Washington, DC, USA, 16–20 February 2019; pp. 69–80. [Google Scholar]
- Gao, C.; Rios-Navarro, A.; Chen, X.; Liu, S.C.; Delbruck, T. EdgeDRNN: Recurrent Neural Network Accelerator for Edge Inference. IEEE J. Emerg. Sel. Top. Circuits Syst. 2020, 10, 419–432. [Google Scholar] [CrossRef]
- Zeng, S.; Guo, K.; Fang, S.; Kang, J.; Xie, D.; Shan, Y.; Wang, Y.; Yang, H. An efficient reconfigurable framework for general purpose CNN-RNN models on FPGAs. In Proceedings of the 2018 IEEE 23rd International Conference on Digital Signal Processing (DSP), Shanghai, China, 19–21 November 2018; pp. 1–5. [Google Scholar]
- Elman, J.L. Finding structure in time. Cogn. Sci. 1990, 14, 179–211. [Google Scholar] [CrossRef]
- Le, Q.V.; Jaitly, N.; Hinton, G.E. A simple way to initialize recurrent networks of rectified linear units. arXiv 2015, arXiv:1504.00941. [Google Scholar]
- Gers, F.A.; Schraudolph, N.N.; Schmidhuber, J. Learning precise timing with LSTM recurrent networks. J. Mach. Learn. Res. 2002, 3, 115–143. [Google Scholar]
- Schuster, M.; Paliwal, K.K. Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 1997, 45, 2673–2681. [Google Scholar] [CrossRef] [Green Version]
- Kadetotad, D.; Berisha, V.; Chakrabarti, C.; Seo, J.S. A 8.93-TOPS/W LSTM recurrent neural network accelerator featuring hierarchical coarse-grain sparsity with all parameters stored on-chip. In Proceedings of the 2019 IEEE International Solid-State Circuits Conference (ISSCC 2019), San Francisco, CA, USA, 17–21 February 2019; pp. 119–122. [Google Scholar]
- Intel. Stratix V Device Handbook; Intel: San Jose, CA, USA, 2020. [Google Scholar]
- Xilinx. Xilinx Design Flow for Intel FPGA SoC Users; Xilinx: San Jose, CA, USA, 2018. [Google Scholar]
- Marcus, M.; Santorini, B.; Marcinkiewicz, M.A. Building a large annotated corpus of English: The Penn Treebank. Comput. Linguist. 1993, 19, 313–330. [Google Scholar]
- Kim, T.H.; Shin, J. A Resource-Efficient Inference Accelerator for Binary Convolutional Neural Networks. IEEE Trans. Circuits Syst. II Express Briefs 2020, 68, 451–455. [Google Scholar] [CrossRef]
Number of Steps | RNN Model Type | Workload (KOP/Step) | Accuracy (%) |
---|---|---|---|
16 | Vanilla RNN | 73 | 98.11 |
GRU | 222 | 98.83 | |
LSTM | 296 | 98.86 | |
32 | Vanilla RNN | 70 | 97.14 |
GRU | 218 | 98.80 | |
LSTM | 292 | 98.84 | |
64 | Vanilla RNN | 66 | 73.98 |
GRU | 210 | 98.19 | |
LSTM | 288 | 98.47 |
Register | Alias | Role |
---|---|---|
R0 | DST | Destination address in AM |
R1 | SRC0 | First source address in AM |
R2 | SRC1 | Second source address in AM |
R3–R7 | - | Placeholders |
R8 | BIAS | Bias address in BM |
R9 | WEIGHT | Weight address in WM |
R10 | DST_BOUND | Bound of the destination address |
R11 | SRC0_BOUND | Bound of the first source address |
R12–R15 | - | Placeholders |
Instruction | Function | Behavior | Format |
---|---|---|---|
MVMA | Matrix-vector MAC |
| |
EMAC.. | Elementwise MAC, where indicates that the result is accumulated and indicates the bitwise inversion of the first operand. |
| |
ENOF. | Elementwise non-linear function, where indicates the function type. |
| |
CSL Ra, | Constant load, where and is given by the 8-bit immediate constant. | | |
SHL Ra, | Shift and load, where and is given by the 8-bit immediate constant. | | |
ACC Ra, Rb, | Accumulate, where and is given by the 4-bit immediate constant. | | |
SAC Ra, Rb, | Shift and accumulate, where and is given by the 4-bit immediate constant. | |
Case a | Product | Example |
---|---|---|
x is truncatable. | x = 0xFF80, y = 0xABCD → = 0x80 × 0xABCD | |
x is not truncatable and y is truncatable. | x = 0x1234, y = 0x007D → = 0x1234 × 0x7D | |
Neither x nor y is truncatable and is on. | x = 0x12F4, y = 0x0BCD →≈ 0x12F4 × 0x0B ≪ 8 | |
Neither x nor y is truncatable and is off. | x = 0xAB12, y = 0xABCD →≈ 0xAB × 0xABCD ≪ 8 |
RNN Model Type | Vanilla RNN | GRU | LSTM | GRU | LSTM | Bi-Directional LSTM [22] | Peephole LSTM [21] | Bi-Directional LSTM [22] |
---|---|---|---|---|---|---|---|---|
Number of steps | 16 | 16 | 16 | 32 | 32 | 32 | 64 | 64 |
State size | 128 | 128 | 128 | 96 | 96 | 96 | 128 | 128 |
Workload (KOP/step) | 73.73 | 222.34 | 295.81 | 111.46 | 148.13 | 296.26 | 172.93 | 444.16 |
Processing latency (s/step) | 3.24 | 9.70 | 12.93 | 4.88 | 6.50 | 13.00 | 7.60 | 19.47 |
Inference accuracy (%) | 97.32 | 97.91 | 98.47 | 97.36 | 97.59 | 98.00 | 97.88 | 97.94 |
Normalized resource usage (LUT/step/s) | 0.06 | 0.17 | 0.23 | 0.09 | 0.12 | 0.23 | 0.14 | 0.35 |
Normalized energy consumption (J/step) | 0.45 | 1.34 | 1.79 | 0.67 | 0.90 | 1.80 | 1.05 | 2.69 |
RNN Model Type | LSTM | Bidirectional GRU [22] | GRU |
---|---|---|---|
State size | 64 | 64 | 128 |
Workload (KOP/step) | 98.75 | 148.61 | 222.34 |
Processing latency (s/step) | 4.33 | 6.50 | 9.70 |
Perplexity per word | 120.86 | 116.9 | 108.94 |
Norm. resource usage (LUT/step/s) | 0.08 | 0.12 | 0.17 |
Norm. energy consumption (J/step) | 0.60 | 0.90 | 1.34 |
Inference Processor | AERO | [6] | [7] | [8] | [9] | [10] a | [11] | [13] | |
---|---|---|---|---|---|---|---|---|---|
FPGA device | Name | Cyclone®-V | Kintex®-UltraScale | Zynq®-7000 | Zynq®-7000 | Virtex®-7 | Virtex®-7 | Arria®-10 | Stratix®-V |
Part number | 5CSXFC6D6 | XCKU060 | XC7Z030 | XC7Z020 | XC7VX485T | XC7VX690T | GX1150 | N.A. | |
Model reconfigurability | Yes | No (only LSTM) | No (only LSTM) | No (only LSTM) | No (only LSTM) | No (only LSTM) | No (only LSTM) | No (only GRU) | |
Inference speed (GOP/s) | 23.0 | 282.2 | 8.08 | 0.29 b | 7.26 | 131.1 | 304.1 | 126.70 | |
Precision c | FxP-16 | FxP-12 | FxP-8 | FxP-16 | FP | FxP-16 | FxP-16 | FP | |
Resource usage | LUT (K) | 18.0 | 293.9 | 23.0 | 7.6 | 198.3 | 504.4 | 578.0 d | 592.0 d |
BRAM (Kbit) | 1620 e | 34092 | 6480 | 576 | 38592 | 34768 | 48760 | 4000 | |
DSP | 64 | 1504 | 0 | 50 | 1176 | 2675 | 1518 | 256 | |
FF (K) | 10.1 | 453.1 | 28.4 | 12.9 | 182.6 | 199.7 | N.A. | N.A. | |
Resource efficiency (MOP/s/LUT) | 1.28 | 0.96 | 0.35 | 0.04 | 0.04 | 0.26 | 0.52 | 0.21 | |
Power consumption (W) | 0.77 (0.138) f | 41.00 | 1.19 | 1.94 | 19.63 | 22.00 | 19.10 | N.A. | |
Energy efficiency (GOP/J) | 29.08 (166.31) f | 6.88 | 6.79 | 0.15 | 0.37 | 5.96 | 15.92 | N.A. |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Kim, J.; Kim, J.; Kim, T.-H. AERO: A 1.28 MOP/s/LUT Reconfigurable Inference Processor for Recurrent Neural Networks in a Resource-Limited FPGA. Electronics 2021, 10, 1249. https://doi.org/10.3390/electronics10111249
Kim J, Kim J, Kim T-H. AERO: A 1.28 MOP/s/LUT Reconfigurable Inference Processor for Recurrent Neural Networks in a Resource-Limited FPGA. Electronics. 2021; 10(11):1249. https://doi.org/10.3390/electronics10111249
Chicago/Turabian StyleKim, Jinwon, Jiho Kim, and Tae-Hwan Kim. 2021. "AERO: A 1.28 MOP/s/LUT Reconfigurable Inference Processor for Recurrent Neural Networks in a Resource-Limited FPGA" Electronics 10, no. 11: 1249. https://doi.org/10.3390/electronics10111249