*3.2. System-Level Simulation for Pattern Recognition*

To validate the functionality of the proposed programming schemes, the single-layer artificial neural network system for the Modified National Institute of Standards and Technology (MNIST) pattern recognition was simulated. The MNIST database is a large database of handwritten digits, which contains about 60,000 learning images and 10,000 test images [19]. A total of 784 input neurons represent 28 × 28 pixels of each image and 10 output neurons represent 10 digits (0 ~ 9). We also used a rectifier linear unit (ReLU) as an activation function of neuron, which is one of the popular activation functions [20]. For the learning process, a supervised learning method was used. At first, the error was calculated at the output neurons. Next, the target change in synaptic weight (the number of programming pulses) was determined by the gradient descent method. After that, the synaptic weight value is updated based on fitted equations for the conductance modulation characteristics of a successive-pulse programming scheme and the ISPP scheme.

Figure 7a shows the system-level simulation result of the pattern recognition accuracy with the 10,000 test image samples. Compared to our previous work [11], the ISPP scheme can increase recognition accuracy by about 6% (a successive-pulse programming scheme in our previous work: 79.83% [11], and the ISPP scheme in this work: 85.9%). This result is in good agreement with the other papers that the linear conductance modulation characteristic is essential for the better performance of neuromorphic systems [5,21]. The synaptic weight maps after training 10,000 samples with the ISPP scheme are illustrated in Figure 7b.

**Figure 7.** Modified National Institute of Standards and Technology (MNIST) pattern recognition result. (**a**) Recognition accuracy comparison between a successive-pulse programming and the ISPP; (**b**) Synaptic weight map after training 10,000 samples with the ISPP scheme.

In addition, we examined the synaptic weight modulation characteristics according to the various values of *V*step in the ISPP scheme. As illustrated in Figure 8a, a smaller *V*step allows for fine conductance modulation, which means that the number of the synaptic weight level can be increased. As a result, the fine conductance modulation ability with a smaller *V*step can obtain more accurate pattern recognition rate, as shown in Figure 8b. It should be noted, however, that the retention characteristics (the ability to distinguish each level for a long time) can deteriorate when the interval between each synaptic weight level becomes narrow. Therefore, the magnitude of *V*step should be determined considering the trade-off relationship between the retention characteristic and the accuracy.

**Figure 8.** MNIST pattern recognition result by using the ISPP scheme. (**a**) The gradual conductance change by applying various *V*step; (**b**) Recognition accuracy as a function of the training samples for the various *V*step. The number means the weight levels (maximum pulse number).

## **4. Discussion**

Currently, numerous researches based on different types of nonvolatile memory devices have been conducted to implement neuromorphic hardware systems. Table 3 summarizes some of the research results.

**Table 3.** Comparison between several research results of neuromorphic applications.


Almost all previous studies are based on the 2-D synapse array structure, but for the first time we proposed the 3-D stacked synapse array structure. This paper has addressed several issues associated with the design of the 3-D synapse array architecture in terms of a full-chip level. This will be an important guideline for designing a 3-D stacked synapse array. The approach of stacking CTF devices is a mature technology that has been already used in commercialized 3-D NAND flash memories. Consequently, the proposed 3-D synapse architecture is expected to have a high possibility of actual mass production. Also, it can achieve excellent reliability by utilizing the various technologies used in NAND flash memory. For example, we have demonstrated that the ISPP method can improve the pattern recognition accuracy of a neuromorphic system.

For future work, we will demonstrate the superiority of the proposed 3-D synapse architecture based on an actual fabricated array. In addition, application researches to various artificial neural networks such as a convolutional neural network (CNN) and a recurrent neural network (RNN) will be a crucial topic.
