Real-Time Finger-Writing Character Recognition via ToF Sensors on Edge Deep Learning

Round 1

Reviewer 1 Report

The authors present in this paper a system for writing based on a ToF array. The authors stress how most of the devices commonly used to communicate with a machine are not always usable and therefore how different solutions for acquiring alphabetic characters need to be investigated.

I suggest the authors stress even more on the rationale behind this study. I would not simply say that other input devices are not usable in all circumstances but I suggest they clearly outline scenarios, application contexts or a category of users that could benefit from the use of the device they are proposing.

Overall, the work seems to be very interesting and the experimental validation of the approach presented is correctly set out and discussed. I therefore propose that minor revisions be made before proceeding with the publication of the work.

I suggest the authors merge and homogenise the introduction and related works section as the two report very similar information. it is therefore more appropriate to report a unified introduction section in which the context definition, literature review and definition of the objective of the specific work appear.

In the section "Structure of the Deep Learning Algorithms", the authors should specify how the implemented deep learning algorithms were chosen and justify the choice of neural architectures, perhaps also referring to other works in the literature.

Some minor comments are given below:

- Enlarge figure 1 and make the texts more readable.

- I would report the size of the system in figure 1.

- Figures 3 and 4 are not necessary. Time-of-flight technology is sufficiently familiar to readers and I do not think the figure adds value to the paper. At the same time, the description of the participants is done comprehensively in the written text.

- no figure 6 appears. adjust references to figures currently in the text.

Author Response

The authors present in this paper a system for writing based on a ToF array. The authors stress how most of the devices commonly used to communicate with a machine are not always usable and therefore how different solutions for acquiring alphabetic characters need to be investigated.

I suggest the authors stress even more on the rationale behind this study. I would not simply say that other input devices are not usable in all circumstances but I suggest they clearly outline scenarios, application contexts or a category of users that could benefit from the use of the device they are proposing.

Answer:  Firstly, we have added a little simple rationale behind in this study, such as typical application scenarios for our proposed system, experimental designs. In addition, our goal of this manuscript is not only to measure the prediction performance of the deep models, but also to demonstrate the capability of our on-device reference algorithm at resource-constrained MCUs in decoding the finger-writing patterns efficiently in real-time.

Also, for providing an efficient and low-cost text input solution targeted at small screen devices (e.g., smartwatch) or peoples with poor eyesight, we propose a finger-writing character recognition system based on an array of ToF Sensors.

Overall, the work seems to be very interesting and the experimental validation of the approach presented is correctly set out and discussed. I therefore propose that minor revisions be made before proceeding with the publication of the work.

 I suggest the authors merge and homogenise the introduction and related works section as the two report very similar information. it is therefore more appropriate to report a unified introduction section in which the context definition, literature review and definition of the objective of the specific work appear.

 Answer: According to your suggestion, we have merged the introduction and related works section.

In the section "Structure of the Deep Learning Algorithms", the authors should specify how the implemented deep learning algorithms were chosen and justify the choice of neural architectures, perhaps also referring to other works in the literature.

 Answer: We have improved the section "Structure of the Deep Learning Algorithms”, including reasons of selecting deep learning models and advantages of specific deep models for our proposed recognition system.

Some minor comments are given below:

- Enlarge figure 1 and make the texts more readable.

Answer: we have edited figure 1.

- I would report the size of the system in figure 1.

Answer: We have edited figure 1. And please see it in the updated draft, thanks.

- Figures 3 and 4 are not necessary. Time-of-flight technology is sufficiently familiar to readers and I do not think the figure adds value to the paper. At the same time, the description of the participants is done comprehensively in the written text.

Answer: We have deleted Figures 3 and 4.

- no figure 6 appears. adjust references to figures currently in the text.

Answer: We have adjusted the figure ID in our manuscript.

Reviewer 2 Report

The manuscript proposes a new finger-writing character recognition using ToF sensors and Deep Learning methods.

Some remarks:

1- The 1st paragraph presents much important information, but it lacks references;

2- " some works on" -> capital letter;

3- What is a shallow CNN?

4- All variables must be in italics;

5- The LSTM is usually applied to time series forecasting. Why use this architecture in recognition?

6- Is the CNN based on some known architecture, such as AlexNet or LeNet? Moreover, why did the authors not use a widely known approach?

7- Is it possible to use some shallow ANN? Please discuss.

Author Response

The manuscript proposes a new finger-writing character recognition using ToF sensors and Deep Learning methods.

Some remarks:

1- The 1st paragraph presents much important information, but it lacks references;

Answer: We have merged the introduction and related works section with more references.

2- " some works on" -> capital letter;

Answer: We have edited it.

3- What is a shallow CNN?

Answer: Convolutional neural networks (CNNs) are a type of neural network that works on the deep learning principle. CNN architecture includes several blocks including convolution layers, pooling layers, and fully connected layers. a conventional CNN model often requires lots of computing resources. While a shallow CNN architecture consists of fewer layers and small convolution kernels size as compared to deep architectures.

 The primary goal for the shallow CNN was to create a light architecture with a small number of parameters (weights) in order to avoid long computational times. therefore, the shallow CNN architecture is not only computationally efficient but also is able to avoid possible overfitting.

4- All variables must be in italics;

Answer: We have edited them.

5- The LSTM is usually applied to time series forecasting. Why use this architecture in recognition?

Answer: In our research, handwriting data were collected by an array of ToF every 30ms forming time-sequential information like that used in forecasting, and the results also showed LSTM achieved a high accuracy in recognition.

6- Is the CNN based on some known architecture, such as AlexNet or LeNet? Moreover, why did the authors not use a widely known approach?

Answer: Smaller deep learning models can work without issues on mobile, wearable, and IoT devices. however, more complex CNN models, such as AlexNet, do not work well under the resource-constrained edge devices [1]. of course, LeNet-A，the lightweight deep neural networks, is suitable for resource-constrained MCUs [2-3]. Next, we shall evaluate LeNet-A in our finger-writing character recognition.

Our reference:

1. Chauhan, Jagmohan, et al. "Breathing-based authentication on resource-constrained IoT devices using recurrent neural networks." Computer 51.5 (2018): 60-67.
2. Wang, Zhepeng, et al. "Lightweight run-time working memory compression for deployment of deep neural networks on resource-constrained MCUs." Proceedings of the 26th Asia and South Pacific Design Automation Conference. 2021.
3. Wu, Yawen, et al. "Enabling on-device cnn training by self-supervised instance filtering and error map pruning." IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 39.11 (2020): 3445-3457.

7- Is it possible to use some shallow ANN? Please discuss.

Answer: I think that a shallow Artificial Neural Network (ANN) could be applied in our finger-writing character recognition. as a simple supervised learning model, the shallow ANN model consists of an input, a hidden, and an output layer where the information moves in one direction from the input to the output layer. Compared with the conventional ANN and deep models (i.e., convolutional neural networks), the shallow ANN is able to maintain satisfactory accuracy with relatively low overhead on memory and run-time latency. the shallow ANN is a good choice for small and low-power embedded intelligent devices. However, there is need to handcraft features and employ a feature selection procedure using expert domain knowledge. And feature extraction and selections are challenging works. the key differences between shallow ANN and deep learning are the ability of deep model to automatically discover and extract the features from the big data. thus, the deployment of deep learning is more convenient.

Round 2

Reviewer 2 Report

Accept