Principal Component Analysis-Based Logistic Regression for Rotated Handwritten Digit Recognition in Consumer Devices

Abstract

Handwritten digit recognition has been used in many consumer electronic devices for a long time. However, we found that the recognition system used in current consumer electronics is sensitive to image or character rotations. To address this problem, this study builds a low-cost and light computation consumption handwritten digit recognition system. A Principal Component Analysis (PCA)-based logistic regression classifier is presented, which is able to provide a certain degree of robustness in the digit subject to rotations. To validate the effectiveness of the developed image recognition algorithm, the popular MNIST dataset is used to conduct performance evaluations. Compared to other popular classifiers installed in MATLAB, the proposed method is able to achieve better prediction results with a smaller model size, which is 18.5% better than the traditional logistic regression. Finally, real-time experiments are conducted to verify the efficiency of the presented method, showing that the proposed system is successfully able to classify the rotated handwritten digit.

1. Introduction

Computer vision has been widely used in many fields to enable image recognition [1]. As a result, there have been many applications of image recognition, such as automatic following car, license plate recognition, and facial recognition [2,3]. The work in [3] developed a Convolutional Neural Network (CNN)-based recognition system to recognize Peruvian license plates. The research developed a model with 100% accuracy, 0% failure, and 100% sensibility, with 100% specificity. It is a typical use of computer vision to perform image recognition. Among all of its uses, one of the most common applications in the field of consumer electronics is handwritten digit recognition, such as handwriting input on smartphones or smart pads.

A common use of the handwritten digit recognition system in common consumer mobile phones is shown in Figure 1. With a handwritten character “8”, as illustrated in Figure 1a, the typing system can automatically recognize the digit. However, this experiment shows that handwritten digit recognition only works successfully when the digit is vertically well aligned. Recognition failure will be induced when the digit is subjected to apparent rotation, as shown in Figure 1b.

Therefore, the novelty of the research is that it proposes a handwritten digit recognition system with high accuracy for rotated images that significantly reduces the requirements of high computation complexity and memory, such as the Convolutional Neural Network (CNN)-based method shown in ref. [4]. Unlike the CNN-based pattern recognition methods, the proposed method is adequate for implementation in low-cost embedded systems or consumer electronics with restricted storage capacities. To develop such a low-cost consumer electronics system, it is necessary to choose appropriate methods of image pose correction and image recognition algorithms.

There are a lot of studies working on handwritten image recognition. For example, ref. [5] proposed a Deep Convolutional Self-Organizing Maps (DCSOM) network to learn unlabeled visual data. The experimental result showed that the proposed DCSOM had a high accuracy when predicting noise digits. Ref. [6] proposed an offline handwritten digit recognition system, which was trained using the MNIST dataset. The proposed system is a CNN-based recognition system, which is one of the most common types of computer vision model. Ref. [7] proposed a DeblurGAN-CNN model to recognize character images with noise, which is a system composed of two networks. The proposed system mentioned in ref. [7] uses different datasets to test the system, and the experimental results showed that the recognition system could recognize images with noise. The methods proposed by different researchers all have relatively high accuracy compared to basic methods. However, training such large models often takes long time. This issue is one of the most significant disadvantages of using these methods to perform image recognition.

Due to the long training and recognition time, lots of studies have tried to solve this problem. Ref. [8] focused on the detection Speed of Synthetic Aperture Radar (SAR), which became faster using faster region-based CNN (R-CNN). Though the accuracy of the proposed model is roughly the same as that of conventional R-CNN, the recognition speed is 8 times faster. Ref. [9] developed a CNN-based Range Partition (CRP), which satisfied the requirements of fast packet classification and online rule updating. As seen in the two studies, nowadays, developing recognition system with shorter training times and faster recognition speeds is one of the most important objectives.

Out of all of the methods of solving recognition problems, logistic regression is one of the most common [10]. The idea of this method is to use the concept of probability to solve the multiple-class classification problem, which can, thus, be used to enable recognition. As it is a basic but powerful method, there are several studies using logistic regression to perform different types of tasks. Ref. [11] used online logistic regression to approach the parameter estimation problem to replace Sequential Monte Carlo (SMC), while ref. [12] used logistic regression to approach the binary dynamical process used to reconstruct a network. Logistic regression is used in these articles for different purposes, which shows that it is a basic but powerful algorithm in terms of modeling. In this article, it will be used to perform digit image recognition.

The other important field of digit recognition is image straightening. Inspired by the ideas of [13,14], the proposed method applies PCA to image rotation to increase the accuracy of handwritten digit identification. The two previous studies showed that PCA can be used to perform image alignment. Using PCA to straighten the rotated images, the accuracy of recognition can be improved. However, it is found that PCA has a problem of orientation uncertainty. When an image is aligned via PCA, the image might be upside down, as PCA only uses the direction of the first principal component. To solve this problem, ref. [13] used a method of finding the direction and the angle of the rotation. In this research, PCA is not only used to reduce the dimension of the data, but also to find appropriate axes to perform image rotation. Therefore, the result of the “principal component” via PCA has its physical meanings and is hard to replace with other feature extraction methods, such as Linear Discriminant Analysis (LDA). The LDA can produce a better data separation result, since it can maximize the separation between different classes and minimize it between the data with the same classes. However, the coordinates found via LDA are not guaranteed to be orthogonal, which may cause the digits to distort after rotation.

There are also different methods used to find the orthogonal coordinates to prevent distortion, such as Sparse Principal Component Analysis (SPCA) and Multilinear Principal Component Analysis (MPCA). However, they require more time and computation complexity compared to conventional PCA. Considering the real-time application in electronic devices with limited storage capacities, PCA will be the best candidate.

Unlike the paper in ref. [13], PCA is also utilized to reduce the dimension of the features to perform classification, as illustrated in refs. [15,16]. With a reduced data dimension, the training process can be much faster than traditional logistic regression. This speed allows the proposed method to use fewer parameters and reduces the storage capacities required to perform implementation. Similarly, although the identification accuracy in refs. [17,18] may have better results than the proposed method, it can be challenging to realize these methods in embedded systems or consumer electronic devices. In contrast, the proposed method has the advantage of seamless integration into real-time applications of constrained storage devices.

Based on this advantage, PCA is, thus, used in some studies that focus on high-dimensional data. Ref. [19] proposed a deep method based on PCA and DCNN to classify normal traffic and abnormal traffic, while ref. [20] used PCA in many different cases to verify that PCA and random projection can be used to perform dimension reduction. On the other hand, the low recognition time can make the real-time recognition frequency much higher. This fact helps improve the performance of a trained model when performing the real-time task.

In this article, an improved PCA is proposed to work with logistic regression, which solves the orientation uncertainty of traditional PCA. The model trained via PCA-based logistic regression has a much better performance than that trained via traditional logistic regression and Neural Networks (NN).

Experiments show that the model trained via the proposed method can successfully recognize the oblique handwritten digits that cannot be recognized on current mobile phones, as shown in the case demonstrated in Figure 1. Accordingly, the method’s three main advantages are summarized as follows:

• Better robustness of the trained model. The model trained via the proposed method can recognize a rotated image, which allows the model to work well, despite rotation issues affecting the incoming data.
• Faster training process and higher real-time recognition frequency. Using PCA to reduce the data dimension, the model can recognize the image using much fewer features, which makes the model easier to train and require less memory storage to perform implementation.
• Higher accuracy than many other classifiers. The PCA-based logistic regression presented in this article can solve the problem of orientation uncertainty, which leads to the higher accuracy of testing data.
• Owing to the light computational complexity, the developed handwritten digit recognition algorithm can be realized in an embedded system or integrated into current consumer 3C devices.

Using the proposed PCA-based logistic regression method, the accuracy of testing using the rotated testing set can reach 77.5%. Compared to other algorithms trained via the MATLAB toolbox, the proposed method has a much faster training process and higher testing accuracy. The experimental result shows that the three contributions make the proposed method a potential tool for performing real-time recognition.

In Section 2.1, the research roadmap will be given to enable a clear understanding of the structure of the proposed method. The used methodology will be introduced, as will the processes of recognition. The detail of each method will then be explained in the rest of Section 2. The experimental platform description and associated result are presented in in Section 3 and Section 4, respectively. The further comparison studies are considered and discussed in Section 5. Finally, the conclusion of the article is given in Section 6.

2. Methodology

2.1. Description of Research Roadmap

The structure of the PCA-based logistic regression proposed in this article is shown in Figure 2. The proposed method was divided into two parts: data pre-processing and logistic regression. The first part used PCA to straighten the image, smoothened the image using a convolution filter, and reduced the dimension of the whole dataset. The second part was logistic regression, which could efficiently identify the class of each image.

To straighten a rotated image, the coordinates of each pixel on the image were treated as data points. By finding the first and second principal components of all of the data points on one image, the two components were used as a new basis. This new basis spanned a subspace onto which the data points were mapped. Once the data points were mapped onto the subspace, the digit image composed of these data points became straight.

However, there was a downside to using data points in the subspace to draw a digit. As the new coordinate of each pixel might not have been an integer after mapping, the data points would deviate from the correct pixel after rounding. This issue led to some noise being present on the straightened digit image. To remove the noise caused by the image straightening, a convolution filter was used in this study. The details on how to identify the type of noise and choose the kernel are described in the methodology section.

Having a straightened digit image, the whole dataset dimension was reduced via PCA. Each image in MNIST had 28 × 8 pixels, which is a large number for model training. It was extremely inefficient to train a model with all 784 features derived from 60,000 images. To make the training more efficient, this study used PCA to find the principal components of the whole dataset. The MNIST dataset was mapped onto the subspace spanned by sufficient principal components.

After all of the previous pre-processing, the dataset was used to train a model using logistic regression. The recognition model trained via the proposed method contained the data pre-processing and the matrix from logistic regression. It was tested via a validation set to check the result and determine necessary adjustments. The testing dataset was used to test the model after producing a proficient accuracy of validation. In the experiment, the validation set was not used again to perform testing. The original data were split into a training dataset, validation dataset, and testing dataset. The model after training was first evaluated using the validation set. If the validation accuracy was high enough, the model was then examined using the testing dataset.

The model was eventually used to perform real-time digit recognition. It was proved that the proposed method could be used to recognize different digits in the experiment. This result made the proposed method a potentially highly useful tool for usage in consumer electronics.

2.2. Logistic Regression

Compared to linear regression, logistic regression is a more advanced method with a logic function. The structure of the logistic regression is shown in Figure 3, which shows that the class with the highest probability will be considered to be the class to which the input belongs.

The input is multiplied by a weighting matrix with a bias added, which will be placed into a logic function to generate a probability-based output [21]. The mathematic expression can be written as follows:

$$\begin{array}{ccc}{f}_{w_i}\left(x\right)=s\left(w_i{}^{T}x\right)={\widehat{y}}_i& ,& s\left(w_i^{T}x\right)=\frac{e^{-w_i^{T}x}}{{\displaystyle \sum _{i=1}^m{e}^{-w_i^{T}x}}}\end{array}$$

(1)

To identify the difference between the output and the target, a loss function is required. The difference between two probability distributions can be found using maximum likelihood as follows:

$$L\left(W\right)={\displaystyle \prod _{j=1}^n{\displaystyle \prod _{i=1}^{m}s{\left(w_i^T{x}_j\right)}^{y_{ji}}}}$$

(2)

By taking the negative nature log of the likelihood, the equation becomes cross-entropic in nature [22].

$$\begin{array}{cc}-\ln L\left(W\right)& =-{\displaystyle \sum _{j=1}^n{\displaystyle \sum _{i=1}^m{y}_{ji}\ln s\left(w_i^T{x}_j\right)}}\hfill \\ & =-{\displaystyle \sum _{j=1}^n{\displaystyle \sum _{i=1}^m{y}_{ji}\left(w_i^{T}x-\ln {\displaystyle \sum _{i=1}^m{e}^{-w_i^T{x}_j}}\right)}}\hfill \end{array}$$

(3)

In other words, finding maximum likelihood has the same purpose of finding minimum cross-entropy, which is shown in (3). Taking the first order of the $a_{th}$ class that the $j_{th}$ data belong to, it becomes

$$\frac{\partial -\ln L\left(W\right)}{\partial w_a}=-{\displaystyle \sum _{j=1}^n{\displaystyle \sum _{i=1}^m{y}_{ji}\left(\frac{\partial w_i^{T}x}{\partial w_a}-\frac{\partial \ln {\displaystyle \sum _{i=1}^m{e}^{-w_i^{T}x}}}{\partial w_a}\right)}}$$

(4)

The minimum cross-entropy can be found by differentiating (4) on a term-by-term basis as follows

$$\frac{\partial w_i^T{x}_j}{\partial w_a}=\{\begin{array}{l}\begin{array}{ccc}{x}_j& ,& i=a\end{array}\\ \begin{array}{ccc}0& ,& i\ne a\end{array}\end{array}$$

(5)

$$\begin{array}{cc}\frac{\partial \ln {\displaystyle \sum _{i=1}^m{e}^{-w_i^T{x}_j}}}{\partial w_a}& =\frac{1}{{\displaystyle \sum _{i=1}^m{e}^{-w_i^T{x}_j}}}\frac{\partial {\displaystyle \sum _{i=1}^m{e}^{-w_i^T{x}_j}}}{\partial w_a}\hfill \\ & =\frac{x{e}^{-w_a^T{x}_j}}{{\displaystyle \sum _{i=1}^m{e}^{-w_i^T{x}_j}}}=s\left(w_a^T{x}_j\right)x_j\hfill \end{array}$$

(6)

Notice that $\frac{\partial w_i^T{x}_j}{\partial w_a}=x_j$ only at the $a_{th}$ class. Substitute both terms back to the cross-entropy equation as follows:

$$\begin{array}{cc}\frac{\partial -\ln L\left(W\right)}{\partial w_a}& =-{\displaystyle \sum _{j=1}^n{\displaystyle \sum _{i=1}^m{y}_{ji}\left(\frac{\partial w_i^T{x}_j}{\partial w_a}-\frac{\partial \ln {\displaystyle \sum _{i=1}^m{e}^{-w_i^T{x}_j}}}{\partial w_a}\right)}}\hfill \\ & =-{\displaystyle \sum _{j=1}^n{\displaystyle \sum _{i=1}^m{y}_{ji}\left(x_j\left(i=a\right)-s\left(w_a^T{x}_j\right)x_j\right)}}\hfill \\ & =-{\displaystyle \sum _{j=1}^n{y}_{ja}\left(1-s\left(w_a^T{x}_j\right)\right)x_j}\hfill \\ & =-{\displaystyle \sum _{j=1}^n\left(y_{ja}-s\left(w_a^T{x}_j\right)\right)x_j}\hfill \end{array}$$

(7)

This equation is the result of the $a_{th}$ class to which the $j_{th}$ data belong. Now, if we take the first-order differentiation of $b_{th}$ class to which the $j_{th}$ data do not belong, the result becomes

$$\begin{array}{cc}\frac{\partial -\ln L\left(W\right)}{\partial w_b}& =-{\displaystyle \sum _{j=1}^n{\displaystyle \sum _{i=1}^m{y}_{ji}\left(\frac{\partial w_i^T{x}_j}{\partial w_b}-\frac{\partial \ln {\displaystyle \sum _{i=1}^m{e}^{-w_i^T{x}_j}}}{\partial w_b}\right)}}\hfill \\ & =-{\displaystyle \sum _{j=1}^n{\displaystyle \sum _{i=1}^m{y}_{ji}\left(0\left(i=b\right)-s\left(w_b^T{x}_j\right)x_j\right)}}\hfill \\ & =-{\displaystyle \sum _{j=1}^n{y}_{ja}\left(0-s\left(w_b^T{x}_j\right)\right)x_j}\hfill \\ & =-{\displaystyle \sum _{j=1}^n\left(y_{jb}-s\left(w_b^T{x}_j\right)\right)x_j}\hfill \end{array}$$

(8)

where $y_{ja}=1$ and $y_{jb}=1$. The result is the same, no matter which class of weight we differentiate. Thus, to perform the update of the matrix of all classes, we have

$$W_{new}=W-\eta \cdot {\displaystyle \sum _{n=1}^i-\left[Y_j-s\left(W^T{x}_j\right)\right]x_j}$$

(9)

The physical meaning is that the weights will be updated according to the error of the output, considering the learning rate. This approach can lead to the weighting matrix that yields the output with high accuracy.

2.3. Principal Component Analysis

In this article, image pre-processing will be focused on applying PCA to each of the digit images to perform image straightening and the whole dataset to perform data dimension reduction. To explain how PCA works with regard to image rotation, the article first employs a two-dimensional image dataset as an illustration example.

Let $P$ be a two-dimensional dataset, with $n$ data points having two features as follows:

$$\begin{array}{ccc}P=\left[\begin{array}{cccc}{p}_1& p_2& \cdots & p_n\end{array}\right]\in {ℝ}^{2\times n}& ,& p_i=\left[\begin{array}{c}{x}_i\\ y_i\end{array}\right]\end{array}$$

(10)

where $x_i$ and $y_i$ represent the pixel coordinate of a digit for the x-axis and y-axis, respectively. The idea of PCA is to find a new basis for a subspace of the space to which the original features belongs, which gives the largest variance of the projected feature.

This demand of maximum variance can be achieved by taking

$$\underset{{\alpha }_1\in {\mathrm{R}}^{m\times 1}}{\arg \max }{\displaystyle \sum _{i=1}^n{\left({\alpha }_1^T\left(p_i-\mu \right)\right)}^2}=\underset{{\alpha }_1\in {\mathrm{R}}^{m\times 1}}{\arg \max }{\displaystyle \sum _{i=1}^n{\left({\alpha }_1^T{p}_{ci}\right)}^2}=\underset{{\alpha }_1\in {\mathrm{R}}^{m\times 1}}{\arg \max }{\displaystyle \sum _{i=1}^n\left({\alpha }_1^T{p}_{ci}{p}_{ci}^T{\alpha }_1\right)}$$

(11)

where $\mu$ represents the mean value of (10), and ${\alpha }_1$ represents the 1st principle axis.

Consider the unit vector constraint ${\alpha }_1^T{\alpha }_1=1$, imposing a Lagrange operator as follows:

$$\underset{{\alpha }_1\in {\mathrm{R}}^{m\times 1}}{\arg \max }{\displaystyle \sum _{i=1}^n\left({\alpha }_1^T{p}_{ci}{p}_{ci}^T{\alpha }_1\right)}-{\lambda }_1\left({\alpha }_1^T{\alpha }_1-1\right)$$

(12)

where the maximum value can be found by taking the partial derivative with respective to ${\alpha }_1$ and ${\lambda }_1$, which leads to

$$\frac{\partial }{\partial {\alpha }_1}\underset{{\alpha }_1\in {\mathrm{R}}^{m\times 1}}{\arg \max }{\displaystyle \sum _{i=1}^n\left({\alpha }_1^T{p}_{ci}{p}_{ci}^T{\alpha }_1\right)}-{\lambda }_1\left({\alpha }_1^T{\alpha }_1-1\right)=0\Rightarrow {\displaystyle \sum _{i=1}^n\left(p_{ci}{p}_{ci}^T{\alpha }_1\right)}-{\lambda }_1{\alpha }_1=0$$

(13)

and

$$\frac{\partial }{\partial {\lambda }_1}\underset{{\alpha }_1\in {\mathrm{R}}^{m\times 1}}{\arg \max }{\displaystyle \sum _{i=1}^n\left({\alpha }_1^T{p}_{ci}{p}_{ci}^T{\alpha }_1\right)}-{\lambda }_1\left({\alpha }_1^T{\alpha }_1-1\right)=0\Rightarrow {\alpha }_1^T{\alpha }_1-1=0$$

(14)

Based on (13), it is indicated that

$$M_c{\alpha }_1-{\lambda }_1{\alpha }_1=0$$

(15)

where $M_c={\displaystyle \sum _{i=1}^n\left(p_{ci}{p}_{ci}^T\right)}$ is the covariance matrix, which plays a key role in the PCA algorithm.

To calculate the second principal axis, consider the constraints

$${\alpha }_2^T{\alpha }_2=1,{\alpha }_1^T{\alpha }_2=0$$

(16)

Equation (16) shows that the principal axes are mutually orthogonal, which guarantees the shape and angle preservations during the rotation.

Based on (16), we considered another maximization problem using Lagrange operators as follows:

$$\underset{{\alpha }_2\in {ℝ}^{m\times 1}}{\arg \max }{\alpha }_2^{T}M{\alpha }_2-{\lambda }_2\left({\alpha }_2^T{\alpha }_2-1\right)+{\phi }_1\left({\alpha }_1^T{\alpha }_2-0\right)$$

(17)

Taking the partial derivative of (17) w.r.t ${\alpha }_2$ gives

$$2M{\alpha }_2-2{\lambda }_2{\alpha }_2+{\phi }_1{\alpha }_1=0$$

(18)

Multiplying ${\alpha }_1$ on both side gives

$$\begin{array}{l}\Rightarrow 2{\alpha }_1^{T}M{\alpha }_2-2{\lambda }_2{\alpha }_1^T{\alpha }_2+{\phi }_1{\alpha }_1^T{\alpha }_1=0\\ \Rightarrow {\phi }_1\underset{1}{\underbrace{{\alpha }_1^T{\alpha }_1}}=0\\ \Rightarrow {\phi }_1=0\end{array}$$

(19)

Therefore, from (15) and (18), we can conclude that

$${\mathrm{M}}_c{\alpha }_1={\lambda }_1{\alpha }_1$$

(20)

$${\mathrm{M}}_c{\alpha }_2={\lambda }_2{\alpha }_2$$

(21)

which leads to the standard “eigen-value” and “eigen-vector” problem.

So far, it can be seen that the first and second principal axes satisfy the orthogonal properties. Thus, these axes can be applied for 2D image rotation without causing pixel distortions. As a result, when the two eigen-vectors with the first two largest eigen-values are set to be the projections vectors, it is guaranteed that the two vectors are orthogonal, and the variances after the projection remain the largest.

For high-dimensional conditions, the PCA can be used to perform dimensional reduction. Based on the above derivation, we can find the following extension

$$\left[\begin{array}{cccc}{M}_c{\alpha }_1& M_c{\alpha }_2& \cdots & M_c{\alpha }_m\end{array}\right]=\left[\begin{array}{cccc}{\lambda }_1{\alpha }_1& {\lambda }_2{\alpha }_2& \cdots & {\lambda }_m{\alpha }_m\end{array}\right]$$

(22)

where

$$\{\begin{array}{l}\begin{array}{ccc}{\alpha }_i^T{\alpha }_j=1& if& i=j\end{array}\\ \begin{array}{ccc}{\alpha }_i^T{\alpha }_j=0& if& i\ne j\end{array}\end{array}$$

(23)

Recalling the singular value decomposition (SVD), we give a data set $P_c\in {ℝ}^{n\times m}$, and we have

$$\begin{array}{cc}{M}_c& =P_c^T{P}_c={\left(U\Sigma V^T\right)}^T\left(U\Sigma V^T\right)\hfill \\ & =V{\Sigma }^T{U}^{T}U\Sigma V^T\hfill \\ & =V{\Sigma }^T\Sigma V^T\hfill \end{array}$$

(24)

where $V:=\left[\begin{array}{cccc}{v}_1& v_2& \cdots & v_m\end{array}\right]$ and ${\Sigma }^T\Sigma :=diag\left({\sigma }_1,{\sigma }_2,\cdots ,{\sigma }_m\right)$, which represent the singular vector and singular value of $M_c$.

Equation (24) can be re-represented by

$$\left[\begin{array}{cccc}{M}_c{v}_1& M_c{v}_2& \cdots & M_c{v}_m\end{array}\right]=\left[\begin{array}{cccc}{\sigma }_1{v}_1& {\sigma }_2{v}_2& \cdots & {\sigma }_m{v}_m\end{array}\right]$$

(25)

Comparing (25) to (22), it can be seen that the eigen-values of the covariance matrix $M_c$ are also the singular value of the covariance matrix $M_c$. Moreover, for (24), the covariance matrix $M_c$ can be written as the combination of matrices, that is

$$M_c={\sigma }_1{v}_1{v}_1^T+{\sigma }_2{v}_2{v}_2^T+\cdots +{\sigma }_m{v}_m{v}_m^T={\displaystyle {\sum }_{i=1}^m{\sigma }_i{v}_i{v}_i^T}$$

(26)

Equation (26) represents the essential concept of the matrix significance contribution and can be applied to perform dimension reduction, which will be introduced later in this paper.

Using the first two principal components is the same as using the first two eigenvectors as a new basis to perform a linear transformation. We applied the PCA using the first two principal components of dataset $P$, and the result is shown in Figure 4.

It shows that the data points have been rotated after applying PCA, which is the key property that will be applied to the MNIST dataset to solve the image straightening problem. We noticed that the second principal component needed to be used as the negative basis to prevent the numbers from flipping.

2.4. Image Straighten Using PCA

As shown in the previous section, a two-dimensional dataset can be straightened via PCA. We implemented PCA using the MNIST dataset, and the result is shown in Figure 5. Regarding image rotation of the 2D cases, a computation trick is presented in Appendix A, which can greatly enhance the computation speed and is useful when performing real-time implementation.

This result shows that it is possible to straighten the image via PCA if each pixel is viewed as a data point instead of a feature. However, there are two problems involved in using PCA to straighten an image. The first problem is orientation uncertainty. Using principal components found via traditional PCA as the new basis leads to digit straightening according to the direction of the components. However, the direction of the components might not be appropriate for some digits to follow. An improved PCA is proposed in this study, which is described in Section 2.5. The other problem concerns the new data point locations determined via PCA, which are composed of decimals rather than integers. As the image is composed of 28 × 28 pixels, the locations of the data points need to be integers. This issue causes some noise when the locations are rounded to integers. Therefore, an appropriate smoothing method needs to be provided to work with PCA to perform image straightening. The image smoothing issue is discussed in Section 2.6.

2.5. PCA Improving

Although PCA can be used to straighten an image, the traditional PCA remains subject to some problems. The examples of the two types of problems are shown in Figure 6.

The first problem is orientation uncertainty. As the first principal component might not be pointing upward, the straightened image might be rotated. The second problem is horizontal digits. If the digit has a horizontal ellipse shape, the image might be rotated 90 degrees in a leading or lagging manner.

To solve these problems, the first step is to consider the principal component matrix to be a rotation matrix as follows:

$$U=\left[\begin{array}{cc}{u}_{1,1}& u_{1,2}\\ u_{2,1}& u_{2,2}\end{array}\right]=\left[\begin{array}{cc}\cos \left(\theta \right)& -\sin \left(\theta \right)\\ \sin \left(\theta \right)& \cos \left(\theta \right)\end{array}\right]$$

(27)

As the matrix $U$ is the new basis of the projected subspace, the matrix will lead to the first issue if the component $u_{2,2}$ is negative. Therefore, the matrix should be multiplied by the negative component if $u_{2,2}$ is negative as follows:

$$u_{2,2}<0\Rightarrow U=-\left[\begin{array}{cc}{u}_{1,1}& u_{1,2}\\ u_{2,1}& u_{2,2}\end{array}\right]$$

(28)

On the other hand, the pre-processing will encounter the second issue if the image is rotated more than 60 degrees. According to the practical observations, human-written digits are rarely rotated over 60 degrees. Furthermore, some digits may become similar when the rotation angle is too large, such as the digits “6“and “9”. Therefore, it is reasonable to have a rotation angle threshold for the proposed method.

Whether the matrix is 90 degrees leading or lagging can be found based on the value of $\mathrm{atan}2\left(u_{1,1},u_{1,2}\right)$. The image is rotated clockwise if the value is positive, which means that the angle of the rotation matrix should be positive, and vice versa.

$$\begin{array}{c}\mathrm{atan}2\left(u_{1,2},u_{1,1}\right)>0\Rightarrow U=U_{CW}\\ \mathrm{atan}2\left(u_{1,2},u_{1,1}\right)<0\Rightarrow U=U_{CCW}\end{array}$$

(29)

Equation (29) shows how the rotation matrix is corrected. By correcting the matrix encountering the second problem in this way, the image can be correctly straightened. These two matrix corrections improve the performance of PCA. The model using this improved PCA has a 87.15% training accuracy, while the model using traditional PCA only has an 81.22% accuracy.

The proposed improved PCA prevents a normal digit from rotating due to image straightening. It makes the model performance better compared to using traditional PCA.

2.6. Image Smoothing

As mentioned in the previous section, the straightened images have some noises, which worsens the performance of the model. Therefore, it is necessary to remove those noises.

The noises are caused by missing pixels after straightening, as the data points might not be an integer after linear transformation. The rounding of image pixel values from decimals to integers leads to some holes on the digits, such as the rounding results, not being continuous. The holes become noises when the data points are seen as one image.

Considering the causes and distribution of noises, the noise type is very similar to pepper noise, which is often seen in computer vision studies. Therefore, an ideal filter for filtering the image should be a filter that can remove this kind of noise. There are two common filters used to perform image demising: the Gaussian filter and the median filter [23].

The way that convolution works is shown in Figure 7. A kernel, which is also known as filter, needs to be chosen to perform convolution. The kernel matrix is implemented on an image following a certain direction, normally from left to right and top to bottom. The matrix kernel and a part of an image with 3 × 3 pixels is multiplied on an element-by-element basis. The summation value of the part is used as the pixel value of the new image. For the number of pixels used as a filter, 3 × 3 pixels is the minimum number required to perform image smoothing. This result shows that the proposed recognition system is able to use fewer parameters to achieve high accuracy in terms of handwritten digit recognition.

For a 3 by 3 Gaussian filter, the equation can be written in kernel form as follows:

$$K_g=\frac{1}{{\displaystyle \sum G\left(x,y\right)}}\left[\begin{array}{ccc}G\left(-1,-1\right)& G\left(0,-1\right)& G\left(1,-1\right)\\ G\left(-1,0\right)& G\left(0,0\right)& G\left(1,0\right)\\ G\left(-1,1\right)& G\left(0,1\right)& G\left(1,1\right)\end{array}\right]$$

(30)

where

$$G\left(x,y\right)=\frac{1}{2\pi {\sigma }^2}\exp \left(-\frac{x^2+y^2}{2{\sigma }^2}\right)$$

(31)

The kernel (30) is implemented on an image and generates the output via convolution. The result of this kernel is a low-pass filter, which works by giving different weights to each pixel to smoothen the image.

$$K_m=\left[\begin{array}{ccc}{I}_{x-1,y-1}& I_{x,y-1}& I_{x+1,y-1}\\ I_{x-1,y}& I_{x,y}& I_{x+1,y}\\ I_{x-1,y+1}& I_{x,y+1}& I_{x+1,y+1}\end{array}\right]$$

(32)

where $I_{x,y}=median\left(K_m\right)$. For the median filter, the median value of the image that the kernel covered is taken to replace $I_{x,y}$, which is shown as (32). This approach can prevent the image’s features from becoming smoothed too much while removing the values that are extremely different from those of its neighbors. Due to this property, the median filter is a more appropriate filter for removing impulse noise.

The image’s appearance when filtered using each filter is shown in Figure 8. The image filtered using the median filter retains more features than that using the Gaussian filter. Therefore, this article chooses the median filter to work with PCA to perform image pre-processing.

3. Experiment Background

3.1. Data Image

Figure 9 shows the images of MNIST. The MNIST dataset has been well studied in recent years with regard to image recognition, especially handwritten digits. Therefore, the MNIST dataset is utilized to train and validate the proposed method in this paper. In this article, the training data will be divided into a training set and a validation set, which contain 50,000 and 10,000 images, respectively.

The testing set will only be used to perform the final testing to ensure the objectivity of the testing result. To verify whether the model is able to recognize the rotated digits that are not similar to those of the training set, the testing set is manually rotated. Each of the figures is composed of 28 × 28 pixels.

3.2. Hardware Specification

The hardware specifications used in the experiment and real-time recognition are shown in

Table 1. Hardware specification for experiment.

Model	Information
System	Windows 10 22H2
Software	MATLAB 2023a
CPU	11th Gen Intel(R) Core(TM) i7-11800H @ 2.30 GHz
RAM	16 GB
Handwritten Board	Bamboo CTH-470

. All of the models are trained using the same computer for a fair comparison. Despite the proposed model being trained via the computer, a much simpler computer or computation unit is sufficient to perform the training of the proposed model.

3.3. Real-Time Recognition System

The model is eventually used to create a real-time handwritten digit recognition system, as shown in Figure 10. The recognition system is built via MATLAB to enable the calculation and user interface, and a handwriting board is used to perform digit writing.

The system is connected as shown in Figure 11. The digit is written by the user using the handwriting board, which is the input of the recognition system. Both the recognition result and the handwritten digit display are calculated using the same computer. explanation

3.4. Comparison Study Platform

The results of other comparison methods shown in Table 1 are conducted via the MATLAB Classification Learner toolbox. The interface can be seen in Figure 12. The MATLAB toolbox is able to train different methods and optimize their recognition results.

4. Experimental Result

4.1. MNIST Pre-Processing

Before performing logistic regression, image pre-processing needs to be implemented on the images derived from MNIST. The result of the pre-processed testing data is shown in Figure 13. The pre-processed data have much reasonable digit orientation, which makes the digits easier for the model to recognize.

After image pre-processing, PCA is implemented on the whole dataset to perform dimension reduction. We recall (26), where a data were actually composed of all the principal component. Therefore, the first few principal components with larger singular values are sufficient to perform recognition. The result of the training with different numbers of features is shown in Figure 14. The training result using only 100 features would be as accurate as that using 784 features.

Each of the principal components can be reconstructed to form a 28 by 28 image using each data component, as shown in Figure 15. The result shows the importance of the principal component that drops with the order of it, where the 784th component has nothing at all.

As mentioned, the accuracy of the logistic regression using 100 features is as high as that using 784 features, the input images of which are shown in Figure 16, along with reconstructed images of different dimension subspaces. The $m$ represents the dimension of the projected subspace of the original 784-dimensional feature space.

The result shows that the PCA can help to reduce the data to a sufficient dimension, which apparently reduces the training time.

4.2. MNIST Recognition Result

The final training method uses both training and validation sets to perform model training without rotation, while the testing set is rotated manually. The reason for keeping the training set in a non-rotated form is to make sure that all of the compared methods are fairly trained and used. The experimental result is shown in Figure 17.

The training accuracy is 87.15%, while the testing accuracy is 73.50%. The accuracy is calculated by dividing the number of the successfully recognized images by the total number of MNIST images.

The result shows that even if the model is trained using raw images derived from MNIST, it can still recognize a rotated MNIST digit image. The result shows that the model trained via the proposed method can be used on consumer electronics, which are often required to be robust to the unseen rotated images in the training data.

The model trained via the proposed method can be used to recognize the digit that cannot be recognized on a mobile phone, as shown in Figure 18. The characters in red blocks represent the recognition results for smart phones, while the number with the largest probability in the bar plot indicates the identification result of the proposed system. The correct digit can be successfully identified by applying the presented method. Therefore, the developed classification scheme is indeed a potential tool for the recognition of handwritten digit subjects to orientation uncertainties and can be further integrated into existing low-cost consumer electronic devices.

5. Comparison Study

To verify how much better the proposed method is than the traditional logistic regression, a logistic regression model without image straightening is used to recognize the rotated images in this study. The testing accuracy of the traditional logistic regression is only 55%. The result shows that the proposed method, the testing accuracy of which is 73.5%, does improve the performance of the rotated digit recognition.

To further validate the effectiveness of the proposed method, different methods are applied to conduct a classification comparison. The associated statistical result is summarized in

Table 2. Accuracy comparison between different methods.

Model	Training Time (Sec)	Validation Acc (%)	Testing Acc (%)	Model Size (kB)
Fine Tree	78.98	71.8	41.5	156
Median Tree	73.60	61.2	34.7	115
Coarse Tree	58.84	36.0	21.8	107
Linear Discriminant	401.89	81.8	49.8	1000
Quadratic Discriminant	Failed
Efficient Logistic Regression	631.88	83.9	59.8	6000
Efficient Linear Support Vector Machine	432.25	85.9	56.8	10,000
Gaussian Naïve Bayes	Failed
Linear Support Vector Machine	7890.60	90.0	59.1	6000
Quadratic Support Vector Machine	9528.80	94.3	66.2	220,000
Cubic Support Vector Machine	12,029.00	95.0	65.1	218,000
Fine Gaussian Support Vector Machine	94,873.00	56.3	10.4	3,000,000
Median Gaussian Support Vector Machine	25,055.00	91.7	37.7	493,000
Coarse Gaussian Support Vector Machine	27,872.00	88.7	56.9	543,000
Fine K Nearest Neighbor	7052.60	90.9	61.2	360,000
Medium K Nearest Neighbor	21,222.00	91.2	62.3	360,000
Coarse K Nearest Neighbor	25,057.00	86.6	59.3	360,000
Cosine K Nearest Neighbor	28,134.00	90.8	57.5	360,000
Cubic K Nearest Neighbor	68,775.00	87.6	43.3	360,000
Weighted K Nearest Neighbor	36,165.00	91.5	62.5	360,000
Booster Trees	28,201.00	73.8	45.6	3000
Bagged Trees	47,474.00	91.9	64.2	60,000
Subspace Discriminant	32,226.00	32.3	51.9	75,000
Subspace K Nearest Neighbor	98,531.00	95.2	80.7	5,000,000
RUSBooster Trees	31,726.00	65.3	35.4	3000
Narrow Neural Network	38,601.00	85.8	45.5	181
Medium Neural Network	34,970.00	89.6	49.5	274
Wide Neural Network	36,564.00	94.2	66.7	740
Bilayered Neural Network	43,035.00	85.8	46.1	183
Trilayered Neural Network	43,716.00	86.1	48.2	184
Support Vector Machine Kernel	107,320.00	95.3	75.6	1,600,000
Logistic Regression Kernel	88,626.00	92.6	68.7	1,600,000
PCA-Based Logistic Regression	15.20	87.2	73.5	4628

. Experiments shows that the PCA-based logistic regression has one of the highest testing accuracies for recognition while using the shortest training time. All of the models are trained using original MNIST training data and tested using rotated MNIST testing data.

In Table 2, most of the models cannot recognize the rotated digits as effectively as the proposed method. Except for the proposed model, there are only two models that have accuracies higher than 70%, the training times of which are 6000 times longer than that of the proposed method. The results show that the proposed method has a better recognition accuracy, as it uses much fewer parameters and spends less time on training, meaning that it shows great potential for application in systems with limited storage capacities.

The recognition results of the different rotated handwritten digits are shown in Figure 19. The result shows that the model can successfully recognize the rotated digit for a real-time application, as the model classifies the digit via an extremely simple but efficient calculation.

As the low-cost recognition system can recognize a rotated digit that has never been trained during the training process, users of the system are able to write the digits without a limited angle restriction.

6. Conclusions

Nowadays, the handwritten digit recognition system is widely used in several kinds of 3C consumer electronics, including mobile phones, laptops, and tablet computers. However, when a handwritten number is rotated due to some human factors, the recognition system might fail to identify the digit. Therefore, this article proposes a PCA-based logistic regression to solve the digit rotation problem. The proposed method has four main contributions. Firstly, the PCA is proven to be able to perform image straightening. This ability can help to improve the practicability of the trained model, as the model can be used when the image is rotated. Secondly, the PCA helps to reduce the dimension of the whole dataset without affecting the accuracy. This ability leads to a lower training time and higher detection frequency, which can improve the performance of logistic regression and real-time detection tasks. Thirdly, an improved PCA for image orientation correction was presented, which makes the model’s performance better than that of the model using traditional PCA. The improved PCA uses the rotated matrix to decide whether a digit has an orientation problem after pre-processing and revising the rotation matrix. The digits can be more appropriately straightened through this improvement. The first three contributions improve the performance of the model trained via the logistic regression and the process of training it. The proposed method leads to a more robust model that has the highest accuracy and shortest training time among the three compared models.

The main limitation of the proposed method is that the alignment process in the proposed method cannot address the rotation angle of the digits over 60 degrees. However, it is rare to find a digit that is rotated over 60 degrees written by humans. Furthermore, the proposed recognition has a high accuracy with the artificial rotated digit, as seen in Figure 10 in this paper. Therefore, it is reasonable to have a rotation angle threshold for the proposed method.

Although there are some issues waiting to be solved in the future, such as improving the accuracy, this study shows that PCA is indeed a potential choice to perform data pre-processing to improve logistic regression, especially when the handwritten digits are subject to rotations. Finally, owing to the low computational complexity, the proposed method can be integrated into the current low-cost portable devices, and real-time recognition can be achieved. The attached demonstration (Supplementary Video S1) verified the effectiveness of the developed system.