Effective Digital Technology Enabling Automatic Recognition of Special-Type Marking of Expiry Dates

Abstract

In this study, we present a machine-learning-based approach that focuses on the automatic retrieval of engraved expiry dates. We leverage generative adversarial networks by augmenting the dataset to enhance the classifier performance and propose a suitable convolutional neural network (CNN) model for this dataset referred to herein as the CNN for engraved digit (CNN-ED) model. Our evaluation encompasses a diverse range of supervised classifiers, including classic and deep learning models. Our proposed CNN-ED model remarkably achieves an exceptional accuracy, reaching a 99.88% peak with perfect precision for all digits. Our new model outperforms other CNN-based models in accuracy and precision. This work offers valuable insights into engraved digit recognition and provides potential implications for designing more accurate and efficient recognition models in various applications.

1. Introduction

Medicine advancement has made healthcare an essential aspect of our daily life. Whether it is for treating ailments or managing minor health conditions, drugs are readily available for purchase, even without a hospital prescription. This easy accessibility aims to enhance the well-being of individuals and promote better health among them. With the convenience of purchasing drugs and food items from diverse sources, adhering to their expiration dates has become essential in maintaining optimal health and protecting individuals from potential health risks. As digital technology advancements continue to revolutionize various aspects of life, automating the expiration date recognition holds a significant importance for individuals, particularly for those with visual impairments or difficulty understanding date notations. This automation not only aids people, but also contributes to a sustainable environment. Engraved digits characterized by unique shadows and edges are commonly used to represent expiration dates (Figure 1). However, recognizing these dates can be challenging due to the various marking methods employed. The human eyes often struggle to accurately read these markings. Digital technology offers a promising solution to assist individuals in automatically recognizing engraved-type expiration dates. To achieve this, we collected a dataset containing expiration date images from medicines, consumables, cosmetic products, and tube-type ointments. This dataset showcases the distinctive properties of digits and the peculiar shadow patterns and edges resulting from the engraved shape in the images. We primarily aim to develop an automatic recognition system for digits in engraved-type expiration dates. In this study, our machine-learning-based approach harnesses a generative adversarial network (GAN) model to augment the dataset with synthetic engraved digit images. The augmented dataset is then used to train various supervised classifiers, including our proposed convolutional neural network for engraved digit (CNN-ED) models.

Unlike handwritten digit recognition, which is a well-established problem in machine learning and uses state-of-the-art models for classification, engraved digit recognition in images presents its own set of challenges, including low contrast, blurry edges, and unusual details that can complicate the achievement of accurate results. These engraved digit features are not well suited for the supervised recognition models developed for more common dataset types such as MNIST [1] and EMNIST [2]. Several algorithms can efficiently recognize handwritten digits and achieve a state-of-the-art performance on benchmark datasets [3,4,5,6,7,8,9]. However, a number of these algorithms may perform poorly on engraved digit image recognition for several reasons. One reason is that many recognition models perform well on datasets they have been specifically trained on and on a specific recognition task, but have difficulty generalizing to other datasets or tasks [10] due to factors such as overfitting or a lack of training data diversity. Another known reason is that some of these models require a huge amount of data to train, which might not be easily available [10]. Developing or identifying the best-performing known models for an unusual dataset type (e.g., engraved digit image) might be necessary when considering the unique data features, data insufficiency, and different types of recognition problems, among others. Accordingly, we develop herein our own CNN-ED model and evaluate it with several CNN-based models for our image recognition task considering their ability to learn complex image representation through multiple layers [11,12,13,14].

Our proposed CNN-ED model achieves an outstanding 99.88% accuracy on the test data, surpassing the 99.77% accuracy of an adapted CNN architecture and outperforming the 99.56% benchmark accuracy achieved by classifiers trained on the MNIST dataset [4,5,15,16,17,18]. In addition to our work’s application in medicine and healthcare, engraved expiry date recognition holds a potential for diverse cross-disciplinary applications. Accurate pattern identification, especially when coupled with secure data handling, is a common thread that binds various domains. This connection emerges from the fundamental need to ensure data accuracy, reliability, and trustworthiness, regardless of whether it is engraved-digit or sensor-generated information. For instance, our work can be extended to scenarios involving the Internet of Things (IoT) data, as discussed in [19], wherein the challenges of authentication and secure data sharing in an IoT context were addressed. While the nature of the IoT data differs from that of engraved digits, the underlying concern as regards accurate data recognition remains consistent. Just as we have developed techniques to ensure the fidelity of engraved digit recognition, the principles of secure data handling and authentication hold relevance in the context of the IoT data interactions. By maintaining focus on accurate data recognition, our work provides insights that can extend to IoT applications, thereby enhancing the integrity and reliability of data exchanges in interconnected systems. Similarly, the application of our work is reflected in the context of machine learning-empowered communication networks, as discussed in [20]. Ref. [20] underlined the importance of leveraging machine learning technologies to amplify the network communication capabilities. Communication networks can proactively manage products with engraved expiry dates, furthering the vision of data-driven decision making, optimal resource utilization, and enhanced network performance by applying an efficient recognition model. As emphasized, the seamless integration of machine learning techniques within networks paves the way for smarter and more responsive systems. Our work effectively bridges the gap between these principles and the practical application of engraved date recognition, underscoring its far-reaching implications in shaping intelligent, interconnected systems. This study presents the three following key justifications for adopting a precise CNN recognition model, thereby contributing to pattern recognition and providing valuable insights into the challenges of recognizing engraved digits in real-world applications:

• Augmenting the dataset with GAN improves the classifier accuracy, enhances the performance, and instills trust in the model’s results, which are crucial for ensuring a reliable recognition of engraved digits, even in the presence of physical deformations, aging, rust, shadows, and blurriness.
• The optimal performance in supervised classifier selection enhances the model’s reliability and applicability in diverse engraved digit recognition.
• The CNN-ED model’s efficiency and adaptability make it possible to accurately identify engraved digits in diverse datasets, ensuring reliable recognition in practical applications.

The rest of the paper is organized as follows: Section 2 describes some related works on recent recognition models for similar tasks and their limitations on our datasets; Section 3 presents our CNN-based method for engraved digit image recognition, called the CNN-ED model; Section 4 discusses the results of our approach compared to those of well-known classic classifiers, such as decision tree (DT), K-nearest neighbor (KNN), support vector machine (SVM), Gaussian Naive Bayes (GNB), stochastic gradient descent (SGD), random forest (RF), and some CNN-based models in recent studies; and Section 5 concludes the paper and mentions the future work.

2. Related Works

Recent years saw a significant increase in the amount of research conducted in the image recognition field, leading to the development of new algorithms and techniques and the application of image recognition in new domains. These algorithms underwent a significant advancement over time and became more sophisticated. Researchers often utilize ensemble methods that involve combining multiple classifiers to improve the classifier performance on well-known datasets in terms of accuracy, running time, and computational complexity [12,21,22,23,24,25,26].

Ahmed et al. [27] proposed a deep learning technique for recognizing Arabic handwritten digits. Their method used a CNN with LeNet-5 trained and tested on the MADBase database of Arabic handwritten digits comprising 60,000 training and 10,000 testing images. In [28], a new model based on deep neural networks was introduced to tackle the challenges of recognizing offline handwritten Arabic text, including isolated digits, characters, and words. A supervised CNN model that extracts optimal features in a contextual manner and includes batch normalization and dropout regularization parameters was suggested. The model underwent extensive evaluation and demonstrated excellent classification accuracy compared with the traditional offline Arabic handwriting recognition methods across six benchmark databases, namely, MADBase (digits), CMATERDB (digits), HACDB (characters), SUST-ALT (digits), SUST-ALT (characters), and SUST-ALT (names).

Savita [5] introduced a hybrid model comprising a CNN and a support SVM to recognize handwritten digits from the MNIST dataset. The model achieved superior performance by combining the essential features of both classifiers. In this model, the CNN functioned as an automatic feature extractor, while the SVM acted as a binary classifier. The proposed algorithm was trained and tested using the MNIST dataset of handwritten digits. In [23], Yoshihiro et al. proposed a new method for handwritten digit recognition by combining the pre-trained CNN and the SVM. The pre-trained CNN, that is, AlexNet, was used as a pattern feature extractor, while the SVM was employed as a trainable classifier. The 60,000 training samples on the MNIST database were trained by the SVM. The feature vectors of the character patterns were passed to the SVM from AlexNet. Without augmentation, the experimental result of the test error rate was 1.03% on the 10,000 test data. The MNIST database showed that the proposed method was effective in handwritten digit recognition. In [13], Tanuja et al. utilized the CNN, KNN, and SVM to recognize isolated handwritten digits. By implementing and training these models on the same dataset and comparing the results, they discovered that the CNN is the most optimal machine learning technique for classifying handwritten digits, exhibiting a remarkable 99.59% accuracy.

In [29], Vijay Kumar et al. proposed an approach for recognizing isolated digits. They utilized geometrical and host pot features to extract features from digit images. The proposed method was tested on four databases, namely, the MNIST database with 60,000 training and 10,000 testing samples of numerals, the CENPARMI database with 4000 and 2000 images in the training and testing sets, respectively, the CEDAR database consisting of 18,468 and 2711 training and testing sets of digits, respectively, and the USPS database with 7291 training and 2007 testing sets. The numerals in these databases were classified using either a KNN or an SVM classifier. In [30], Rohan et al. aimed for recognition works related to handwritten digit recognition. They applied the SVM for recognition and classification and the convex hull algorithm for feature extraction.

In [31], Bhunia et al. presented a new method for recognizing word-level Indic script. Their approach employed character-level data in the input stage and a multimodal neural network that can accept both offline and online data as input to explore information from both modalities for text/script recognition. Representative of the real scenario of the data fed into the network, the proposed multimodal fusion scheme combined offline and online data. The system was evaluated on English and six Indian scripts to verify its effectiveness. In [32], Obaidullah et al. conducted a comprehensive survey on the development of an OCR system that recognizes Indic script in multiscript document images. They discussed various preprocessing techniques, feature extraction methods, and classifiers used in script recognition. In [33], Ahmed et al. utilized a one-dimensional bidirectional long short-term memory approach for recognizing handwritten Urdu letters. They tested their approach by creating a medium-sized database of handwritten Urdu letters collected from 500 individuals. In [6,7,34,35,36], CNN models were used to improve the accuracy of handwritten digit recognition on the MNIST dataset. Meanwhile, in [34], a CNN was used with the MNIST dataset and the Deeplearning4J (DL4J) framework to achieve 99.21% accuracy and a reduced computational time for training and testing. In [35], the CNN architecture was implemented for the MNIST dataset, acquiring 98.85% accuracy in a short training period. Meanwhile, variations in the accuracies for different numbers of hidden layers in CNN models were observed in [36]. The highest accuracy on the MNIST dataset was 99.21%, while the lowest accuracy was 97.07%. In [7], a CNN was employed for handwritten digit recognition, which yielded 96.1 and 95.7% accuracies for the training and testing sets, respectively, after 500 training epochs. In [6], a CNN was also used to classify grayscale images, achieving 98% accuracy after the MNIST dataset training.

The literature review indicated that several studies were conducted on image recognition, particularly on handwritten image recognition. A large set of classification techniques was applied to this problem, including linear classifiers (e.g., KNN, DT, RF, GNB, SGD, SVMs, and CNN-based algorithms). However, most of these classifiers had several limitations that warrant model evaluation on our datasets to identify the most suitable model for the engraved digit dataset. A number of these classic classifiers are still being used in research, basically because of their simplicity, easy implementation, short time with low computational cost, good performance with a smaller dataset, capability of being effective in certain situations (particularly when the underlying data distribution is understood well), relatively small number of features, and role as a useful baseline for comparison when evaluating more advanced techniques.

The main limitations of these classifiers are as follows:

• Low accuracy: Existing classifiers, such as DT, SGD, and RF, have relatively low accuracies when classifying engraved digits because they cannot capture the fine details and variations present in the engraved digits, resulting in misclassification errors.
• Lack of robustness and insensitivity to fine details: Except for CNN-based classifiers, others are not robust to the variations in the orientation, size, and lighting conditions of the engraved digits and are insensitive to fine details. In other words, they perform poorly when the engraved digits are not captured in ideal conditions.

The abovementioned limitations of the current classic classifiers highlight the need for a new recognition model or an evaluation and selection of the best recent model that can overcome these challenges and enhance the classification accuracy for engraved digits.

3. Methodology

We conducted herein a comprehensive dataset evaluation using well-known and state-of-the-art supervised algorithms, including DT, SVM, SDG, GNB, KNN, RF, CNN, and hybrid classifiers. Including a mix of classic and deep learning models in our work allowed us to capture a wide spectrum of algorithmic approaches. In machine learning, classification methods can basically be broadly categorized into classic and deep learning classifiers. Classic classifiers encompass a range of algorithms widely used in various domains before the advent of deep learning. These methods are often based on mathematical and statistical principles relying on well-defined features extracted from the data. Examples of classic classifiers include SVMs, DTs, KNN, and Naive Bayes. These classifiers excel in scenarios with relatively structured and well-defined data, which show an explicit relationship between features and classes. The algorithms are known for their ability to handle various data types and offer insights into feature importance, which aids in understanding the underlying patterns contributing to the algorithm performance. Deep learning classifiers represent a modern approach that has gained immense popularity due to their ability to automatically and directly learn complex patterns and hierarchical representations from the data. These classifiers employ artificial multiple-layer neural networks, aptly called deep neural networks. The architecture enables the model to discover complex features and relationships that might not be apparent through manual feature engineering. CNNs and recurrent neural networks (RNNs) are prominent examples of deep learning classifiers, with CNNs excelling in image-related tasks and RNNs being well-suited for sequential data.

The DT is a non-parametric algorithm that creates a tree-like model of decisions and their possible consequences that can be useful for datasets with nonlinear relationships between features [37,38]. The SVM is a popular algorithm for separating data into different classes by identifying a hyperplane that maximally separates the classes [39]. The SDG [40] is an algorithm used for large-scale machine learning tasks. The GNB is a Naive Bayes algorithm type that assumes independence among features [41,42]. Meanwhile, the RF is an ensemble learning method that uses multiple DTs to improve the classification accuracy [43,44]. The KNN is used to classify or predict the value of a new data point based on the class or values of its closest neighbors in the training data [45,46]. The cutting-edge hybrid classifiers, CNN–SVM [5] and CNN–RF [47], aim to combine the strengths of two classifier techniques to achieve improved accuracy, robustness, or efficiency in solving a particular classification problem. In these classifiers, the CNN performs feature extraction through convolutional and pooling layers, while the SVM/RF classifies features into different classes using a decision boundary or a hyperplane in the feature space.

In this study, we implemented traditional classic algorithms using Scikit-Learn and prepared our images for these algorithms by performing preprocessing steps, including standardization, reshaping, and conversion to grayscale, which resulted in 128 × 128 × 1 input images. This format was more effective for engraved digit recognition. We optimized time and efficiency by making minimal adjustments to the model settings, ensuring a streamlined approach to our experiments.

Figure 2 shows the engraved digit recognition workflow. The “Data Preprocessing” stage prepares raw engraved digit images by normalizing the data for an accurate classification. The “Data Augmentation” phase involves applying GAN techniques to enhance the dataset, thereby increasing its diversity and improving the performance of the classification models. The “Classification Models” phase focuses on evaluating different machine learning models tailored for engraved digit recognition. This includes the training and fine-tuning of the classifier algorithms. Model validation ensures the performance and the generalizability of the trained classifiers by testing them on a separate dataset. Model evaluation employs various metrics, such as F1 score, accuracy, recall, and precision, to assess the classifiers’ effectiveness on the dataset, thereby providing a comprehensive performance analysis. The “Recognition” phase entails applying trained classification models to accurately recognize and identify engraved digits. Lastly, the “Information Retrieval System” is a system integrating the classification and recognition components and enables the retrieval of relevant information based on the recognized engraved digits, facilitating applications in several domains.

3.1. Engraved Digit Datasets

Engraved digits refer to numerical characters etched or inscribed onto surfaces, such as metal, plastic, and wood. These digits are typically machine generated and commonly found in various industrial, commercial, and manufacturing settings. However, the legibility and the visibility of engraved digits can significantly be compromised due to various factors, including physical deformation, aging, rust, shadows, and blurriness. This presents a significant challenge in accurately identifying and recognizing digits, especially in real-world scenarios, where engraved surfaces may undergo wear and deterioration. The engraved digit dataset has varied image quality, lighting conditions, and distortion caused by the engraving process (Figure 1). These variations in the appearance and quality of the engraved digits pose difficulties to traditional recognition methods and require the development of specialized approaches to effectively address the abovementioned challenges. However, the variability also makes the dataset suitable for evaluating the efficiency of the proposed classifier algorithm.

The engraved digit dataset used herein comprises images from digits engraved on ointments, creams, and tubes, each preprocessed with a 128 * 128 resolution. Collecting a large volume of this dataset was a challenging task because finding a large number of high-quality and sufficiently varied engraved digits to train an efficient recognition model was difficult. We overcame this challenge by augmenting the datasets with images generated by a Wasserstein GAN with a gradient penalty (WGAN-GP) and a Wasserstein divergence for the GAN (WGAN-DIV) [48]. In [48], we conducted a comprehensive evaluation of various state-of-the-art GAN models on engraved digit datasets, including MMGAN, NSGAN, LSGAN, ACGAN, DCGAN, WGAN, WGAN-GP, WGAN-DIV, DRAGAN, BEGAN, EBGAN, and VAE. We assessed the performance of each GAN model by combining a visual inspection of the generated samples and a calculation of the Fréchet Inception Distance values.

Our findings indicated that WGAN-GP and WGAN-DIV had remarkable levels of stability and effectiveness in the data augmentation context. We selected and integrated WGAN-GP and WGAN-DIV to further enhance the augmented dataset quality. The GAN-based data augmentation enhanced the classifier performance by enriching the dataset with diverse and realistic samples, mitigating overfitting, improving feature representation, reducing bias, and improving the classifier’s ability to handle noise and outliers. These advantages collectively contributed to a more accurate and reliable classification. Among several other GAN models, the WGAN-GP and WGAN-DIV models demonstrated exceptional performances during execution on the engraved digit dataset.

Figure 3 depicts a sample of the engraved digit dataset. The engraved digits exhibited a wide variation range and were quite different from the handwritten MNIST dataset. In most recognition cases, this high image variability presents a challenge for recognition models, making the dataset a suitable benchmark for evaluating the proposed algorithm’s efficiency. Each image was labeled with the corresponding digit from 0 to 9. The dataset was divided into training and testing sets in a 70:30 ratio. In the initial experiment set, using the original dataset of roughly 30 images per digit class (Original dataset), all classifiers showed low performances compared to those in the second and third experiment sets, in which we used larger and more varying datasets. The second experiment set comprising datasets with approximately 150 images per digit class, including the data generated using the WGAN-GP (Augmented1), resulted in a significant improvement in the classifier performance. In the third experiment set with approximately 250 images per digit class, all classifiers showed even better performance enhancements by combining the original dataset with the images generated by both WGAN-GP and WGAN-DIV (Augmented2).

3.2. CNN-Based Engraved-Digit Recognition Model

CNNs demonstrate an excellent performance in image identification applications, including handwritten digit recognition [5,6,7,11,12,13,14,18,21,23,27,28,34,35,36]. A simple convolutional layer achieves over 99% validation accuracy on the MNIST dataset [3,4,5,6,7,8,9].

We developed herein our own CNN-ED model for recognizing engraved expiry date digits and evaluated this model against several known CNN models, all of which were implemented using their respective hyperparameters on our augmented datasets. These CNN models were straightforward and differed not only in the number of filters, but also in the network architecture, including the presence of multiple convolutional layers. We varied these architectural parameters to comprehensively analyze the performance of different CNN configurations and design a model that achieves an optimal performance for our specific task. Figure 4a illustrates our CNN-ED model, which is compared to a hybrid CNN classifier model (b) and Figure 5 shows the model summary.

The CNN usually performs better with large datasets [23,47]. However, collecting a large and diverse dataset for engraved digit images is difficult and time consuming. First, we adopted GANs to produce additional, but synthetic engraved digit images [48]. We then employed the CNN-ED model to perform the recognition task on our dataset. Using GANs as a data augmentation technique to artificially increase the training dataset size, we effectively increased the training dataset size, which consequently increased the chance of the recognition model to be more efficient. Figure 6 and Figure 7 depict the samples of the generated images produced by WGAN-GP and WGAN-DIV at epochs of 1000, 5000, 10,000, and beyond.

For our CNN-ED model in Figure 4a, we employed the TensorFlow Library to create a high-performing neural network architecture for digit recognition, achieving an impressive 99.88% accuracy. The model was designed with a CNN layer (Conv2D) comprising 32 filters with a kernel size of (3,3). The activation function used in this layer was the rectified linear unit (ReLU) known for its ability to effectively handle the non-linearities in the data. Following the first convolutional layer, a MaxPooling2D layer with a pool size of (2,2) was applied to reduce the spatial dimensions of the feature maps, consequently enhancing the model’s ability to detect essential patterns while reducing the computational complexity. The second convolutional layer (Conv2D) consisted of 64 filters that further extracted intricate features from the feature maps produced by the first layer. Similar to the previous layer, the ReLU activation function was also used. Another MaxPooling2D layer with a pool size of (2,2) was introduced to downsample the feature maps, thereby further improving the model’s pattern recognition performance. The convolutional layer output was then flattened into a 1D vector, preparing the data for fully connected layers. The first dense layer comprised 2048 neurons and employed the ReLU activation function and the “he_uniform” initializer to ensure a stable and efficient weight initialization. This layer was designed to capture higher-level representations of the input data, contributing to the model’s discriminative power. The next dense layer consists of 784 neurons with the ReLU activation function. This layer further refined the learned features, consequently enhancing the model’s ability to differentiate between different classes. Finally, the output layer comprised 10 neurons equal to the number of classes (digits 0–9) in the recognition task. The activation function used here was the softmax function that generates a probability distribution over classes, allowing the model to predict the probability of an input belonging to each class. Softmax is usually employed as the activation function for the CNN output layer to classify the input image into one of the predefined classes. This property was particularly useful in the context of our work because our goal herein was to classify images into one of 10 possible classes (i.e., 0–9). The softmax function is mathematically expressed as follows for a given input vector x:

$$\begin{array}{c}\hfill {\displaystyle \mathrm{P}(\mathrm{y}=\mathrm{i}/\mathrm{x})=\frac{exp\left(x^T{w}_i\right)}{{\sum }_{k=1}^{K}exp\left(x^T{w}_k\right)}}\end{array}$$

(1)

where x is the input, $w_i$ is the weight vector for the ith class, and k is the total number of classes [47]. Our model was compiled using the SGD optimizer with a 0.01 learning rate, giving us the best result.

As regards the CNN–SVM and CNN–RF hybrid classifiers (Figure 4b), the CNN was responsible for the feature extraction task. It extracted relevant features from the input data. The SVM and RF classifiers then utilized these extracted features as the input for their respective classification tasks. The CNN acted as a feature extractor capturing complex patterns and representations from the input data. The SVM and the RF classifiers leveraged these extracted features to make the final classification decision. The combined approach of using the strengths of both the CNNs and the traditional classifiers led to an improved performance and accuracy in certain tasks because the CNN–SVM or CNN–RF hybrid classifiers can leverage the capabilities of both the CNNs and the SVM/RF classifiers for effective feature extraction and classification. To understand the impact of minor modifications to certain model hyperparameters, we focused on altering the type of activation function, learning rate, optimizer (Adam and SGD), batch size, and number of epochs while keeping the number of convolutional and dense layers and pool size constant.

4. Evaluation and Analysis

During our experimentation of the supervised classic classifiers (i.e., KNN, SVM, GNB, SDG, RF, and DT) for the engraved digit dataset, we evaluated the classifiers across digits 0–9 and recorded the average metric values (e.g., precision, accuracy, and F1 score). The F1 score is the harmonic mean of precision and recall that provides a single number to compare the overall performances of different classifiers. It balances both precision and recall and is often used when both are important. Precision measures how often a classifier correctly identifies positive samples. A high precision indicates a low false positive rate. The recall value is a performance metric measuring the percentage of positive instances correctly identified by the model [8,17,49,50]. Recall measures how often a classifier correctly identifies positive samples out of all actual positive samples. A high recall indicates a low false negative rate. The SVM, KNN, and DT models showed good performances across all classes. We also discovered that the performance of almost all classifiers improved as the dataset size increased, which was consistent across all three datasets used in our experiments (

Table 1. Classic classifier accuracies across the three datasets (%).

Classifier	Original Dataset	Augmented1	Augmented2
		(Original + WGAN-GP)	Augmented1 + WGAN-DIV
KNN	26.1	79.2	91.6
SVM	22.0	72.3	90.8
GNB	13.9	25.1	21.9
SDG	18.3	31.9	34.0
RF	25.0	23.2	24.5
DT	20.8	55.7	78.9

and

Table 2. Average F1 score for the traditional classifiers across datasets (%).

Classifier	F1 Score-1	F1 Score-2	F1 Score-3
KNN	24.0	80.0	92.0
SVM	12.0	73.0	91.0
GNB	12.0	23.0	20.0
SDG	10.0	27.0	29.0
RF	13.0	14.0	20.0
DT	21.0	58.0	79.0

).

In the initial experiment set that used the original dataset of roughly 30 images per digit class (Original dataset), all classifiers showed low performances compared to those in the second and third experiment sets, in which we used larger and more varying datasets. The second experiment set comprising datasets with approximately 150 images per digit class, including the data generated using WGAN-GP (Augmented1), resulted in a significant improvement in the classifier performance. In the third experiment set with approximately 250 images per digit class, all classifiers showed even better performance enhancements by combining the original dataset with the images generated by both WGAN-GP and WGAN-DIV (Augmented2).

We evaluated the experimental setup using the standard metrics commonly used for classifiers: accuracy, precision, recall, and F1 score [8,13,49]. Table 1 shows that the KNN algorithm achieved 26% accuracy in the first experiment set and 79.2 and 91.6% accuracies in the second and third experiment sets, respectively. Similarly, the SVM model achieved 22, 72.3, and 90.8% accuracies in the first, second, and third experiment sets, respectively. The classification report summarizes these measures for each set of our dataset. The SVM was found as the best-performing classifier (Table 1), exhibiting approximately 19% improvement from the second to the third datasets, followed by the KNN with a 12% accuracy increment. The low-performing classifiers were SDG, GNB, and RF, which showed marginal progress in their performances.

Table 2 presents the F1 scores of the Original dataset, Augmented1, and Augmented2 as F1 score-1, F1 score-2, and F1 score-3, respectively. Our results suggest that the performances of the supervised classifiers in accurately classifying the engraved digit dataset were significantly improved by increasing the dataset size and diversity. This result was in alignment with the result of the prior studies highlighting the significance of having a large and diverse dataset for enhancing the efficacy of machine learning models [23].

Similar to that in [24], our assessment of the classic classifiers involved an examination of their performances for each dataset class. Figure 8, Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13 illustrate the metric values of the selected digits in classes 0, 2, 6, and 8 for each dataset (i.e., Original dataset (Org_data), Augmented1 (Aug_1), and Augmented2 (Aug_2)). Classes 0 and 2 showed significant performance improvements across all datasets. Moreover, all classifiers recorded high performance scores for the Digit 2 class. The RF classifier displayed a subpar performance in almost all classes, except in Class 2. As shown in the images, Figure 8 classifier demonstrates its performance in recognizing digit 0 on the Aug_1 dataset. However, the SVM classifier (Figure 13) outperforms others for digit 0 on the Aug_2 and Aug_3 datasets. The DT classifier (Figure 8) secures the second-best performance for digit 2 across all augmented datasets, and ranks third in recognizing digits 6 and 8 on all three augmented datasets. The SVM classifier (Figure 13) excels in recognizing digit 0 on both the Aug_2 and Aug_3 datasets, securing second place for digit 0 on the Aug_1 dataset. Impressively, the SVM classifier claims the top position for digit 2 across all augmented datasets, while securing second place for digits 6 and 8 on these datasets. Meanwhile, the KNN classifier (Figure 12) secures the second-best performance for digit 0 across all augmented datasets and ranks third for digits 2, 6, and 8 on all three augmented datasets. RF classifier (Figure 9) achieves the third-best performance for digit 0 on all three augmented datasets, and secures the second-best results for digit 2 across all three augmented datasets. The RF classifier ranks third for digits 6 and 8 on the Aug_1 and Aug_2 datasets, and second for the Aug_3 dataset. While the performance of the RF classifier improves marginally with augmented data, it still lags behind the substantial improvements observed in the DT, KNN, and SVM classifiers due to its greater susceptibility to noise and outliers. Among the classifiers used, SGD (Figure 10), GNB (Figure 11), and RF (Figure 9) had the worst performance. They showed minimal improvement in their respective performance metrics. In our comprehensive experimentation involving all the digits ranging from 0 to 9, two classifiers have consistently demonstrated exceptional performance; the SVM classifier and the KNN have consistently stood out as the top-performing models.

In

Table 3. CNN-based classifier accuracies across the three datasets (%).

Classifier	Original Dataset	Augmented1	Augmented2
		(Original + WGAN-GP)	(Augmented1 + WGAN-DIV)
CNN-SVM [5]	65	95	97
CNN-RF [47]	48	89	94
CNN [6]	17.58	98.11	99.72
CNN [7]	18.68	96.52	99.67
CNN-ED	26.37	98.7	99.88
CNN [35]	50.55	98.55	99.77
CNN [36]	37.36	97.54	99.27
CNN [34]	45.05	98.26	99.06

, the CNN-ED accuracy was compared with that of other CNN-based models. The CNN-based models demonstrated remarkable performances, with the lowest accuracy (for the CNN–SVM) beginning at 89% from the first augmented dataset (Augmented_1). Following the performance of our CNN model, CNN [35] also achieved an impressive 99.77% accuracy in the third experiment.

Table 4. CNN-based classifier Precision, Recall, and F1 score across the three datasets (%).

Classifier	Precn, Recall, F1 Score	Precn, Recall, F1 Score	Precn, Recall, F1 Score
	(Original Dataset)	(Augmented1)	(Augmented2)
CNN-SVM [5]	73, 73, 72	95, 95, 97	99, 99, 98
CNN-RF [47]	48, 47, 40	93, 94, 94	96, 97, 97
CNN [6]	12, 21, 14	98, 99, 99	99, 100, 100
CNN [7]	79, 70, 68	97, 96, 96	99, 99, 99
CNN-ED	40, 43, 35	99, 99, 99	100, 100, 100
CNN [35]	67, 61, 53	98, 98, 99	100, 100, 100
CNN [36]	43, 48, 42	98, 99, 99	100, 100, 100
CNN [34]	69, 61, 59	97, 98, 99	100, 100, 100

depicts the precision, recall, and F1 score for the CNN-based models.

The CNN-based models in Figure 4 shared a similar architecture, where the first and second convolution layers had 32 and 64 feature maps, respectively. However, notable differences were observed in the kernel input of the CNN-based models, MaxPooling sliding strides, and activation functions used. Note that while the CNN models shared some layers, the model architecture, particularly the position of each layer in the design and the hyperparameter values, can significantly influence the model’s overall performance. In the CNN–SVM [5] and CNN–RF [47], hybrid classifiers (SVM and RF) were used for the binary classification instead of softmax and sigmoid. The CNNs were used for feature extraction from the input data. The extracted features were fed into the SVM and RF algorithms for binary classification. The CNN models aimed to learn the hierarchical representations of the input data, while the SVM and RF classifiers aimed to classify the extracted features into binary classes. The two optimizers used for the optimization during the CNN training were Adam and SGD.

An analysis of the experimental results revealed accuracy values, which served as a strong indication of the superiority of the CNN-ED model over other models in terms of the precision, recall, and F1-score across digits 0–9 (Table 3). This result indicated that the CNN-ED model outperformed the other CNN models in the accurate identification of engraved digits. Our observation further reveals that the CNN-ED model achieved a precision and recall of 1.00 for all digits, indicating its correct identification of all instances of each digit with no false positives. The exceptional accuracy, precision, recall, and F1 score of the CNN-ED model demonstrated its superiority over the other CNN-based models. Its outstanding performance in engraved digit recognition in medicine expiration dates makes it a promising model for various real-world applications, where precise and reliable recognition is of utmost importance.

After exhaustive experimentation using the various hyperparameters on our CNN model, we discovered that the optimal parameter configuration was nearly identical to that of other models. Despite attempts to fine-tune the parameters, no significant accuracy increase was observed, leading us to conclude that optimal hyperparameters should be used.

Based on the accuracy values obtained from the evaluation and comparison of the CNN and hybrid models on our engraved digit dataset, the CNNs in [6,7,35] were other top-performing CNN models that achieved high accuracy scores of 99.77, 99.71, and 99.67%, respectively. These models showed exceptional performances in correctly predicting the engraved digits in our dataset. Apparently, all simple CNN-based models demonstrated high accuracies in engraved digit recognition, further validating their effectiveness. However, the CNN–SVM model achieved 97% accuracy, which was comparatively lower than those of the other CNN models. Similarly, the CNN–RF model achieved 94% accuracy, depicting the lowest value among all evaluated CNN-based models. Figure 14, Figure 15 and Figure 16 illustrate the CNN classification accuracy, with the leading blue line representing the model’s training accuracy and the orange line, indicating its validation accuracy throughout the training process. The y-axis represents the accuracy values, while the x-axis corresponds to the epoch value set at 10 for our evaluation. The observation of the accuracy curves demonstrated that the model’s training accuracy, which measures its ability to fit the training data, apparently improved as it learned data pattern identification. The validation accuracy assesses the model’s ability to generalize. The graphs depict that it also concurrently increases with the training accuracy. In other words, the model successfully learned to identify data patterns without overfitting, thereby demonstrating its proficiency in pattern recognition.

While we achieved the highest accuracy level with CNN-ED, i.e., a 99.88% accuracy, we also conducted further experiments to create a deeper, but simple model by tuning the hyperparameters. We modeled the new model after the CNN-ED model, i.e., the ones in [6,35], using an architecture similar to that in Figure 4a. However, after multiple tuning rounds, the highest accuracy we achieved was only 99.6% on the third dataset after adding three convolutional layers (Conv2D) with 64 filters. Further attempts of adding more layers and tuning resulted in lower accuracy values, indicating that a deeper model did not necessarily lead to an improved performance. Despite other attempts of increasing the model depth, the best accuracy of 99.6% on the third dataset was only achieved when specific layers were added. We attempted to improve the model by adopting transfer learning with VGG-16 following the application of the second augmentation technique. The outcome was regrettably unsatisfactory. Our accuracy measure never surpassed 56%. This underwhelming performance was attributed to the pre-trained VGG network being not optimally tailored to our specific task or our data size and complexity being a challenge. The features learned by the VGG network may not have been pertinent to our task, thereby resulting in a poor performance.

In addition to assessing the performance of various classifiers for our task, we also considered the models’ computational requirements. Notably, the CNN-based models had considerably longer training times compared to the classic KNN, RF, GNB, and SVM classifiers. On average, these CNN-based models achieved an approximate inference speed of 16 frames per second (fps). The other classic models exhibited slower inferencing speeds that ranged from 24 to 29 fps. The CNN-based models specifically took approximately 1 h and 25 min of processing time, including the time taken for data preprocessing, feature extraction, and training. By contrast, the other models took 48 to 60 min. Note that the classic classifiers did not have a distinct training phase because they performed the learning step during the classification process. Therefore, the training time for these classifiers was often conflated with the testing time, which is defined as the time taken to classify new instances based on the learned model. In contrast, the CNN-based model required a separate training phase, in which the neural network weights were updated based on the training data before they can be used for classification. The longer processing time taken by the CNNs was attributed to the CNNs being generally more complex and computationally intensive because they involved multiple layers with a number of learnable parameters [18,51,52,53] compared to the classic models that used simpler algorithms and had fewer parameters. Our evaluation suggests that the SVM and the KNN may be suitable in practical applications with critical processing times. Additionally, some other low-performing classifiers might show better results with a different dataset. For instance, in [34], the DT classifier outperformed the MLP and the KNN in recognizing Sindhi handwritten numerals. Similarly, in [54], the Naive Bayes classifier displayed a superior accuracy over the KNN. Lastly, in [31,51], the RF outperformed the SVM and KNN.

5. Conclusions and Future Works

Our investigation into engraved digit recognition led to the development of our CNN-ED model with an exceptional performance. We evaluated this model alongside other supervised classifiers on our dataset and obtained remarkable results. Our CNN-ED model achieved an impressive 99.88% accuracy with perfect precision on our dataset, surpassing all other classifiers, including the classic DT, SDG, GNB, and RF classifiers, which failed to meet the 91% benchmark accuracy. The SVM and KNN classifiers achieved 91% accuracies. The CNN-ED model, which was trained on a combined dataset of original and WGAN-GP- and WGAN-DIV-generated images, achieved an outstanding accuracy. This result highlights the effectiveness of augmentation techniques in enhancing a model’s performance in recognizing engraved digits. Leveraging the augmentation power significantly improved the model’s accuracy, making it a reliable solution for various real-world applications that rely on accurate engraved digit recognition.

In conclusion, CNN-based models and the SVM achieved high accuracies on our dataset, outperforming other classifiers. Simple CNN models were particularly effective in recognizing engraved digits, demonstrating a superior accuracy over classic classifiers. These models can generalize to new examples and efficiently handle various digit image variations, making them an ideal choice for applications requiring engraved digit recognition. Our future work will involve expanding the dataset to include diverse samples of engraved digits from various sources. This expansion will enable a comprehensive evaluation of the digit styles and variations, contributing to the development of more accurate and robust recognition models. It will also facilitate the examination of the classifiers’ generalizability across multiple datasets, revealing potential biases or limitations in the current models. Lastly, this dataset expansion will allow for a thorough comparison of different classification algorithms, which will highlight their respective strengths and weaknesses in engraved digit recognition.