Real-Time Machine Learning for Accurate Mexican Sign Language Identification: A Distal Phalanges Approach

Abstract

Effective communication is crucial in daily life, and for people with hearing disabilities, sign language is no exception, serving as their primary means of interaction. Various technologies, such as cochlear implants and mobile sign language translation applications, have been explored to enhance communication and improve the quality of life of the deaf community. This article presents a new, innovative method that uses real-time machine learning (ML) to accurately identify Mexican sign language (MSL) and is adaptable to any sign language. Our method is based on analyzing six features that represent the angles between the distal phalanges and the palm, thus eliminating the need for complex image processing. Our ML approach achieves accurate sign language identification in real-time, with an accuracy and F1 score of 99%. These results demonstrate that a simple approach can effectively identify sign language. This advance is significant, as it offers an effective and accessible solution to improve communication for people with hearing impairments. Furthermore, the proposed method has the potential to be implemented in mobile applications and other devices to provide practical support to the deaf community.

1. Introduction

Sign language recognition (SLR) has become crucial for bridging the communication gap between hearing and deaf people, thus facilitating assistive technologies, primarily through mobile applications [1,2,3]. In recent decades, advances in computer vision and machine learning (ML) have led to significant progress in SLR [4,5]. At the same time, research efforts have focused on developing technology-based solutions that improve communication and overall quality of life for the hearing-impaired community [6].

This study focuses on Mexican sign language (MSL) recognition, although it can be applied to other sign language, such as American sign language (ASL), using hand-angle analysis [7,8]. A MediaPipe hand skeleton descriptor is used for training, and the set of six angles for each letter is plotted using OpenCV [9,10,11]. These data are stored in a dataset and used to train a decision tree (DT C4.5) and label the predicted letter. To validate the model, the angles of the hand signals are compared to the training dataset. If a match is found, the model classifies it. Otherwise, it makes a prediction. A schematic of the hand signals of the MSL letters is shown in Figure 1.

This method differs from other ML-based approaches by requiring a relatively modest amount of input data, significantly reducing computational time for accurate MSL interpretation. This demonstrates that classical ML approaches can be more effective for real-time classification problems than some state-of-the-art ML algorithms [12,13]. The main contributions of our method can be summarized as follows:

i.

only six measurements (angles) are used as feature descriptors;

ii.

efficient and accurate prediction;

iii.

relatively small training dataset;

iv.

high efficiency in letter prediction.

The proposed method based on hand-angle analysis achieves significantly higher accuracy and efficiency in MSL recognition than other methods reported in the field, as demonstrated and mentioned below. A relatively small training dataset was used, and feature extraction using six specific hand angles allows for accurate and fast interpretation of sign language gestures, which improves communication and quality of life for the hearing impaired.

The following sections of this paper are organized as follows. Section II discusses the previous related work. Section 3 provides the materials and methods. Section IV presents the results of experimental performance metrics and their analysis. Section 5 presents the discussion, and Section 6 contains the conclusions.

2. Related Work

Sign alphabet recognition is a form of communication that involves images or videos depicting one or two hands. Experts have devoted efforts to predicting letters, words, and ideas in sign languages to assist people with hearing impairments using advanced technologies and artificial intelligence algorithms. Despite the many types of research in this field, there are limitations and opportunities for improving communication between hearing and non-hearing individuals. In this section, we will focus on exploring one-handed SRL. To this end, we present a brief literature review of recent works on SRL using deep-learning techniques, particularly convolutional neural networks (CNNs) and ML based on digital image processing and sensors [14,15,16,17].

Among the reviewed works, Ameen et al. (2016) explored the applicability of deep learning for sign language interpretation by developing a CNN to classify images based on manual spelling. They achieved 82% accuracy and 80% recall using intensity and depth data [18]. Thongtawee et al. (2018) presented an efficient feature-extraction method and algorithm to distinguish American sign language (ASL) from static and dynamic gestures, achieving 95% recognition using an artificial neural network (ANN) [19]. Rastgoo et al. (2020) addressed the challenge of real-time SLR using extra spatial hand relation (ESHR) and hand pose (HP) features, a 2D CNN, singular value decomposition (SVD), and long short-term memory (LSTM) with an accuracy of 86.32% [20,21].

Sharma et al. (2020) proposed a systematic statistical analysis and evaluated previously trained deep models for static Indian sign language (ISL) recognition. They achieved 99.0% and 97.6% recognition accuracy for numbers and letters, respectively, using a public ISL dataset [22]. Katoch et al. (2022) reported a technique using the bag-of-visual-words (BOVW) model to identify ISL alphabets and digits in a live video stream, achieving an average accuracy of 89.24% using support vector machine (SVM) and CNN [23,24,25]. Subramanian et al. (2022) proposed an optimized hand skeleton descriptor integrating the model of closed recurrent unit (MOPGRU) for ISL recognition, achieving an average accuracy of 95% using the bidirectional long short-term memory network (BiLSTM) [26]. Sundar et al. performed a hybrid with long short-term memory (LSTM), achieving an accuracy of 99% [27,28]. Pathan et al. investigated SLR using CNN and an ASL image dataset, achieving a test accuracy of 98.98% [29]. Sanchez et al. investigated a word-level SLR methodology on the Corpus LIBRAS dataset (Brazilian sign language), obtaining an accuracy of 94.33% using BiLSTM [30]. Mohsin et al. focused on letter and number recognition in ASL and achieved 96% accuracy using InceptionV3 [31]. Amangeldy et al. proposed an improved method for continuous recognition of Kazakh sign language, achieving an average accuracy of 97% [32]. Finally, Wali et al. comprehensively reviewed emerging frameworks and algorithms in SLR, identifying state-of-the-art techniques and suggesting new research directions [33].

2.1. The Taxonomy of Sign Language Recognition

Farooq et al. (2021) propose a taxonomy of SLR that includes applications, avatar technology, gesture recognition, natural language to sign language translation, and repositories of written text units, such as letters, words, or sentences. Individual signs in sign language consist of gestures or hand movements, each representing a specific word. They can also be considered static images, where one image corresponds to one word. Signed phrases, or running signs, contain multiple words and are considered sign alphabets or static signs. They are easier to recognize than full sentences. This section reviews several studies in this area [31,34].

Research on sign language translation identifies several possibilities. These efforts aim to improve communication between people with and without hearing impairments. Our proposed approach is highlighted in the white box in Figure 2.

2.2. Machine Learning Models in Sign Languages

Among the ML models similar to ours, MediaPipe was used by Bajaj et al. (2010) to experiment with different combinations of landmarks and classification algorithms, such as K-nearest neighbors (KNN), random forests, and neural networks. After preprocessing to bring the landmarks into a single reference frame, these researchers obtained average accuracies of 82.19% for KNN, 85.30% for random forests, and 90.95% for neural networks, respectively [35,36,37,38]. Sahoo et al. (2014) used various classification techniques, such as position and motion-based SLR using MultiStream (HMM), the Boost Map method (BoostMap), and neural networks, for sign language-related gestures. Their approach focused on determining the shape and center of gravity of the hands in images captured by a digital camera. They used landmarks to train signers and achieved satisfactory accuracy [39,40,41].

In 2021, Shah et al. used SVM multiple-kernel learning classification to recognize static Pakistani sign language alphabets. They extracted visual features from images, such as local binary patterns (LBP), histogram of oriented gradient (HOG), and speeded-up robust features (SURF). They classified them using multiple kernels, achieving an average accuracy of 89.24% [42,43,44]. Hussain et al. (2022) focused on ASL and Irish sign language (ISL) annotation, using MediaPipe to extract features and classify hand gestures. They achieved a maximum accuracy of 93% and 96.7%, respectively, with different classifiers [45]. Despite the extensive existing research, in this study, we will compare ML methods such as SVM, naive Bayes, KNN, random forests, XGBoost, and LightGBM, in addition to DT C4.5 [46,47,48,49].

Unlike previous research, our approach is characterized by its simplicity. It is based on only six features in real-time SLR. The careful feature selection and efficiency of the model support its applicability in practical situations.

3. Materials and Methods

3.1. Software and Hardware Characteristics

For this work, a CPU with an AMD Ryzen 5 5600 G processor and Radeon graphics at 3.90 GHz, 16 GB of RAM, and a Logitech Model C920 HD Pro 1080p 960-000764 webcam were used. The model was built using the MediaPipe classifier as a descriptor of the hand skeleton, OpenCV to measure the internal angles of the hand, and the Python sklearn library (version 9.11.9) to construct the C4.5 decision-tree classifier [50].

3.2. Data Loading and Dataset Preparation

The process began with collecting a dataset explicitly designed for our task, extracted from a proprietary database stored in an Excel file with 5690 records collected and trained by four people of different genders and ages. Furthermore, the database can be downloaded from https://github.com/gggvamp/pdi/blob/main/datosO.xlsx (accessed on 4 April 2024), since we focus on classification and prediction in the language domain. The last column of the dataset contains the class labels corresponding to the sign language letters, while the previous six columns contain the relevant predictive features of the hand, which can be seen in Figure 3a,b. The training set was used to fit the model letter by letter. We apply essential preprocessing techniques to ensure the quality and adequacy of the data. This process included the identification and removal of outliers, which could bias the results of the analysis, and the normalization and standardization of the characteristics were also carried out. We adjusted the scales of the six features to achieve a standard distribution, which facilitates the comparison of different hand signs (letters) and improves the stability of the analysis models. This is conducted for each letter [51,52]. See Figure 3a,b.

We split the data into training and test sets, allocating 20% of the data to the test set and 80% to the training set to ensure adequate representation of the algorithms [53]. Additionally, we used the numerical value of 42 to initialize the random number generator, ensuring the reproducibility of the data splitting. This separation allows for adequate model training with independent data before the evaluation of the test set, avoiding overfitting and providing a more accurate assessment of model performance under real-time and user-independent conditions [54].

Once the data have been collected, the requisite preprocessing is initiated to ensure the quality and suitability of the data for subsequent analysis. This process involves several crucial steps designed to preserve the integrity and consistency of the data. These steps include identifying and eliminating outliers that could skew the analysis results. Additionally, the features are normalized or standardized, adjusting their scales to achieve a standard distribution. This step facilitates the comparison of different variables and improves the stability of the analysis models.

3.3. The Performance Evaluation of Trained Models

The ML algorithms were evaluated using several metrics, including prediction robustness, completeness, sensitivity, specificity, precision, recall, and F1 score (a combination of precision and recall), in addition to time and accuracy in prediction, training, and validation.

Once the models were trained, their performances were evaluated using the test suite, which involved calculating numerous metrics designed to provide a complete and detailed understanding of each model’s performance. Among these metrics are confusion matrices, which provide a robust evaluation of each classification method on datasets containing 21 classes (letters).

The confusion matrix is a valuable source of information about the classifier’s predictions. To improve clarity, it is common practice to normalize the confusion matrix by converting absolute counts to proportions. This normalization allows for comparing model performance between classes with different sample sizes. Additionally, the matrix is presented as a heat map, where darker shades represent higher values. This visualization helps to identify areas of high confusion, as shown in

Table 1. Description of a confusion matrix. The confusion matrix evaluates classification accuracy and determines a classifier’s overall performance. It defines key concepts, such as precision (P), recall (R), true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN).

Actual Class
		Positive	Negative
Predicted Class	Negative	False Negative (FN)	True Negative (TN)	Neg Pre. Value TN/(TN + FN)
	Positive	True Positive (TP)	False Positive (FP)	Precision TP/(TP + FP)
	F1-score (2xPxR)/P + R	Recall TP/(TP + FN)	Specificity TN/(TN + FP)	Accuracy TP + TN/(TP + TN + FP + FN)

, and provides a clear understanding of model performance.

3.4. Selection of Algorithms to Compare

Several ML algorithms have been selected for comparison to assess their performances in addressing the classification and prediction problem. The selected algorithms are:

• SVM: data classification is achieved by identifying the optimal separating hyperplane between classes in a multidimensional space;
• Naive Bayes: this approach is based on Bayes’ theorem and assumes independence between characteristics and the classes;
• KNN: data classification is performed by assigning labels based on nearest neighbor labels;
• Decision trees: this method classifies data using a decision tree, where each node represents a feature and each leaf a label;
• Random forests: combine multiple decision trees to classify data, reducing overfitting;
• XGBoost: implements gradient boosting to improve model accuracy using sequential decision trees with regularization and parallelization;
• LightGBM is another efficient implementation of gradient boosting. it uses sampling techniques to build decision trees faster and with lower memory usage;
• CatBoost is an ML algorithm developed by Yandex for gradient boosting on decision trees. It is particularly effective at handling categorical features directly, preventing overfitting, and offering high performance with both CPU and GPU support;
• RNNs: recurrent neural networks are designed for processing sequential data by maintaining a hidden state that captures information from previous time steps. They are widely used in applications like time-series forecasting and natural language processing.

The selected algorithms were implemented at this stage using well-known ML libraries, such as Scikit-learn 1.4.2, XGBoost, and LightGBM. Each model was carefully configured and trained using the training set prepared during the data-preprocessing phase. During the training process, rigorous monitoring was conducted by recording the time required for each model’s training. Several evaluation metrics were calculated and thoroughly analyzed, including training accuracy, a crucial measure of each model’s predictive ability and fit. It should be noted that all algorithms compared were subjected to the same circumstances and initial conditions. This meticulous approach ensured a thorough understanding of the models’ performance and allowed the identification of areas for improvement to optimize their performance in future phases of the project [55,56].

3.5. Proposed Work

This work uses a hand-feature extractor called the MediaPipe Hand Landmarker. This tool identifies key points on the hands in an image. These points can be used to detect significant locations on the hands and apply visual effects to them. See Figure 4a,b.

Measuring the internal angles between the distal phalanges and the palm is crucial to our methodology. These angles capture distinctive features without relying on correlation and convolution processes. Calculating these angles provides a concise representation of the hand signals and, through OpenCV, stabilizes the plot regardless of the position or distance of the hand from the camera. This stability is vital for improving the accuracy and speed of letter prediction in sign language. The angle is calculated using the direction vectors of the two lines $\overrightarrow{\mathit{u}}=\left(u_1{,u}_2\right)$ and $\overrightarrow{\mathit{v}}=\left(v_1{,v}_2\right)$, and the angle formed by these two lines can be calculated using Equation (1):

$$\cos \alpha =\left({\displaystyle \frac{\left|\overrightarrow{u}·\overrightarrow{v}\right|}{\left|\overrightarrow{u}\right|\left|\overrightarrow{v}\right|}}\right)={\displaystyle \frac{\left|u_1{v}_1+u_2{v}_2\right|}{\sqrt{u_1^2+u_2^2}+\sqrt{v_1^2+v_2^2}}}$$

(1)

where $\left|\overrightarrow{\mathit{u}}\right|$ and $\left|\overrightarrow{\mathit{v}}\right|$ are the modules of vectors u and v, respectively, the angles between the distal phalanges and the palm are obtained. It should be noted that this contribution is particularly significant, since not only are the features reduced from 21 to 6 dimensions but also, by focusing on distal angles, the values of these angles are consistent regardless of the position or distance relative to the camera. This is a significant advantage in processing, classifying, and, above all, predicting MSL letter labels. See Figure 5a–d.

3.5.1. Training Stage

During the training stage, real-time video recordings are performed for each letter of the Spanish alphabet using the MSL variant. The training begins by labeling each sign with its corresponding letter, starting with the sign for the letter A and ending with the sign for Y, excluding the letters J, K, Ñ, Q, X, and Z due to their movement nature. The six most critical characteristic points are plotted using Equation (1) to derive and record the six angles, from which 21 of the 27 MSL letters are obtained.

These data are stored in a dataset containing each letter of the corresponding alphabet. For example, in Figure 6, the letter ‘B’ is trained. Records of the features of each letter are stored in a more extensive dataset called ‘alphabet classes’, which is accessible in the training and validation phase. See Figure 6.

The training algorithm outlines a system that uses a camera to detect and track hands in real time. The process begins with importing the necessary libraries for image processing and hand detection. Video capture from the camera is set in a continuous loop, where the algorithm captures a frame and verifies the success of the capture. The algorithm then searches for the user’s hand within the frame. If a hand is detected, the algorithm analyzes it and calculates the coordinates of the hand landmarks. These coordinates are used to draw lines on the frame, representing the hand. Additionally, the algorithm calculates six angles using the hand landmark coordinates, which are stored. The modified frame with the detected hand and calculated angles is displayed on the screen. The loop continues until the user stops it, at which point the camera resources are released. For a detailed description of this phase, refer to

Table 2. Training and labeling process.

Pseudocode 1 Hand Angle Data Collection and Training

1: Start 2: Import necessary libraries (cv2, mediapipe, numpy, pandas) 3: Configure mediapipe for hand detection and capture video from the camera 4: Write data into an Excel file 5: Initialize counter to zero 6: While true: 7:     Capture a frame from the camera 8:     If the frame was captured successfully: 9:             Flip the frame horizontally 10:             Convert the frame to RGB format 11:             Process the frame with the hand-detection model 12:              If hands are detected in the frame: 13:                  For each detected hand: 14:                      For each hand landmark: 15:                           Calculate the coordinates of finger landmarks and wrist 16:                           Draw circles at finger landmarks and wrist 17:                           For each pair of landmarks forming a hand: 18:                           Calculate the angles between fingers and wrist 19:                           Draw lines between landmarks to represent the hand 20:                           Display the calculated angles near the landmarks 21:                           Print the calculated angles 22:                           Show the frame with detected hands and calculated angles 23:                           If the 'ESC' key is pressed: 24:                            Exit the loop 25:                           Release camera resources and close all windows 26:                            End

The process begins with the importation of the requisite libraries and the configuration of hand detection. This is followed by writing data to an Excel file and initiating an infinite loop to acquire and process images captured by the camera. Within this loop, the algorithm detects the hand in the image, calculates the reference angles, displays the results, and stores the calculated angles.

.

3.5.2. Validation Phase

Validation is a crucial step to ensuring the accuracy and reliability of the character recognition system. For this purpose, the DT C4.5 ML method, specifically the C4.5 algorithm, is employed [57]. The validation process comprises several key steps. Initially, hand signals are processed analogously to those used during training. Six internal angles of the hand are extracted as features for validation. These angles are compared to a set of “alphabet classes,” which contain information about the signs corresponding to each of the 21 letters of the alphabet. The objective is to determine if the captured hand signal corresponds to one of the previously trained alphabet classes. If no direct correspondence with the alphabet classes is found, the DT C4.5 algorithm is activated. This algorithm is renowned for its capacity to construct decision trees based on the information provided by the training data. These decision rules may be employed to classify unknown records from the captured hand signal. In this manner, the DT C4.5 functions as a predictor, determining which letter of the alphabet the signal belongs to and labeling it correctly. For further clarification, please refer to Figure 7.

This system employs a camera and a hand recognition model to capture and process gestures. The camera initiates video capture, while hand-gesture data are read from an Excel file, potentially containing information from a previous training session. An infinite loop is established to capture frames from the camera continuously. Each frame is processed with the hand recognition model, extracting the coordinates of the landmarks and calculating the angles between them. The processing cycle persists until the user elects to terminate it. At this juncture, the DT C4.5 model categorizes the captured signals. The image depicts the classified signal, and the resulting video with the gesture labels is displayed for future reference.

Table 3. Validation process for hand-gesture recognition system with DT C4.5.

Pseudocode 2 Real-Time Hand Gesture Recognition Validation

1:    Start 2:    Import necessary libraries (cv2, mediapipe, matplotlib, numpy, pandas, sklearn) 3:    Configure mediapipe for hand detection and capture video from the camera 4:    Read hand gesture data from an Excel file 5:    Initialize necessary variables and data structures 6:    With mediapipe.Hands( 7:        static_image_mode = False, 8:        max_num_hands = 2, 9:        min_detection_confidence = 0.5) as hands: 10:       While True: 11:            Read a frame from the camera 12:            If the frame was read successfully: 13:                Process the frame with the hand detection model 14:                If hands are detected in the frame: 15:                    For each detected hand: 16:                        Extract coordinates of finger landmarks and wrist 17:                        Calculate angles between finger landmarks and wrist 18:                        Visualize landmarks and hand lines on the frame 19:                        Display calculated angles near the landmarks 20:                        Store the calculated angles 21:            Show the frame with detected hands and calculated angles 22:            If the ‘ESC’ key is pressed: 23:               Perform gesture classification using a decision tree model 24:               Display the predicted gesture label on the frame 25:               If needed, save the output video with gesture labels 26:               Exit the loop 27:        Release camera resources and close all windows 28:    End

provides a detailed description of the validation process.

The pseudocode presented elucidates the real-time validation of hand-gesture recognition. As stated, it imports libraries and reads gesture data from an Excel file. After initializing the requisite variables, it initiates an infinite loop to capture and process images from the camera.

3.6. Decision Tree

This algorithm needs to be explained because it performed best compared to the ML algorithms in the results section; this decision algorithm and its possible results are presented as a tree. It is used for classification and regression tasks, especially DT C4.5, which has served as the basis for several variants and improvements in the design of decision-tree algorithms, such as random forests [58,59]. In this context, the tool predicts the category (letter) of the hand signs not found in the previously trained set of alphabet classes. Although this tool is not new, our experiments have shown extraordinary efficiency, responsiveness, and accuracy despite its simplicity of application. Its use in this work is further detailed by explaining the parts that make up this algorithm.

3.6.1. Entropy

Entropy is a physical quantity applied to a thermodynamic system in equilibrium. Its function is to measure the number of microstates compatible with the macroequilibrium state of the system. This measurement can be understood as measuring the degree of organization present in the system in that state. In our case, when the entropy level is zero, it represents the maximum order. The decision tree is created based on the gain of information obtained from the training examples and is then used to classify the test set. The classification task is typically performed with nominal attributes and no missing values in the dataset. If a probability distribution $P=(p1,p2,\dots ,pn)$ is provided, then the information carried by this distribution is known as entropy and is calculated by:

$$EntropyP=-{\sum }_{i=1}^n{p}_i·\mathrm{l}\mathrm{o}\mathrm{g}\left(p_i\right)$$

(2)

3.6.2. Information Gain

To select the attribute of a given node at any position in the tree under construction, it is necessary to determine the gain for a t-test and a position p at that node using the following equation:

$$Gan(p,T)=EntrP-{\sum }_{j=1}^n(pj·Entr)(pj)$$

(3)

The values ($pj$) represent the set of all possible values for the attribute. This measure given in Equation (3) can be used to determine which attribute is better and to construct the decision tree by considering the node that possesses the attribute with the highest information gain of all the attributes that have not yet been considered in the route from the root node.

3.6.3. Algorithm C4.5

The C4.5 algorithm is an extension of the ID3 algorithm proposed by Quinlan to address some of the deficiencies of ID3, in that it was not designed for numeric attributes and does not use pruning to reduce overtraining. To solve these problems, algorithm C4.5 uses a new calculation to measure the gain ratio [60]. Thanks to the introduction of this new calculation, it is possible to calculate a gain ratio:

$$RelGan(p,T)={\displaystyle \frac{Gan(p,T)}{infDiv(p,T)}}$$

(4)

where

$$\mathrm{i}\mathrm{n}\mathrm{f}\mathrm{D}\mathrm{i}\mathrm{v}(\mathrm{p},\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t})=-{\sum }_{j=1}^{k}p\left({\displaystyle \frac{j}{p}}\right)·\mathrm{l}\mathrm{o}\mathrm{g}(p’\left({\displaystyle \frac{j}{p}}\right))$$

(5)

p′($j/p$) is the proportion of elements present at position p, calculated from the umpteenth test. Unlike the entropy in ID3, the gain ratio is independent of the objects distributed in different classes. C4.5 handles attributes with unknown values more effectively by evaluating the gain ratio for these attributes by considering only the datasets for which that dataset is defined. To accomplish this task, the algorithm estimates the probabilities of different outcomes. Then, the new gain criterion takes the form.

$$Gan\left(p\right)=F\left(Info\right(T)-Info(p,T\left)\right)$$

(6)

where $F$ is the number of examples in the dataset with known values for the number of examples in an attribute dataset.

$$\mathrm{I}\mathrm{n}\mathrm{f}\mathrm{o}(\mathrm{T})={\sum }_{i=1}^n(\left(p_j\right)Entropia(p_j))$$

(7)

If we first partition T into sets $p_1$, $p_2$…, $p_n$ based on the value of a non-categorical attribute p, then the information needed to identify the class of an element of T becomes the weighted average of the information required to specify the class of a component of $T_i$, i.e., the weighted average of Info ($T_i$). It also handles attributes with continuous values. Let $p_j$ be an attribute with a continuous value in a range of constant values. The values of these attributes are examined in the training set. The gain for each partition is calculated, and the gain-maximizing partition is selected. An alternative approach used in the C4.5 algorithm is the technique of post-pruning. The algorithm does not stop during execution. Therefore, it also allows over-fitting, and only at the end are pruning rules applied to improve the generalization ability. Another difficulty is handling continuous value attributes, such as real numbers [61]. Lastly, C4.5 uses a pruning technique to minimize the error rate. This technique reduces the size of the tree by removing parts that may be due to incorrect or missing data, thereby reducing the complexity of the tree and improving its classification performance.

3.6.4. Measure of Gini Impurity

Gini impurity is a metric used to create classification trees, providing more insight into the data distribution per node than classification accuracy alone. It is calculated by considering the proportion of each target category among all the records at a node. The Gini impurity is computed as the sum of the squares of these proportions subtracted from one. For example, when splitting a node, the algorithm seeks the split that maximizes the reduction in impurity, defined as the impurity of the parent node minus the weighted average impurity of the child nodes. The overall objective is to minimize the Gini impurity.

$$Gini(Xq)=1-{\sum }_{k=1}^k{\left(p_{k,q}\right)}^2$$

(8)

This is used for categorical attributes. This criterion attempts to estimate the information provided by each attribute based on “information theory”. Entropy is a measure of the uncertainty or randomness of a random variable “x”. By calculating the entropy for each attribute, the information gain of the tree can be determined.

4. Results

4.1. Evaluation Metrics and Their Relevance

Various metrics have been used to evaluate model performance, including training and validation accuracy and training and prediction time. These metrics are essential for understanding different aspects of model performance. Accuracy provides a measure of the quality of predictions, while training and prediction time give information about the computational efficiency of the algorithms, which is critical for real-time or large-scale applications.

Several ML models have been trained using algorithms, including random forest, SVM, naive Bayes, KNN, decision tree C4.5, LightGBM, XGBoost, CatBoost, and RNNs. This broad range allows for a comprehensive comparison between different modeling paradigms, which is crucial for understanding which approaches best fit a dataset and the sign language predictor. The importance of this analysis lies in its ability to determine the algorithm that offers the best performance in terms of accuracy, computational efficiency, and generalizability, all of which are essential for the success and utility of our sign language predictor. The results illustrate how the accuracy of the models varies with increasing the training time.

Table 4. Performance Models.

Model Results
Metric/Method	D T C4.5	SVM	N B	k-NN	RF	XGBoost	LightGBM	CatBoost	RNNs
Training Accuracy	1.00	0.98	0.97	0.99	1.00	1.00	1.00	1.00	0.92
Training Loss	0.00	0.02	0.03	0.01	0.00	0.00	0.00	0.00	0.27
Testing Accuracy	0.99	0.96	0.96	0.98	0.99	0.99	0.99	1.00	0.96
Testing Loss	0.01	0.04	0.04	0.02	0.01	0.01	0.01	0.00	0.04
Accuracy	0.99	0.96	0.96	0.98	0.99	0.99	0.99	1.00	0.96
Precision	0.99	0.96	0.97	0.98	1.00	0.99	0.99	1.00	0.96
Recall	0.99	0.96	0.96	0.98	0.99	0.99	0.99	1.00	0.96
F1 Score	0.99	0.96	0.96	0.98	0.99	0.99	0.99	1.00	0.96

The numbers marked in red are the ones with the best performance.

presents the performance comparison of the ML models.

Throughout this process, we closely examined how performance metrics, such as accuracy, varied between the training and test sets, and how the performances of the different classification models were reflected in terms of precision, recall, F1 score, and accuracy.

Table 5. Training time and prediction.

Execution Times
Time/Method	D T C4.5	SVM	NB	k-NN	RF	XGBoost	LightGBM	CatBoost	RNNs
Training Time	0.01 s	0.05 s	0.00 s	0.02 s	0.36 s	0.31 s	0.51 s	2.42 s	7.09 s
Prediction Time	0.00 s	0.03 s	0.01 s	0.03 s	0.02 s	0.00 s	0.10 s	0.00 s	0.12 s

The numbers marked in red are the ones with the best time.

presents the training and prediction time metrics.

The decision tree, naive Bayes, and k-NN are fast at training and prediction. SVM and random forest have moderate training and prediction times. XGBoost and CatBoost are quick predictors, but CatBoost has a longer training time. LightGBM is efficient for large datasets but slower in prediction. RNNs have long training and prediction times. The choice of algorithm depends on the problem requirements and available computational resources. Cross-validation scores are used to assess the generalization ability of a machine-learning model; see

Table 6. Indicates the cross-validation metrics.

Cross-Validation Scores
Metric/Method	D T C4.5	SVM	NB	k-NN	RF	XGBoost	LightGBM	CatBoost	RNNs
Cross-Validation Time	0.07 s	0.45 s	0.02 s	0.24 s	1.78 s	1.58 s	2.41 s	11.17 s	29.00 s
CV Scores	[0.977, 0.985, 0.984, 0.984, 0.985]	[0.946, 0.960, 0.932, 0.952, 0.951]	[0.954, 0.963, 0.936, 0.926, 0.976]	[0.965, 0.982, 0.954, 0.944, 0.973]	[0.984, 0.992, 0.971, 0.974, 0.976]	[0.937, 0.988, 0.967, 0.976, 0.981]	[0.937, 0.989, 0.967, 0.976, 0.981]	[0.990, 0.992, 0.991, 0.995, 0.993]	[0.967, 0.970, 0.967, 0.968, 0.980]
CV Mean	0.98	0.96	0.95	0.96	0.98	0.97	0.97	0.99	0.97
CV Standard Deviation	0.00	0.01	0.02	0.01	0.01	0.02	0.02	0.00	0.01

The numbers marked in red are the ones with the best performance.

. Cross-validation is a technique that helps ensure that the model performs well on the training dataset and previously unseen new data.

All models have a high overall accuracy, ranging from 0.96 to 0.99. This indicates that most of the models’ predictions are correct compared to the total predictions. Accuracy measures the proportion of correct positive predictions. The models’ accuracy ranges from 0.96 to 0.99, indicating a very high ability to predict instances correctly. State-of-the-art algorithms, such as CatBoost, XGBoost, and LightGBM, are the most accurate, although they struggle with training time. Metrics using clustered bar graphs facilitate visual comparisons between the different models. However, it is essential to interpret the results with caution. For example, a model with a high training accuracy but a low validation accuracy may indicate overfitting, while longer training and prediction times can be problematic in time-critical applications. See Figure 8.

It is essential to contextualize the results concerning the specific problem and data under consideration. What works well in one dataset may not apply to another. In addition, evaluation results should be cautiously generalized and validated on independent datasets to ensure their robustness and reliability. The implementation provides a valuable exploration of the performance of different ML algorithms. However, to describe the performance of all models based on the graph provided, we note the following:

i.

Random forest: Although random forest is known for its ability to handle large datasets with many features, it appears to be more time-consuming in training and prediction compared to other models. Nonetheless, it provides good accuracy on both training and test sets;

ii.

SVM: it has longer training and prediction times than other models, and its accuracy is not the highest on this dataset;

iii.

Naive Bayes: Although it has shorter training and prediction times, its accuracy is lower than other models. However, it could be a good choice if speed is a priority, and the required accuracy is reasonably high but not critical;

iv.

KNN: It shows very short training times, but the prediction times are longer. Its accuracy is relatively high, but its distance-based nature may not be optimal for large datasets;

v.

DT C4.5: it shows shorter training and prediction times compared to other models, and its accuracy is comparable to and even better than that of other more complex models on this dataset;

vi.

XGBoost and LightGBM: These gradient-boosting models have good accuracy results but longer training times than DT C4.5. However, their prediction times are shorter than those of random forest and SVM;

vii.

CatBoost: It performs excellently, with an accuracy of 1.00. However, its training time is significantly higher (2.42 s), and the cross-validation time is also longer (11.17 s);

viii.

Neural networks: although neural networks have good accuracy and recall (0.96), their training times (7.09 s) and cross-validation (29.00 s) are much longer compared to the decision tree and boosting models.

4.2. Confusion Matrix of the Compared Models

The confusion matrix, a fundamental component in evaluating the performance of the classification model, serves to quantify the model’s accuracy by showing the number of correct and incorrect predictions for each class (letter). The confusion matrix is visualized, highlighting the relationships between classes and facilitating the identification of patterns of classification errors. Below, you can see the confusion matrices that were compared to determine which sign language predictor to choose. See Figure 9a–i.

The last matrix, DT C4.5, shows the number of correct and incorrect predictions for each class in a tabular format. It provides a detailed understanding of how the model classifies instances in each class. The model is accurate overall, with many correct predictions for most classes. The accuracy varies between the classes, indicating that the model may perform better for some classes than others. The high-performing classes are A, C, G, P, T, W, and Y, mostly with correct predictions with no significant false positives or negatives. This indicates that the model classifies these classes effectively. Some classes, such as M, R, and U, show false positives or negatives. These classes present opportunities for improvement to enhance the model’s accuracy in classifying these instances. Overall, the confusion matrix results provide valuable information about the performance of the DT C4.5 model, highlighting areas of strength and areas that need improvement for more accurate classification. Based on these comparative analyses, we could decide which algorithm best balances accuracy and computational efficiency for our sign language predictor. This evaluation allowed us to select the most appropriate model to meet the problem’s requirements and ensure optimal performance in practical situations.

4.3. Performance of the DT C4.5 Classification Model

The following metrics provide a clear and concise evaluation of the proposed method’s performance with DT C4.5 for classifying different letters. These metrics, including accuracy, precision, recall, and F1 score, are based on the most recent results and are presented straightforwardly in Table 4, Table 5 and Table 6 and Figure 8. Please refer to

Table 7. Metrics for the proposed method using the decision tree C4.5.

Letter	Accuracy	Precisión	Recall	F1-Score	Support
A	0.99	0.98	1	0.99	82
B	0.99	0.98	1	0.99	47
C	0.99	1	1	1	54
D	0.99	1	0.95	0.98	42
E	0.99	1	0.99	0.99	70
F	0.99	0.94	0.97	0.96	34
G	0.99	1	1	1	67
H	0.99	1	0.97	0.99	39
I	0.99	1	1	1	62
L	0.99	1	1	1	80
M	0.99	0.98	0.94	0.96	52
N	1	0.97	1	0.99	71
O	0.99	0.97	0.98	0.98	62
P	1	1	1	1	47
R	0.99	1	0.96	0.98	50
S	0.99	0.97	0.94	0.96	35
T	1	1	1	1	54
U	1	0.98	0.97	0.97	60
V	0.99	0.95	0.98	0.97	64
W	1	1	1	1	36
Y	1	0.97	1	0.98	30
Weighted Avg	0.99	0.99	0.99	0.99	1138

for a detailed breakdown.

Table 7 shows the high performance of the proposed method using DT C4.5 in classifying letters.

• Accuracy: This represents the proportion of correct predictions from the total predictions made for each class, providing a general measure of model performance. Values range from 0.99 to 1.0, indicating that the model is highly accurate for most classes;
• Precision: Indicates the proportion of instances correctly classified as positive out of all instances classified as positive. It measures the model’s ability to avoid misclassifying a negative instance as positive. Values range from 0.94 to 1.0, indicating that the model has a low false-positive rate for most classes;
• Recall: Represents the proportion of positive instances correctly identified by the model out of all true-positive instances. It measures the model’s ability to identify all relevant instances in a dataset. Values between 0.94 and 1.0 indicate that the model correctly identifies the most positive instances in each class;
• F1 Score: Measures model accuracy by considering both precision and recall. The harmonic mean of precision and recall balances the two metrics. Values range from 0.96 to 1.0, indicating a good balance between precision and recall for most classes;
• Support: values vary by class and represent the number of instances of each class in the test dataset.

4.4. Structure of Prediction in DT C4.5

A DT C4.5 is a graphical representation of a set of decision rules used to classify examples or predict outcomes, where each internal node represents a feature (attribute), each branch represents a decision based on that attribute, and each leaf represents the result of the decision. The following describes how decisions are made based on specific features by dividing the dataset into smaller groups at each internal node until a prediction is obtained. This leads to the classification of hand angles into letters of the alphabet. A decision tree is generated and visualized based on the given data, and the resulting image is saved. This process helps understand how DT C4.5 classifies different data and which features are most important for classification. See Figure 10.

4.5. The Results of the Characteristics of DT. C4.5

i.

Feature Names: In the context of a C4.5 DT, these are the features used to make decisions at each tree node. It is almost always specified that default feature names be used;

ii.

Gini: The Gini impurity measures how impure a node is. Determining which features and split values are best for dividing the dataset into smaller subsets is critical. In the decision-tree graph, nodes are split based on the Gini impurity value to minimize impurity in the resulting nodes;

iii.

Examples: In the context of DT C4.5, this refers to the number of data instances that arrive at a particular node during the tree’s training process. The number of samples arriving at each node can be shown in the decision-tree graph, providing information about the data distribution in the tree;

iv.

Value: The value at a node represents the class distribution of the samples arriving at that node. In the decision-tree graph, the value of a node can be visualized as a list showing how many samples of each class are present at that node;

v.

Class: The leaf nodes of the decision tree represent the predominant class of samples arriving at that node. In the decision-tree graph, the class of a leaf node can be displayed as a label indicating the predominant class at that node.

The leaf nodes in the tree indicate the predominant class of arriving samples, constituting the final classification of letters in sign language. These steps are essential for understanding and effectively applying the classification process using a DT C4.5 classifier in SLR.

In summary, the code uses a DT C4.5 classifier to predict letters based on the characteristics of the detected hands. The decision-tree graph provides a visualization of how these decisions are made, and terms such as Gini, impurity, samples, value, and class refer to different aspects of the tree construction process and data distribution in the tree. The images in Figure 11 show the manual sign results for the 21 non-movement letters that define MSL.

It is important to note that the results from the instantaneous completion of the process are obtained from real-time video, not photographs. As with all video systems, the two limitations of the method are the lighting quality and the lack of focus of the webcam. The images in Figure 11 show the 21 letters of the alphabet, excluding those with movement.

5. Discussion

This work is distinctive in the scientific literature because it analyzes six features that represent the angles between the distal phalanges and the palm. This approach minimizes the need for complex image processing. Additionally, there is limited research on MSL.

In recent MSL research published by Gonzalez et al. [12], MediaPipe was used as a descriptor of the face, body, and hands to create avatars. The system includes an easy-to-use graphical interface with modes to translate between MSL and Spanish in both directions. Users can enter characters or text and receive corresponding translations. The performance evaluation shows high accuracy, with the bidirectional neural network model achieving an accuracy of 98.8%. Like us, they reduce the dimensionality of the features in their work to 11 for the face and 5 for the body but keep the 21 critical points of the hand. In contrast, in our work, the dimensionality is reduced to six features, obtaining similar results in accuracy.

On the other hand, Sosa et al. [13] conducted a study using MSL. They proposed a system to recognize and animate signs related to general medical consultations with avatars in real time. This system facilitates dynamic and non-intrusive interaction between hearing doctors and deaf patients. The recognition module uses an MS Kinect sensor to capture sign trajectories and images processed in real time by hidden Markov models (HMMs). The study involved 22 participants and demonstrated the recognition of 82 different signs, achieving average accuracy rates and obtaining F1 scores of 99% and 88%, respectively. The work uses MSL, but the Kinect sensor requires two computers to program speech and train an avatar using motion caption (MoCap), which cannot track finger movements. Therefore, it needs to be adapted afterward. The researchers’ contribution is valuable because it focuses on helping hearing-impaired people communicate in a medical context. Compared to our work, we used a low-cost camera with a medium-capacity computer, which processes finger images much faster. We do not need to train an avatar beforehand or the patient; communication is facilitated directly by the person in need, regardless of hand size or skin color.

According to the same methodology but without the incursion of MSL, we found a promising work by Subramanian et al., who employed a hand-feature descriptor integrating an optimized MediaPipe called gated recurrent units (MOPGRU) for ISL recognition, obtaining an average accuracy of 95% [26]. In contrast, our work did not require an optimized MediaPipe, and we obtained similar results. Just as Hussain [45] identified two alphabets, ASL and ISL-HS, using different kinds of ML, including random forest, DT C4.5, and naive Bayes, to classify hand gestures using a dataset with 28 gestures between letters and 2 signs. The random forest classifier was the best-performing classifier, showing an accuracy of 96.7% with ISL and 93.7% with ASL. However, in our study, the random forest showed more extended training and prediction. However, it is our second-best performance after DT C4.5. Our work has some limitations that make it imperfect; we only have 21 letters of the alphabet. It was not possible to include letters that imply movement, and it would be desirable to complete the MSL. Some classes, such as M, R, and U, show false positives or negatives. These classes are of utmost importance and require special attention to improve the accuracy of the model in classifying these letters.

6. Conclusions

This study presents a comprehensive analysis of various ML models applied to the classification of MSL. Our decision tree C4.5 algorithm demonstrated remarkable performance, achieving near-perfect precision, recall, and an F1 score of 99%. Compared to cutting-edge algorithms like random forest, XGBoost, LightGBM, CatBoost, and neural networks, the DT C4.5 algorithm stands out for its balance between computational efficiency and predictive accuracy. While models such as CatBoost and neural networks offer competitive accuracy, they require significantly longer training times, which may not be ideal for real-time or large-scale applications. CatBoost, in particular, exhibited excellent performance in accuracy and handling categorical data, but its training time was considerably longer compared to DT C4.5.

Although the neural network was adequate, with a precision and recall of 96%, it presented the most extended training and cross-validation times among all the models tested. This makes it less practical for scenarios requiring quick deployment and iteration. However, its ability to handle complex patterns in the data is noteworthy, suggesting its potential for future improvements where computational resources are less constrained.

Our findings highlight the importance of model selection based on the specific needs of the application, such as training speed, prediction time, and accuracy. DT C4.5 proved to be the most balanced option for our MSL predictor, offering robust performance without the drawbacks associated with more computationally demanding models. This study underscores the potential of simpler models like DT C4.5 to achieve high accuracy in specialized tasks where the advanced models’ complexity and resource demands may not be justified.