Fusion of Attention-Based Convolution Neural Network and HOG Features for Static Sign Language Recognition

Abstract

The deaf and hearing-impaired community expresses their emotions, communicates with society, and enhances the interaction between humans and computers using sign language gestures. This work presents a strategy for efficient feature extraction that uses a combination of two different methods that are the convolutional block attention module (CBAM)-based convolutional neural network (CNN) and standard handcrafted histogram of oriented gradients (HOG) feature descriptor. The proposed framework aims to enhance accuracy by extracting meaningful features and resolving issues like rotation, similar hand orientation, etc. The HOG feature extraction technique provides a compact feature representation that signifies meaningful information about sign gestures. The CBAM attention module is incorporated into the structure of CNN to enhance feature learning using spatial and channel attention mechanisms. Then, the final feature vector is formed by concatenating these features. This feature vector is provided to the classification layers to predict static sign gestures. The proposed approach is validated on two publicly available static Massey American Sign Language (ASL) and Indian Sign Language (ISL) databases. The model’s performance is evaluated using precision, recall, F1-score, and accuracy. Our proposed methodology achieved 99.22% and 99.79% accuracy for the ASL and ISL datasets. The acquired results signify the efficiency of the feature fusion and attention mechanism. Our network performed better in accuracy compared to the earlier studies.

1. Introduction

Various software-based applications have been implanted with the advancement in computer technology, giving multiple ways of interacting with humans through computers. Nowadays, different interfaces are text or speech-based. People with speech and hearing impairments communicate and express their feelings with the help of sign language gestures. It is composed of manual and non-manual features. Manual features represent hand shape, position, palm orientation, and trajectory. Non-manual features comprise movement of the body, shoulders, and facial expressions. So, the deaf community uses gestures to convey meaningful information that is only understandable by sign language experts or the native people. Hence, there is a need to bridge this interaction gap by implementing an intelligent computer and efficient Sign Language Recognition (SLR) system. This system is required to detect the acquired data of sign gestures and provide meaningful information extracted to the machine to develop real-world applications. There are two standard methods in SLR: device-based approach and vision-based methods. The device-based methods involve sensors and data gloves, and the subjects must wear heavy devices that make them uncomfortable performing the signs. These methods enhance accuracy but could not be more efficient in real-world applications [1]. However, the vision-based methods involve non-invasive techniques like high-definition cameras like stereo, monocular, and non-invasive sensors. Vision-based methods involve various image processing techniques to enrich the data visualization for good classification results. Most earlier studies used classical machine learning algorithms that perform feature extraction and classification as two individual steps. However, with the growth of intelligent deep neural networks and the advancement in artificial intelligence, significant improvement has been made in image and video processing. The convolutional layers in the convolutional neural network (CNN) extract many low-level features like edges, corners, curves, etc., and high-level features like object detection from images. There are various classical feature descriptors like scale-invariant feature transform (SIFT), local binary patterns (LBP), and speeded-up robust features (SURF) for the extraction of features [2,3,4]. However, most of these features are subject to scaling, rotation, and brightness variations. So, to solve these issues, a system is needed that uses both traditional handcrafted feature learning methods and deep learning-based models. Nowadays, most researchers use attention-based networks to enrich the model’s efficiency and minimize the computational cost. The main objective of these networks is to focus on relevant features and remove redundant features to classify the gestures [5] accurately. In our proposed methodology, the sign language gesture images are pre-processed and fed to a feature extraction module that extracts specific features from two strategies: histogram of oriented gradients (HOG) and convolutional block attention module (CBAM) attention-based CNN. Finally, the merged features produce a feature vector. Combining two models aims to implement a highly robust architecture to various uncertainties and achieve reasonable accuracy. The summary of the significant contributions of this research work is as follows:

(1)

We implemented an efficient feature extraction technique using the fusion of HOG feature descriptor and attention-based CNN for recognizing hand sign gestures.

(2)

The CBAM module is embedded in CNN to enhance the learning of features at various scales so that the model can handle different alterations in the location and shape of hand gestures and focus on effective features.

(3)

We tested the efficiency of the approach among two datasets using three evaluation metrics, namely precision, recall, and F1-score.

The remaining work of this article is organized into various sections: Section 2 describes the literature survey and the related studies for the prediction of sign language gestures. Then, in Section 3, the architecture of our designed framework is described. Section 4 discusses the overall analysis of the results. At last, we concluded the paper in the Section 5.

2. Related Work

Several researchers have proposed various methodologies to implement a system to recognize sign language gestures. We have categorized the literature survey into traditional, deep learning, and attention-based SLR methods.

Traditional SLR methods: Marin et al. extracted features based on the angle and location of the fingertips. Support vector machine (SVM) model is used to predict classes and achieved 89.70% accuracy [6].

The authors in [7] merged HOG and LBP feature descriptors for hand gesture recognition with a recognition accuracy of 92% on the National University of Singapore (NUS) dataset. In [8], the authors implemented a machine learning-based framework that applied segmentation and used discrete wavelet transform (DWT) with SURF for feature extraction.

Multiclass SVM with the bag of words technique is used to classify hand gestures with an overall accuracy of 96.5%.

The authors in [9] employed HOG features with two classifiers for hand gesture recognition with an accuracy of 96% and 98% with the K-nearest neighbor (KNN) and SVM, respectively.

A multimodal feature vector is obtained using distance transform, contour feature extraction, and K-curvature convex defects techniques for recognizing hand gesture RGB depth dataset [10]. The system acquired a recognition accuracy of 97.4%.

Deep learning based: In the paper [11], the authors used the VGG16 model to classify 36 static signs, including 26 alphabets and ten digits from the American Sign Language (ASL) database. They fine-tuned the parameters using the transfer learning technique on the VGG16 model, which is already trained on ImageNet data and comprises more than a million images. With this methodology, they acquired an impressive accuracy of 96%. Sruthi et al. [12] proposed an Indian Sign Language (ISL) static alphabet recognition system. The face was detected using a face detection algorithm and segmented the hand region using color-based segmentation. The approach uses a deep learning algorithm to recognize 24 sign classes. The methodology obtained a validation accuracy of 98.64%.

The authors have presented a system by concatenating features from two CNN streams on the ASL alphabet dataset using various deep learning models for classification. The system achieved the best accuracy of 97.57% with ResNet50 as classifier [13].

A seven-layered CNN [14] for hand gesture recognition with data augmentation and skin segmentation has attained 96.5% and 96.57% accuracy on two benchmark datasets, respectively.

Attention-based: Nowadays, attention-based deep learning frameworks have shown tremendous growth in achieving more accurate recognition than other deep learning models. The features from the previous layers are fused into an attention block to extract knowledgeable features [15].

The authors in the paper [16] proposed a methodology that used a residual attention-based system that helped enhance feature strength. However, the residual attention module is not portable and computationally expensive.

Park Jongchan et al. [17] designed a framework applying channel and spatial attention with the bottleneck attention module (BAM). It focused on the refinement of the intermediate features of the CNN and provided better results than the other related frameworks.

Xiang Li et al. [18] developed a selective kernel networks (SkNet) model to analyze the attention process in the convolution layer. It assigned weights dynamically with different scaled convolutional kernels. This approach has shown the value of receptive fields in the recognition of images.

An attention-based network consisting of detail, subject, and channel attention modules for the enhancement of features was proposed and validated on two public datasets that achieved remarkable results compared to the baseline models [19].

The literature shows deep learning networks and traditional feature descriptors have achieved good gesture recognition results. However, most existing methods have focused only on the global image features. They have yet to consider the importance of local image features. Also, some deep learning methods are memory-intensive and computationally extensive.

We have proposed a framework using a feature extraction module involving feature fusion of classical HOG features with attention-based CNN features to recognize static sign gestures using global and spatial dependencies. The main advantage of using CBAM-based CNN with channel and spatial attention modules is focusing on potential hand gesture features.

3. Methodology

This section discusses the overall details of the datasets and the architecture of the implemented methodology with a detailed description of the significant steps to recognize static sign language gestures in the developed framework shown in Figure 1.

3.1. Dataset

The benchmark dataset, the Massey University dataset [20], consists of 2515 images of 5 individuals belonging to 36 sign classes. The images in the dataset have few changes in rotation, scale, and lighting captured from five different directions with variations in hand size with the camera. Figure 2 shows a sample of the sign gesture images taken from this image database. The static ISL dataset [21] comprises 36 sign gestures of digits (0–9) and alphabet (A–Z), with more than 1200 sample images in each class captured using a web camera. There is a total of 34,554 images in the database having a size of 250 × 250 with augmentation to introduce variability in the data. Figure 3 shows the sample images from the ISL dataset.

3.2. Pre-Processing

The images are first pre-processed after acquisition to prepare the images for feature extraction. The input image is scaled down to 128 × 128 to minimize the computational complexity [22]. All the input images are normalized by dividing the pixel intensities of each image by 255.

3.3. Feature Extraction Module

The process of learning meaningful features is quite challenging in image processing. Various features like edges, corners, blobs, etc., can be extracted from an image that gives a different representation and identification of an image to the computers for understanding and learning. Many features lack robustness when subjected to rotation, scaling, and illumination variations. In the proposed structure, the handcrafted HOG-based feature vector and attention-based CNN features are used [23].

3.3.1. HOG Feature Descriptor

For image data, there is a whole branch of feature engineering that deals with the extraction of features from images. HOG computes pixel-wise gradients and orientations and plots them on a histogram. The main motive is to simplify the representation of the image by extracting the most useful information from the image and removing the redundant parts. In this way, it helps in minimizing the noise. This technique detects objects by calculating feature descriptors using horizontal and vertical gradients. The image splits into 8 × 8 cells, with each cell pixel having direction and magnitude that helps develop a compact representation that can be quickly learned by computer [24]. This method computes the gradient of each pixel in x (horizontal) and y (vertical) directions since the image is a discrete function. It is a vector having direction and magnitude given in Equations (1) and (2).

$$\theta =arctan\left(\frac{f_x}{f_y}\right)$$

(1)

$$mag=\sqrt{\left(f_x^2+f_y^2\right)}$$

(2)

A 9-bin histogram that ranges from 0–180 degrees quantizes the orientation of the gradient with values in the range of (0–255). All histograms are merged to produce a feature vector H. The final HOG feature vector is given by Equation (3), which provides shape information of regions or points within an image [24].

$$F_{HOG}=\left\{f_{h1}, f_{h2},f_{h3},\dots ..f_{hn}\right\}$$

(3)

We applied a Laplacian filter to sharpen the edges of the hand gestures and reduce noise. The intensity values of the images are rescaled to improve the image’s contrast. This model extracted 8100 features using the HOG method from each of the input images of the dataset. Figure 4 depicts the input image and the extracted HOG image.

3.3.2. CBAM Attention-Based CNN Feature Extraction

In our work, we introduced a CBAM attention-based module within CNN so that the model can concentrate on particular features that carry essential information about gestures. The architecture of the CBAM module used is shown in Figure 5. This module proposed by Woo et al. [25] aims to efficiently enhance feature extraction and representation. We can represent the intermediate feature map as $F\in {ℝ}^{C\times H\times B}$ where, $C\times H\times B$ is the input feature map resolution. In the channel attention block, the channel carrying meaningful information is decided based on inter-channel relationships. Two pooling functions, global average pooling (GAP) and max pooling, individually squeeze out the spatially extracted features from the intermediate feature vector. Then, these features are fed to a multilayer perceptron (MLP) and merged. Then, the sigmoid function activates the sigmoid feature map. The channel attention module gives a 1D attention map written as $M_c\in$ ${ℝ}^{C\times 1\times 1}$. Then, we will obtain a refined input feature $F^{\prime }$ by multiplying this attention map with the original output. Mathematically, the process of the channel attention module is shown in the following equations.

$$M_c\left(F\right)=f_{sigmoid}\left(MLP\left(GlobalAvgPool\left(F\right)\right)+MLP\left(MaxPool\left(F\right)\right)\right)$$

(4)

$$F^{\prime }=M_c\left(F\right)\otimes F$$

(5)

The spatial attention block concentrates on the inter-spatial relation of the input feature map and it is fed with this refined feature vector and two intermediate feature maps are generated using GAP and max pooling that are concatenated.

Figure 5. Architecture of CBAM mechanism.

Figure 5. Architecture of CBAM mechanism.

The fused feature map is passed through a 7 × 7 kernel and sigmoid function to generate a spatial feature map with attention weight $M_S\in$ ${ℝ}^{1\times H\times w}$ and results in a more efficient feature representation by multiplying the spatial feature map with the refined input as given in Equations (6) and (7).

$$M_s\left(F\right)=f_{sigmoid}\left(Conv\left(\begin{array}{c}[GlobalAvgPool\left(F\right);\\ MaxPool\left(F\right)]\end{array}\right)\right)$$

(6)

$$F^{″}=M_s\left(F^{\prime }\right)\otimes F^{\prime }$$

(7)

where the symbol “⊗” represents elementwise multiplication, and $F^{″}$ is the final refined feature map [26,27].

In this study, we employed a CBAM attention-based module that consists of three CNN convolution blocks to generate attention-based feature maps from the input dataset. Figure 6 demonstrates the flowchart of the CBAM-CNN model. The convolutional layer is the initial layer in each block, and each convolution layer generates features and gives a feature map. Then, we call the CBAM attention module to extract channel and spatial data. Each convolution block has batch normalization and max pooling layers. Also, we used equal padding and the rectified linear unit (ReLU) activation function within each block. We used 32, 64, and 128 filters in the CNN model’s first, second, and third stages. Finally, the model results in a 1D feature vector after flattening the 3D feature map obtained after the last layer of max pooling. The flattened 1D vector that contains information on image-based features extracted from convolutional layers is expressed by Equation (8).

$$F_{CNN }=\left\{f_{CNN1}, f_{CNN2}, f_{CNN3},\dots \dots ..f_{CNNm}\right\}$$

(8)

where m is the number of features. Then, the CNN features are fused with the HOG-based feature vector to obtain the training feature vector by Equation (9).

$$F_{concat}=F_{CNN }\oplus F_{HOG }$$

(9)

There are two fully connected dense layers in the network. The first layer, with 300 neurons, takes the combined feature vector as input and performs a linear transformation, and the second layer, with 150 neurons, further processes the previous layer’s output. We have used the L2 regularization technique with hyperparameter λ = 0.0001 for each dense layer to minimize the network complexity and loss while training the model.

Then, a dropout layer is applied to the output with a 0.5 dropout value to prevent overfitting during the model’s training. Finally, the model’s output layer has 36 units and a Softmax activation function. It produces probabilities for each of the 36 possible classes. Algorithm 1 summarizes the overall working of the proposed network.

Algorithm 1: Proposed Algorithm for recognition of sign language gestures

Input: Image Dataset with size (128 × 128 × 3) Output: Sign language Gestures //HOG feature extraction $F_{HOG}$ Step 1: Convert each image from the training dataset into grayscale images. Step 2: For each pixel of the image gradient is calculated ($f_x, f_y)$. Step 3: Divide input images into 8 × 8 cells, and the corresponding histogram of each cell is computed. Step 4: Merge all histograms to obtain the HOG feature vector. Output: HOG feature vector $F_{HOG}$ of dimension 1 × 8100. //Attention-based CNN feature extraction $F_{CNN}.$ Step 1: CNN layers are defined including convolution, normalization, CBAM, and max pooling. Step 2: Flatten the last max pooling layer CNN output to 1-dimensional feature vector $F_{CNN}$. Step 3: Concatenate the flattened CNN feature with HOG feature $F_{Concat}=\left[F_{CNN}, F_{HOG}\right].$  Step 4: The fully connected layer takes the merged feature as input. Step 5: The final output layer classifies 36 sign gestures with a Softmax activation function.

4. Results and Discussion

We used the deep learning TensorFlow framework in the Python 3.7 programming platform with a Windows 10 Pro operating system, Intel(R) Core (TM) with i7 [email protected] GHz, and Nvidia GTX 1060 with a 24G memory graphics card to implement the model architecture and perform the experiments on two datasets. We have used TensorFlow, OpenCV, Pandas, Numpy, and Matplotlib Python libraries for computational graphics, image processing, and mathematical processing of data and visualization of various parameters [5]. We used an Adam optimizer with Nesterov momentum called Nesterov-accelerated adaptive moment (Nadam) estimation [28] with an initial learning rate of 0.001 to improve high fluctuations in the loss and enhance the cost function’s convergence process. We tuned the hyperparameters to achieve good training of the model.

The details of hyperparameters used while training the model to recognize 36 classes are shown in

Table 1. Hyperparameter Details.

Parameters	Values
Input size	128 × 128
Initial Learning Rate	0.001
Optimizer	Nadam
Batch Size	32
Cost Function	Categorical Cross-entropy
Epoch	30
Beta 1	0.9
Beta 2	0.999
Epsilon	1 × 10−8

.

We performed five-fold cross-validation to achieve unbiased results and avoid overfitting while training. This method divides the dataset into five equal folds, in which a fold is unseen data for testing. The remaining folds are for training. The number of samples for each class in the test set for each fold will vary due to randomness in the process of k-fold cross-validation. We assessed our proposed architecture’s efficiency using accuracy, precision, recall, and F1-score for individual folds. Then, we average these metrics across all the folds to assess your model’s performance. This averaging process helps smooth out the effects of sample size variation and gives a more reliable estimate of your model’s performance.

This variation is due to the random partitioning of our dataset into folds. The precision is the ratio of the total count of precisely identified positive observations to the total number of values classified as positive. Recall is the fraction of accurate predictions to the overall correct classes. The F1-score is a widely used metric to measure a model’s efficiency, also known as the balanced mean of the recall and precision metrics [29]. The mean accuracy of all folds is calculated for both datasets, as shown in

Table 2. The average recognition rate for both datasets.

Folds	Dataset 1	Dataset 2
1	99.10	99.69
2	99.22	99.77
3	99.24	99.79
4	99.22	99.81
5	99.30	99.83
Average Accuracy (%)	99.22	99.77

. We computed the precision, recall, and F1-score using the following Equations (10)–(12).

$$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+ \mathrm{FP} }$$

(10)

$$\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}}$$

(11)

$$\mathrm{F}1-\mathrm{score}=2\ast \frac{\mathrm{precision}\ast \mathrm{recall}}{\mathrm{precision}+ \mathrm{recall}}$$

(12)

where “TP”, “FP”, and “FN” are true positives, false positives, and false negatives, respectively. Figure 7 and Figure 8 represent the values of recall, precision, F1-score, and recognition accuracy of our proposed model on both datasets for fold two. Figure 9 shows the accuracy and loss graphs of both datasets for training and validation data.

It is observed from Figure 9 that the loss is reduced after each iteration and reaches a minimum value, and the accuracy metric is attaining the maximum value. In the ASL dataset, the accuracy and loss stabilized after the 25th epoch. In the same way, both metrics stabilize after the 10th epoch for the ISL database. We used categorical cross entropy as a cost function to analyze loss metrics for multiclass classification. Furthermore, we presented the confusion matrix used to predict correct sign classes and give misclassified samples. It provides accuracy of each sign gesture in terms of percentage. The resulting confusion matrix of both datasets is presented in Figure 10 and Figure 11, respectively.

The confusion matrix signifies the actual and the predicted classes in the x and y axes, respectively. The diagonal values in the confusion matrices show the actual recall values of all the signs, and the consistent rows signify the confused classes. Some classes in the ASL dataset have similar gesture signs, such as “0” is misclassified with 10.5% as “O” due to similar hand gestures with little variations in the position of hands, and “V” is misclassified with 14.3% as “2”. The reason behind this misclassification is the similarity in sign gestures with only a change in the position of the thumb in signs “2” and “V”. As shown in the confusion matrix of the ISL dataset in Figure 9, the class label “8” obtained a minimum recognition rate of 92.3% due to misclassification with 2.6% as “4” and 5.1% as “9” due to similarity in sign gesture with a small change in the number of fingers in case of “9” sign gesture and little change in both finger and thumb position in case of “4” sign. The numeral class “2” was misclassified with “3” by 4.7% and recognized with 95% accuracy as it had similar hand gestures but differences in thumb placement with finger and finger orientation. Overall, our model predicted the class labels accurately for both datasets. The diagonal values in the confusion matrices show the actual recall values of all the signs, and the consistent rows signify the confused classes.

4.1. Comparison with Previous Studies

We compared the obtained results with other related models implemented based on accuracy, model parameters, training epochs, inference time, and time complexity.

Table 3. State-of-the-art comparison of Massey ASL and ISL dataset.

Data	Year	Method (Year)	Accuracy (%)	Training Parameters	Training Epochs	Inference Time (s)
	[30]	2-level ResNet-50 (2020)	99.03	25,636,712	200	0.025
Massey ASL	[31]	VGG16+Attention (2022)	98.02	138.35 M	30	-
	[32]	Canny + HOG+ KNN (2022)	97.6	-	-	0.39
	[33]	Spatial feature learning network (2023)	80.44	3.8 M	-	2.23
	Proposed	HOG+ CBAM-based CNN	99.22%	2.3 M	30	0.023 s
Static ISL	[34]	Vision Transformer based approach (2023)	99.29%	7 M	5	-
	[21]	Ensemble (ResNet50 +Attention) (2022)	98.20%	23.5 M	50	-
	Proposed	HOG+ CBAM-based CNN	99.79%	2.3 M	30	0.023 s

shows that our proposed model has 2.3 million parameters, less than the existing networks. Significantly, the acquired recognition rate of the Massey ASL dataset is 99.22%, and the accuracy of the ISL dataset is 99.79%, which is higher than the previous studies with fewer training epochs and an excellent average inference time on testing images.

4.2. Time Complexity and Order of the Proposed Method

We can determine the efficiency of the proposed algorithm using time complexity. The complexity is computed for each operation involved in the algorithm using Big-Oh notation. This technique measures the maximum amount of time taken by the algorithm. In HOG feature extraction, the calculations are performed on each cell and block for each image. It involves various stages of operations having the complexity of O ($p^2$), where p is the number of pixels of an image. The algorithm consists of computing gradients and orientations for each pixel, which requires iterating over all pixels in the image. For CBAM-based CNN, the time complexity is O $\left({{\displaystyle \sum }}_{i=1}^L{S}_L^2\ast K_L^2\ast C_{L-1}\ast C_L\right)$, where S, K, and C are the dimension of the feature maps, convolutional kernel size, and channel count for each convolutional layer L. The batch normalization layer, max pooling layer, and CBAM module layer have a time complexity of O (1) hand and have minor operations that do not depend on the size of the input feature. The flattening layer has an O (n ∗ m) complexity. The concatenation layer is a linear operation with an O (n) complexity. The fully connected layer has the complexity of $O\left({{\displaystyle \sum }}_{i=1}^f{F}_f\right)$, where $F_f$ is the number of neurons and f is the number of fully connected layers. The last classification layer has complexity O (k $\ast$ h), where k is the total sign classes and h is the neuron count in the hidden layer [35]. So, we can express the overall complexity as

$$O \left(p^2\right)+O ({{\displaystyle \sum }}_{i=1}^L{S}_L^2\ast K_L^2 \ast C_{L-1}\ast C_L+{{\displaystyle \sum }}_{i=1}^f{F}_f) +O(k\ast h)$$

(13)

We have discussed the comparison of time complexity with other algorithms. The time complexity of two-level ResNet50 [30] is 2 $\ast$ O (${{\displaystyle \sum }}_{i=1}^L{S}_L^2\ast K_L^2 \ast C_{L-1}\ast C_L$ + ${{\displaystyle \sum }}_{i=1}^f{F}_f$)+ O(k $\ast$ h) since there are two branches of the CNN model for feature extraction for each layer of the model. In [31], the authors have used fused features of pre-trained VGG16 and attention-based VGG16.

The model has a computational complexity of 2$\ast$O (${{\displaystyle \sum }}_{i=1}^L{S}_L^2\ast K_L^2 \ast C_{L-1}\ast C_L$ + ${{\displaystyle \sum }}_{i=1}^f{F}_f$) +O(k $\ast$ h). The study [32] involves canny edge detection, HOG feature extraction, and KNN for classification, which has an overall time complexity of O (m $\ast$ n log m $\ast$ n) + O ($n^2$) + O(d $\ast$ F), where m $\ast$ n is the image size, d, and F is the number of data points and features, respectively. In [34], authors have used a vision transformer encoder with a time complexity of O (8 $\ast$ $s^2$ $\ast$d) + O (8 $\ast$ s $\ast$ $d^2$), where s is the sequence length and d is the depth.

It is observed from the analysis that deep learning models used in [30,31] have more time complexity, training epochs, and model parameters as compared to the proposed model. The author used handcrafted features with KNN as a classifier, which is less computationally expensive than the proposed algorithm. Still, the inference time of prediction is longer, and our methodology has a higher accuracy rate. The spatial feature learning network in the paper [33] has implemented a lightweight network by merging features from two convolutional layers with multiscale filters. This model has 3.8 million trainable parameters and an average inference time of 2.23 s, which is comparatively more than the proposed model. The vision transformer in [34] used eight layers of the encoder with four heads with seven million parameters, which is computationally expensive. The authors in [21] have used ensemble network architecture employing ResNet50 with an attention module with more training parameters and epochs.

The proposed method performs superiorly to the other existing methods on all benchmark datasets due to the following reasons: (i) The model combines the handcrafted features from the HOG feature descriptor with the CBAM attention-based CNN features that gather meaningful information from two different streams for better representation of gesture images. (ii) The model acquired good recognition accuracy with fewer model parameters and built an appropriate balance between accuracy and efficiency. (iii) The framework recognized all the sign gestures accurately with two misclassifications in both datasets, which is comparatively less compared to other existing methods.

5. Conclusions

This work aimed to implement a framework combining CNN with the CBAM module and traditional HOG features to recognize static sign language gestures efficiently. HOG features provide extensive details about hand shape and orientation. The CBAM module employed spatial- and channel-based attention mechanisms within each convolutional layer of the CNN to extract critical and relevant information about images. In our experiment, we analyzed our methodology on two datasets that achieved accuracies of 99.22% and 99.79%, respectively. The model outperformed the state-of-the-art techniques with the least possible parameters and is computationally efficient. We can extend this work to recognize dynamic hand gestures against complex backgrounds for real-time applications and effective recognition of sign language gestures.