Korean Sign Language Recognition Using Transformer-Based Deep Neural Network

Abstract

Sign language recognition (SLR) is one of the crucial applications of the hand gesture recognition and computer vision research domain. There are many researchers who have been working to develop a hand gesture-based SLR application for English, Turkey, Arabic, and other sign languages. However, few studies have been conducted on Korean sign language classification because few KSL datasets are publicly available. In addition, the existing Korean sign language recognition work still faces challenges in being conducted efficiently because light illumination and background complexity are the major problems in this field. In the last decade, researchers successfully applied a vision-based transformer for recognizing sign language by extracting long-range dependency within the image. Moreover, there is a significant gap between the CNN and transformer in terms of the performance and efficiency of the model. In addition, we have not found a combination of CNN and transformer-based Korean sign language recognition models yet. To overcome the challenges, we proposed a convolution and transformer-based multi-branch network aiming to take advantage of the long-range dependencies computation of the transformer and local feature calculation of the CNN for sign language recognition. We extracted initial features with the grained model and then parallelly extracted features from the transformer and CNN. After concatenating the local and long-range dependencies features, a new classification module was applied for the classification. We evaluated the proposed model with a KSL benchmark dataset and our lab dataset, where our model achieved 89.00% accuracy for 77 label KSL dataset and 98.30% accuracy for the lab dataset. The higher performance proves that the proposed model can achieve a generalized property with considerably less computational cost.

1. Introduction

Communication is inevitable for the deaf or hard of hearing (DHH) community to pass their thoughts, requirements and basic needs to others. In total, 450 million people worldwide belong to the DHH community; they use sign language for communication [1]. According to the DHH community, sign language is not easy and is entirely different from other letters, words and sentences. For example, English and American sign language (ASL) are completely different from Bangla sign language (BSL) and Bangla; the same is true for Korean sign language (KSL) and Korean. In addition, sign language differs from country to country and is even based from state to state. Despite being the same language, sign language is different from American sign language. British sign language is not the same because it is not a motion representation like spoken language. The overall situation is unsuitable for the DHH communities’ people to learn sign language, and they often face difficulties in communication due to this variation. Because of this difficulty, there are many obstacles for the DHH community to access their daily requirements, such as public information, social engagement, medical healthcare, education, etc. To ease their lives, we have been lowering the language barrier between the DHH and hearing people community.Many researchers have been working to develop automatic sign language recognition systems with advanced computer vision, robotics and natural language processing to ease their life [2,3,4,5,6,7,8,9,10,11,12,13]. Some researchers worked to direct a translation system from a sign language into a spoken language [14,15]. Sensor-based [16] and vision-based approaches [17] are the most common domains for developing automated sign language systems [18]. In the last decade, CNNs have made an extraordinary contribution to the vision-based sign language recognition approaches to recording one-handed and two-handed statistical or dynamical movement [4,5,19,20,21,22]. In the meantime, self-attention-based architectures have shown an excellent capability of capturing long-distance relationships and have become the de facto most popular approach in natural language processing (NLP) [23]. Inspired by NLP successes, multiple works combine CNN-like architectures with self-attention [24,25], some replacing the convolutions entirely [26,27]. Besides the NLP domain, the transformer model has been applied in different research domains for segmentation, detection, and classification by following the success in NLP [28]. In many works, the CNN architecture is completely replaced by the transformer. Recently, many researchers have proposed a transformer-based method to recognize sign language and classification [29]. For the image processing task, a pure transformer (Vit) is used to replace the CNN backbone [30]. The main concept of transformer-based image processing is firstly to split the original image into a discrete non-overlapping patch by following the word token of the NLP. Then the extracted patches are fed into the transformer for extracting the global relation and feature for the image classification. Inspiration by ViT, many methods have been developed based on the same concept, such as IPT and SETR [31,32,33]. The technique of the ViT is good for the image-based classification because researchers can directly apply the transformer to the sequence of the patches the same as the NLP sequence. However, some basic themes exist between the image-based vision tasks and the NLP sequence. Moreover, due to the fixed size of the patch, it is difficult to extract multi-scale feature maps and low-resolution features. High computational complexity is a well-known property of the transformer as well. To overcome the problem, researchers recently sequentially combined the transformer feature and the convolution layer, focusing on the computer vision-based task instead of the NLP sequence system CNN meet transformer (CMT) [28]. They employed four stages of the transformer and the CNN, where they mainly considered stage-wise design for extracting multi-scale features, reducing the resolution and increasing the dimension flexibility. However, the CMT method plays a crucial role in overcoming the problem of the pure transformer in the image processing task. However, their main concern is they combined CNN with a transformer sequentially and used it four times in four stages, increasing the time and computational complexity exponentially. In addition, few researchers have extracted CNN and transformer-based features in parallel concepts, and no work has been performed for Korean sign language recognition for the deaf and mute communities. To overcome the challenges, we proposed a convolutional layer-based transformer and CNN-based multi-branch model to recognize sign language recognition to overcome existing challenges. In the method, we parallelly extracted CNN and transformer-based features from the sign image and fed the concatenated feature into a classification module for classifying sign language. To overcome existing patches sequence problems [33], we employed a grain architecture module that firstly employed two-three × three convolutions with a stride, then included a three × three convolution with a stride of 1 and an output channel of 32 to reduce the size of input images for extra tracking and better local information extraction. The grain module mainly consists of a convolution and a layer normalization (LN), which is employed before the first stage to reduce the size of the intermediate feature (2× down-sampling of resolution) and project it to a larger dimension (2× enlargement of dimension). The convolutional layer-based transformer can extract short-range and long-range dependencies. Lastly, we employed a classification module with a global average pooling layer, a fully connected (FC) layer and an n-way classification layer with softmax. In addition, we added position embedding, fed into the attention-based deep learning model. The main contribution of the study is given below.

• We present a deep learning adaptable Korean sign language (KSL) video dataset.
• We proposed a multi-branch attention and CNN-based neural network to recognize sign language, which followed two stages to generate feature maps of different stages. The first stage included two parallel branches to generate feature maps of different scales, which are important for the dense prediction task. The second stage is considered a classification module.
• A grain module was employed to reduce the size of the original image and solve the existing patched processing problems. Then, lightweight and CNN features were concatenated in the first stage and fed into the classification module in the second stage.
• We used KSL video and 3 BSL datasets to evaluate the model and achieved 88.00% accuracy for KSL and 96.88% for our proposed dataset.

The presented work is organized as follows: Section 2 summarizes the existing research work and problems related to the presented work, Section 3 describes the benchmark and proposed Korean sign language datasets, and Section 4 describes the architecture of the proposed system. Section 5 shows evaluation of the performance. In Section 6, we draw the conclusion and future work.

2. Related Work

Sign language classification, continuous sign language recognition (SLR), and sign language translation are the main subdomains of the SLR. Many researchers have been working to develop sign language recognition systems for various languages with several machine learning and deep learning algorithms [34,35]. As different countries have their own sign language based on their mainstream language, researchers have been working to develop a sign language recognizer for individual sign languages, among which the Bengali sign language recognizer was developed with the machine learning and deep learning algorithm [4,5,36,37,38]. Pitsikalis et al. applied the hidden Markov model to classify the linguistic labeling sub-unit sign. They collected over 961 images with the Kinect TM device and performed well [39]. Additionally, the HMMs model can learn non-discriminatory features, but sometimes it ignores the data coming from the alternate class. In addition, it cannot sell useful features with it. Ong et al. proposed a multi-class sign language classification model using the sequential pattern trees, namely the SP-tree boosting algorithm, where they mainly extracted hand trajectory features from the subunit of the image [40]. To evaluate their model, they used Greek sign language (GSL), which achieved 93.00% accuracy and German sign language, where they achieved 88.00% accuracy, which is better than the hidden Markov model. Almeida et al. proposed a phonological structure based on decomposition and feature extraction with the RGB-D sign language image. After applying a support vector machine (SVM) as a classifier, they achieved 80% accuracy [41]. Fatimi et al. proposed an ASL recognition model using an artificial neural network (ANN) and SVM, where they achieved higher performance accuracy with ANN compared to the HMM and SVM [42]. In addition, 98.2% accuracy was achieved with SVM for the five wearable sign language devices [43], the fuzzy network was used for Chinese sign language (CSL) [44] and KNN, LDA, and SVM were applied for the ASL for 0-9, which achieved 98% higher accuracy [45]. Moreover, some other machine learning-based algorithms were employed to sign language recognition and achieve satisfactory accuracy, such as the multi-stream hidden Markov model [46], hidden conditional random field (HCRF), and random decision forest (RDF) [47,48]. Computational complexity, the inefficiency of selecting potential images and many other drawbacks are faced by researchers in the general machine learning algorithm for image-based sign language recognition. To overcome the problems, researchers used an ANN-based algorithm for sign language recognition [49,50]. In addition, Kim et al. applied ensemble ANN to classify Korean sign language. Based on the 10 labels and 1500 samples, they achieved 97.4% accuracy by considering only finger spelling signs. Camgoz et al. employed a model for making a dataset for KSA [51], then Ko et al. employed translation modeling for exploiting 2D human pose key points and released a KSL dataset [52]. Recently, the computer vision domain has largely switched toward the deep learning-based model due to the performance and various complexity-related issues. Al-Hammadi et al. proposed a single and fusion parallel 3-dimensional convolutional neural network (3DCNN) to recognize three gesture datasets where they achieved 84.38%, 34.9%, and 70% for 40, 23, and 10 class datasets, respectively, for the signer-independent and +10% for the signer-dependent cases [53]. Their performance accuracy is higher than the same previous method. Sincan et al. [37] proposed CNN and the long short-term memory (LSTM) algorithm based on the feature pooling module (FPM) and attention feature, where they evaluated Turkish sign language and achieved good performance. They mainly used an attention module to achieve convergence points as quickly as possible. Yuan et al. proposed a deep convolutional neural network (DCNN) and LSTM to recognize ASL and CSL sign language. The advantage of their model is that they overcome the gradient vanishing and overfitting problem, whereas their model produces long-distance dependency problems in the future feature network [54]. Aly et al. proposed a deep bi-direction LSTM (BiLSTM) to recognize Arabic sign language using a self-organizing map for hand shape feature extraction [55]. The same 2DCNN, 3DCNN, and LSTM models were used to classify skeleton-based sign language recognition [56]. More recently, researchers used different existing CNN architectures for solving specific problems, such as CNN with attention [57,58,59], AlexNet [21,60] and VGGNet [61], which focus on the convolution and pooling layer, multiple paths of the basic block effectiveness showed by GoogleNet [62] and InceptionNet [63]. ResNet [22] showed better generalization by adding shortcut connections every two layers to the base network. Researchers used an attention module to overcome the existing problem as an operator between adaptable modalities [64]. Stack attention modules used as an intermediate stage between residual networks [65], SENet [24], and GENet [66] were used to analyze the interdependencies among the channels. NLNet employs neural networks as a self-attention mechanism for providing parts interconnection among the spatial position to argue the long-range dependencies [67]. Moreover, MobileNets [68] and EfficientNets [69] were recently used to provide mobile-size portable networks. Transformers are used to achieve remarkably advanced success in natural language processing tasks, and many researchers employed them as effective articles for vision tasks [21,22,23,24,26,27]. The main transformer architecture ViT [30] is directly inherited from NLP to the computer vision domain to recognize sign language classification, although it needs a large-scale dataset to achieve satisfactory performance accuracy [70]. To overcome the large dataset problem, another researcher proposed DeiT to introduce a new training procedure with higher efficiency [33]. Another researcher proposed another extended transformer, T2T-ViT [71], to convert recursively neighboring token aggregation into single tokens. Vit used only the patch sequence, which may cause the loss of some potential information. By including pixel-level information with the patch level, another researcher proposed a TNT transformer with inner and outer blocks, respectively. PVT, CPVT, and CvT combined the transformer and CNN to overcome the long-range dependency problem, but their combination utilization was not satisfactory [72,73,74,75]. Another researcher employed the CMT method to solve the combination inefficiency problem, which included four stages, a transformer, and the CNN [28]. The main drawback of the CMT is the mix-up of long-term and short dependency in each stage, which increases the computational complexity. Moreover, many transformer-based deep learning models have been developed but have not achieved satisfactory performance. We proposed a simple multi-branch transformer-based architecture to recognize sign language datasets to solve the problem. It is known that for modeling long-range dependency, transformer demonstrated higher efficiency. Moreover, in the paper, we concatenated the transformer-based feature and convolution layers feature, where the overall performance proved the efficiency of the model.

3. Dataset

According to the researchers, a few datasets are available for Korean sign language (KSL) [76]. There is only one dataset we found online for working here: the KSL dataset. We created a new dataset to overcome the lack of a dataset. The KSL dataset is described in Section 3.1, and the proposed dataset is in Section 3.2.

3.1. KSL-Dataset

KSL is the first vision-based dataset for the Korean sign language, where they collected 77 words from 20 people. Although there are many words in the Korean language for daily activities, they tried to cover the most usable words that Koreans use in real life. They included 17 locations for the 20 signers, and they collected facial expressions with hand movement. Moreover, they considered different angles and distances for collecting actual videos in accounting for the real-life situation. They collected a total of 1229 videos and 112,564 frames as well. They extracted frames from the video by considering 30 ps, and to avoid noise and empty information, they also discarded some initial and end images [76]. Figure 1 shows the sample sign of this dataset.

3.2. Proposed Dataset

As there are very few datasets available online for Korean sign language, we created an independent Korean sign language dataset using the 20 most significant words in the study. The selected Korean sign language words are thanks, okay, love, happy, sorry, no, hello, late, meet, shame, effort, regrettable, give, yes, help me, welcome, by, who, what, who, and why [75,76,77]. Also, the author selected 77 words that Korean people use to perform daily activities, but all 77 are not the most usable. Some of them are the most important. We selected the 5 most usable words from the 77 words [75]. The five words are as follows: no, who, what, thanks, and sorry. The photographs of each word are shown in Figure 2. Using the webcam, we collected 20 words in the KSL dataset. Twenty-one people participated in collecting the videos. Sample photographs of each significant word gesture are shown in Figure 2. We collected a 4-second video or 120 frames for each action from a person, and 20 people willingly participated in the data collection procedure. In addition, the background of the video appears in different scenarios as much as possible, such as a natural background.

4. Proposed Methodology

In the study, we proposed a convolutional layer-based transformer and CNN as a multi-branch hybrid network to include the transformer and convolutional neural network (CNN) feature by following the architecture of CMT [28], ResNet-50 [22], and DeiT [33,78]. Most of the existing transformer-based systems directly split the input image into multiple patches, and the main drawback of the patch system is that it can be poorly modeled with a linear projection. In the study, though we followed the patches concept, we employed a new architecture, grain architecture, to implement the patches idea instead of splitting the image. In the grain architecture, firstly, we included two 3 × 3 convolutions with stride one. Then we used a 3 × 3 convolution layer with a 2-stride layer where 32 was used as an output channel for reducing the size of the image to establish the patch concept of the transformer and modern CNNs [5,22]. Our model has three stages—initial grain, feature extraction, and classification module, shown in Figure 3a. The initial stage is used to perform the patches mechanism; the second stage is a feature extraction consisting of two parallel branches for generating different scales of feature maps to achieve dense prediction tasks. Finally, the stage is considered a classification module. After concatenating the two branches of features, we produced the final features fed into the classification module. In the convolutional layer-based transformer, we included an initial module, convolutional layer, layer normalization and lightweight multi-head attention layer and MLP convolution. In the initial module, we combined a convolutional layer with a normalization layer to produce the hierarchical representation of the grained features [5,33]. Convolutional layer-based transformer extracted short-range and long-range dependencies among the pixels. Lastly, we employed a classification module, including some convolutional layer with GELU activation and then a global average pooling layer, a fully connected (FC) layer and an n-way classification layer with a SoftMax activation function. Algorithm 1 shows the overall working flow where we included every step we followed in our work.

Algorithm 1 Pseudocode of the proposed system.

Input: Set of Input Dataset $P_i\in P\left(n\right)$ Number of Samples: N, 70% for Training and 30% for Test Output: Set of vector $s_i$ define Model(input=InpuLayer, outputs=ClassificationLayer): $GrainFeature\leftarrow GrainModule\left(D\right)$ $FirstBranchFeature\leftarrow ConvolutionalLayerBasedTransformer\left(GrainFeature\right)$ $SecondBranchFeature\leftarrow CNN\left(GrainFeature\right)$ $FinalFeature\leftarrow Concatenate(FirstBranchFeature,SecondBranch-Feature)$ $PredictedClass\leftarrow Classification-Module(Final-Feature)$ return PredictedClass while$i\ne NumEpochs$do      // For Training      while$Batch\ne NumberBatchTraining$do $PredictedClass\leftarrow Model\left(Batch\right)$ $Loss\leftarrow Criterion(PredictedClass,Trai{n}_{C}lass)$ $Updatetheloss\leftarrow Loss.backward\left(\right),Optimizer.Step\left(\right)$           // For Testing      while$Batch\ne NumberBatchTesting$do $PredictedClass\leftarrow Model\left(Batch\right)$ $Output\leftarrow CPerformanceMatrix(PredictedClass,TestClass)$

4.1. Grain Module

First, we fed the original image into the grain module for fine-grained initial feature extraction—this module we designed by following modern CNNs (e.g., ResNet [33]). We divided this module into two blocks. The first block includes 3 × 3 convolutions with a stride of two and an output channel of thirty-two to reduce the size of input images, followed by another two 3 × 3 convolutions layer by considering one as a stride value. The second stage performs the patch aggregation approach by including the convolution layer and layer normalization.

4.2. Convolutional Layer-Based Transformer

The convolutional layer-based transformer block consists of an initial module, a lightweight multi-head self-attention (LMHSA) module and MLP convolution as described below.

4.2.1. Initialization Module

We applied the initial module to extract local information from the dataset as position encoding, also known as local perception unit (LPU) [25]. One of the main goals of this module is to consider rotation and shift augmentation operation, which are two of the most important manners in the visual task, and these should not alter the rest of the system. In other words, to overcome the image translation dependency on the system [5,73,79], researchers used absolute positional encoding in previously developed transformers to initially leverage the order of tokens. The main concept adds unique position encoding to individual patches [73], but they ignore the local relation [80] and the structure information inside the patch. To overcome the problem we proposed here, which can be defined as the following equations,

$$IM\left(X\right)=EWConv\left(X\right)+X$$

(1)

where the initial module features denote IM, $EWConv(.)$ represents the element-wise convolution. $X\in R^{H\times W\times d}$, where the resolution is represented by $H\times W$, and the dimension of the feature is represented by d.

4.2.2. Lightweight Multihead Self Attention

Self-attention (SA) is a popular, effective model in the neural network [23]. Generally, the input of the self-attention module can be written as $X\in R^{n\times d}$, which is transformed into three matrices, namely query, key and value defined by $Q\in R^{n\times d}$, $K\in R^{n\times d}$, and $V\in R^{n\times d}$ respectively. Here, several patches are represented by $n=H\times W$. The dimension of the input, key and value can be written as d, $d_k$ and $d_v$, respectively. The self-attention module can be written as the following Equation (2):

$$SA=softmax\left(\frac{q{k}^T}{\sqrt{d_k}}\right)\times v$$

(2)

Here, SA is the self-attention, and q, k, v, and $d_k$ are the query, key, value and key dimension. To make self-attention lighter, we employed an element-wise convolutional neural network with stride k to reduce the dimension of key and value before the attention model [28]. Besides this, relative position bias B was added in each of the self-attention modules [81,82]. We can rewrite Equation (2) for the lightweight self-attention (LSA) as follows:

$$LSA=softmax(\frac{q{\overline{k}}^T}{\sqrt{d_k}}+B)\times \overline{v}$$

(3)

Here, LSA is the lightweight self-attention, and q, k, v, and B are the query, key, value, key dimension and relative position. B can be written as $B\in R^{n\times \frac{n}{k^2}}$, which is randomly initialized and learnable. The relative position B can be transferred into other dimensions $\overline{B}$ in bicubic interpolation with a different size $m_1\times m_2$. The transformation equation can be written as $\overline{B}=Bicubic\left(B\right)$. This procedure is excreted for one head, the same way it will be completed for h heads, defined as lightweight multi-head self-attention (LMHSA). An h number of lightweight attention modules are employed to produce attention features. Each individual head outputs a sequence of size $n\times \frac{d}{h}$. These h sequences of attention feature are then concatenated into an $n\times \frac{d}{h}$ sequence. Figure 4 shows the steps of the LMHSA. Because of the parallelization, we used the lightweight self-attention mechanism because of the computational complexity per layer and the minimum number of operations required in the sequence. Convolutional layer-based transformer block computation can be written as the following formula:

$$\overline{X_i}=IM\left(X_{i-1}\right)$$

(4)

$$\overline{\overline{X_i}}=LMHSA(LN\left(\overline{X_i}\right)+\overline{X_i)}$$

(5)

Here, $\overline{X_i}$ and $\overline{\overline{X_i}}$ represent the $IM$ and LMHSA block feature for the individual module i, consequently. Moreover, layer normalization is denoted by the term $LN$.

4.2.3. MLP Convolution

We employed the multilayer perception convolution block after the attention in the MHSA, which include a batch normalization layer, which can be defined by the following Equation (6):

$$\overline{\overline{\overline{X_i}}}=MLPConv\left(conv\left(\overline{\overline{X_i}}\right)\right)$$

(6)

The MLP convolutional module contained a single block consisting of two $1\times 1$ convolution layers [83]. This book also looks like a normal convolution layer, where GELU activation and batch normalization are used after the first $1\times 1$ convolution layer and only batch normalization after the second convolutional layer. Still, their kernel size is 1, which is worked for 1 pixel for the input images. The working procedure of this convolution layer is almost the same as the position-wise dense layer. Usually, normal convolution layers detect local patterns based on spatial information, whereas the MLP convolutional layer does not follow this but uses windows of action in a single position. The main difference between the 2D convolution and the MLP convolution is in the channels or the embedding dimensionality. If channel c is from the 2D convolution, $\overline{c}$ is the channel of the MLP convolution, and w and h are the width and height of both types, then $c\ne \overline{c}$. The main purpose of using CNN is to extract two-dimensional neighborhood structures, whereas MLPConv, after MHSA, converts the global MHSA into local pixel information.

4.3. Convolutional Neural Network (CNN) Module

We applied a four-block of $3\times 3$ convolution layer with GELU activation and batch normalization to extract local features from the grain feature. Although there have been many sign language recognition models that only applied CNN to extract local features from the image dataset, we aim to include this advantage in the transformer.

4.4. Classification Module

In the classification module, we used the modified version of FFN of the ViT [84], which described two linear layers separated by an activation function name GELU [29]. They explained that expanding the dimension with a factor of 4 in the first layer and dimension reduces it in the same ways in the second layer.Here, the weight matrices of the two layers can be written as $W_1\in R^{d\times 4d}$ and $W_2\in R^{4d\times d}$, respectively. In addition, the biases of the two layers are represented by $b_1$, and $b_2$, respectively. The schematic diagram of the proposed classification module is shown in Figure 3c, which is similar to the inverted residual feed-forward network (IRFFN) [28,31] consisting of an expansion layer through the element-wise convolution and two projection layers. This can be written as the following Equation (7):

$$F_{avg}=Avg\left(Fconv\left(conv\left(conv\left(conv\left(x\right)\right)\right)\right)\right)$$

(7)

$$F\left(x\right)=F\left(F_{avg}\right)$$

(8)

where $F_{avg}$ represents the output of the averaging pooling layer, and F(x) represents the output of the fully connected layer. In each layer, we included GELU activation and batch normalization layer. The element-wise convolutional neural network is employed to calculate the local information with a minimum cost and value. After that, we applied a global average pooling layer to average the features into a vector, a fully connected (FC) layer and an n-way classification layer with softmax.

5. Experimental Evaluation

We investigated the effectiveness and superiority of the proposed model in the section by conducting experiments on sign language classification with large-scale Korean sign language datasets. Firstly, we demonstrated the performance of the proposed model with multiple datasets, and then we showed the state-of-the-art comparison of the proposed model.

5.1. Training Setting

To train the model, we divided the dataset into 70% as training and 30% as testing. In the training process, we used a learning rate of 0.001, a weight decay of 0.0001, a dropout rate of 0.1, GELU activation, and a batch size of 32. We used a GPU machine with CUDA version 11.7, NVIDIA-SMI 515, GPU name Persistence-M, and RAM 32 GB to implement the system. Models were run for 300 epochs with the optimizer Adam [85] with the Persistence-M GPU.

5.2. Performance with the Benchmark KSL and Proposed Dataset

We considered two benchmark datasets in the experiment: KSL and our proposed dataset. KSL is the famous Korean sign language dataset we used to evaluate the proposed model. Figure 5 shows the confusion matrix showing that most of the Korean alphabet produces 100% accuracy; two signs achieved more than 95%, and two signs achieved in the range 84% to 90% accuracy. Figure 6 demonstrates the proposed model’s precision, recall, F1-score and accuracy with the proposed 20 classes of Korean sign language datasets. The figure demonstrates 98.50%, 98.35%, 98.40%, and 98.30% for precision, recall, F1-score and accuracy, respectively, which proves the effectiveness of the proposed model. Figure 7 visualizes the proposed model’s precision, recall, F1-score and accuracy with the KSL dataset with 20 classes. The figure demonstrates 94.80%, 87.20%, 90.25%, and 89.05% for precision, recall, F1-score and accuracy, respectively, which proves the effectiveness of the proposed model.

Table 1. Performance accuracy of the proposed model.

Dataset Name	Accuracy [%]	Parameters	Flops
Proposed Dataset	98.30	1.52 M	1.17 GMac
KSL	89.00	1.5 M	245.5 MMac

shows the performance accuracy of the proposed model with the proposed and KSL datasets. For the KSL dataset, our model produced 89.00% accuracy, and for the proposed dataset, our model achieved 98.30% accuracy.

5.3. Comparison for the State-of-the-Art Method for KSL Dataset

In this section, we include the state-of-the-art comparison for the KSL dataset.

Table 2. Comparison of the state-of-the-art model for the KSL dataset.

Dataset Name	Model Name	Accuracy [%]
KSL	TSN [76]	79.80
KSL	TSN [86]	83.66
KSL	Proposed Model	89.00

shows the state-of-the-art method performance for the KSL dataset. Authors of the [55] employed a deep learning model, which recently achieved high performance and 79% accuracy. Our proposed model gained 89.00% accuracy, which is higher than the existing state-of-the-art model.

There are some well-known architectures from which to extract hierarchical feature maps based on different resolutions from the RGB image, such as typical convolutional neural networks [76] and fully convolutional networks [86]. The proposed study can be extracted as a multi-scale representation of the input, where we combined CNN and attention-based features, which proved the model’s effectiveness.

6. Conclusions

In the study, we proposed a novel multi-branch architecture for sign language recognition to address the computational complexity and transformer limitations utilized in a brute-force manner. The proposed model takes advantage of both transformers and CNNs to generate global and local information to increase the representation ability of the architecture. Our study achieved good performance compared to the existing Korean sign language model. Extensive experimentation performance visualized the superiority and effectiveness of the proposed model. In the future, we will add more data to the datasets and compare the results with the other latest benchmark methods. In addition, we plan to reduce the number of parameters of the model by considering the speed of the system and apply it in other domains for multi-modal applications.