Beyond Granularity: Enhancing Continuous Sign Language Recognition with Granularity-Aware Feature Fusion and Attention Optimization

Abstract

The advancement of deep learning techniques has significantly propelled the development of the continuous sign language recognition (cSLR) task. However, the spatial feature extraction of sign language videos in the RGB space tends to focus on the overall image information while neglecting the perception of traits at different granularities, such as eye gaze and lip shape, which are more detailed, or posture and gestures, which are more macroscopic. Exploring the efficient fusion of visual information of different granularities is crucial for accurate sign language recognition. In addition, applying a vanilla Transformer to sequence modeling in cSLR exhibits weak performance because specific video frames could interfere with the attention mechanism. These limitations constrain the capability to understand potential semantic characteristics. We introduce a feature fusion method for integrating visual features of disparate granularities and refine the metric of attention to enhance the Transformer’s comprehension of video content. Specifically, we extract CNN feature maps with varying receptive fields and employ a self-attention mechanism to fuse feature maps of different granularities, thereby obtaining multi-scale spatial features of the sign language framework. As for video modeling, we first analyze why the vanilla Transformer failed in cSLR and observe that the magnitude of the feature vectors of video frames could interfere with the distribution of attention weights. Therefore, we utilize the Euclidean distance among vectors to measure the attention weights instead of scaled-dot to enhance dynamic temporal modeling capabilities. Finally, we integrate the two components to construct the model MSF-ET (Multi-Scaled feature Fusion–Euclidean Transformer) for cSLR and train the model end-to-end. We perform experiments on large-scale cSLR benchmarks—PHOENIX-2014 and Chinese Sign Language (CSL)—to validate the effectiveness.

1. Introduction

Sign language serves as a natural and primary means of communication for hearing-impaired individuals, enabling them to convey complex concepts and emotions. The continuous sign language recognition (cSLR) task aims at converting sign videos into a sequence of glosses, and it involves extracting spatial features and interpreting temporal sequences [1]. Therefore, most cSLR model architectures incorporate a convolutional neural network (CNN)-based visual module for spatial analysis and a recurrent neural network (RNN)-based sequential module for temporal sequence interpretation to acquire comprehensive contextual information [2,3,4,5,6].

The semantic information conveyed through sign language is not solely reflected in hand gestures, but also encompasses high-level semantic features such as body posture, subtle facial expressions, lip movements, and eye gaze [7,8]. Thus, the representation of semantic features in sign language videos is multi-scale, making it crucial to comprehensively extract and integrate spatial features of different granularities. On the other hand, the lack of boundary annotations in prevalent datasets, which only provide gloss word annotations, poses a significant challenge to continuous sign language recognition. Therefore, cSLR is regarded as a weakly supervised learning technique [9]. Additionally, aligning extended sign videos with glosses adds to the complexity [10]. Hence, it can be observed that multi-scale perception of spatial features in continuous sign language gestures and temporal modeling of video sequences are crucial for improving recognition accuracy.

Currently, the predominant research approach involves extracting image features using CNNs, followed by temporal modeling with LSTM or 1D-CNN to perceive the semantic information in sign language videos [11,12]. Three-dimensional (3D)-CNNs have also been used to integrate temporal and spatial features for continuous sign language recognition [6,13,14,15,16]. At the same time, research teams have achieved end-to-end learning for sign language recognition and translation using the Transformer model [17,18]. Moreover, hidden Markov models and iterative EM algorithms were also applied in earlier studies to enhance sequence modeling capabilities [19,20]. Although previous research has improved cSLR accuracy, there are still limitations. First, in terms of spatial feature extraction, current studies mainly use convolutional neural networks (CNNs) to convert RGB or depth information into feature maps [10], without considering multi-level and multi-granular details and non-manual semantic information, such as facial expressions and body postures in sign language, which have not been effectively integrated into existing recognition frameworks. For temporal modeling [21], RNN-based models excel in understanding sign language sequences but struggle with parallelizing sequence processing and capturing long-distance contextual relationships due to their network structure [22,23]. Although the Transformer model can address this issue [24], cSLR is a video-text multimodal task, where the information density and redundancy in the video sequence is much higher than that in the text data [25]. Therefore, the original self-attention mechanism cannot fully capture the contextual information and it is difficult to understand the semantics of sign language gestures, which leads to a weak recognition accuracy [18].

To address the above issues, we propose MSF-ET (Multi-Scaled feature Fusion Euclidean Transformer), a novel framework for continuous sign language recognition. We improve both spatial feature extraction and temporal modeling to enhance the learning of high-level semantic information. Although multi-scale features are widely used in visual tasks, for SLR, it is more important to fully integrate visual information of different granularities to model sign language semantics. However, previous work has overlooked the research on feature fusion methods. We designed a multi-scaled feature fusion module to detect image details at different granularities in RGB images, as shown in Figure 1. Whereas existing work has previously employed the concept of multi-scale thinking to enhance continuous sign language translation [9], employing KNN to achieve multi-scaling along the temporal dimension, our approach diverges significantly. Our multi-scale focus emphasizes the incorporation of multi-granularity visual information. More critically, we introduce a multi-scale feature fusion method into cSLR. Although similar methods have been applied in other fields, the adaptation and integration of granularity-aware feature fusion and attention optimization to enhance cSLR present a distinct implementation tailored to the intricacies of sign language, which involves dynamic and nuanced gestures and expressions. Our spatial perception module outputs fused features, incorporating coarse-grained and fine-grained characteristics, enhancing the model’s understanding of semantic information. Regarding the temporal dimension, we first analyze the problems of the original Transformer architecture in sign language video perception. We observe that the dot-product attention mechanism is susceptible to the influence of vector magnitudes when calculating attention weights. Building on this foundation, we refine the traditional dot-product attention mechanism by designing a Transformer model based on Euclidean distance attention metric for effective modeling of video sequences.

Our main contributions are summarized as follows:

• We introduce a methodology for fusing multi-scale visual feature maps, leveraging a self-attention mechanism to facilitate the flexible and dynamic integration of visual feature maps across diverse dimensions and granularities.
• We analytically examine the limitations of the vanilla Transformer in modeling sign language videos and propose an improvement to the self-attention mechanism by substituting the dot product with Euclidean distance.
• Extensive experiments and ablation studies validate the efficacy of our method, significantly enhancing the performance of Transformer-based models on the continuous cSLR task.

2. Related Work

2.1. Continuous Sign Language Recognition

Continuous sign language recognition (cSLR) aims at converting a sign language video into a gloss sequence. The challenge lies in the lack of explicit boundaries in videos for various glosses, which is a weakly supervised sequence labeling problem [10,26]. Deep learning advances cSLR by focusing on spatial representation and temporal modeling of videos. TWo-dimensional convolutional neural networks (2D-CNNs) are employed for perceiving the static information in video frames. Several state-of-the-art pre-trained CNNs, such as VGGs [27], ResNets [28], and EfficientNets [29] are introduced in cSLR, which strengthens the recognition and understanding of frame semantics [30,31,32]. Cui et al. integrate the 2D-CNN with temporal convolution to learning the spatial–temporal information [3]. Similarly, Cheng et al. propose a fully convolutional network (FCN) for cSLR that extracts frame features and models the feature sequence with 1D-CNN [12]. Furthermore, to comprehend movement information, 3D-CNNs are utilized to learn motion characteristics, providing a more comprehensive understanding of dynamic actions in sign language videos [33,34].

Sequence modeling in cSLR is to learn the mapping between sign language videos and gloss sequences. The hidden Markov model (HMM) was adopted in earlier research to predict the glosses through state transitions. Koller et al. adopted HMM in combination with CNNs to achieve sign language recognition [4,19]. With the emergence of recurrent neural networks, improved models such as LSTM and GRU, more suitable for sequence processing, were applied to this task. Cui et al. introduced the bi-directional LSTM to model the sign sequences [3]. In addition, as the input video sequence is considerably longer than the corresponding glosses, the connectionist temporal classification (CTC) [35] approach was adopted for sequence alignment. Recent studies demonstrated that CTC effectively aligns unsegmented data by dynamically adjusting to varying sequence lengths, making it a powerful solution for handling lengthy and complex video sequences [13,36].

There exist some endeavors aimed at enhancing the performance. Xie et al. designed a position-aware temporal convolution to capture local consistent semantics within temporally neighboring frames [9]. This method addresses the issue of CNNs being agnostic to similarity or dissimilarity. Zhou et al. proposed a spatial–temporal multi-cue (STMC) network to learn implicit visual grammars underlying the collaboration of different visual cues [7]. Cui et al. introduced the iterative training that uses the alignment proposal generated by recognition model as strong supervisory inference to tune the feature extraction module [37]. Additionally, the research applied RGB images and optical flow to capture both appearance and motion information, achieving competitive performance.

2.2. Feature Fusion

Incorporating multi-scale features is crucial for boosting performance in various tasks [38,39]. In the CNN backbone networks, recent studies generally agree that lower-level features possess higher resolution and more detailed information, yet due to the limited number of convolutional layers, they contain less semantic information and more noise. Conversely, the higher-level features exhibit the opposite characteristics. The module in the Inception network consists of four parallel branches: 1 × 1 convolution, 3 × 3 convolution, 5 × 5 convolution, and 3 × 3 max pooling [40]. The output feature maps of the branches are concatenated along the channel dimension, combining information from multiple scales to enhance feature representation. Similarly, U-Net [41] and the feature pyramid networks (FPN) [42] approach involve pooling or upsampling features at various scales before prediction.

There is a paucity of research on the issue of multi-scale feature fusion in continuous sign language recognition, despite its fundamental importance in sign language video understanding. There are still some studies that employ multi-modal data for feature fusion. Cui et al. took RGB image and optical flow for hand regions as input to combine the appearance and motion information. The authors summed the two stream outputs generated by CNN modules for fusion [37]. On the other hand, a spatial multi-cue module in STMC was specifically designed for spatial representation and employs a self-contained pose estimation branch to explicitly decompose the visual features of different cues [7]. This approach also models and fuses each cue using sequence modeling, enriching the spatio-temporal information. The results demonstrate the benefits of feature fusion in learning sign language semantics. However, it is worth noting that current methods still have certain limitations. First, acquiring optical flow data can be challenging, and their accuracy is difficult to determine due to their sensitivity to noise [43,44]. Second, the STMC method relies on a pose estimator for keypoint detection. However, in practice, it can be difficult to accurately detect keypoints in motion images, which can lead to errors [45,46]. Therefore, it is necessary to explore more concise and reliable methods to address these issues.

Based on previous research, we propose a spatial feature extraction module for multi-scale feature fusion. This module uses a Transformer encoder to process feature maps of different receptive fields and concatenate them, effectively combining low-level details with high-level semantic information. This not only enhances the model’s ability to capture fine-grained features but also improves its understanding of complex semantic structures. This approach avoids the need for additional data and modules and enables end-to-end training.

3. Methodology

3.1. Overall Model Architecture

Formally, the proposed model aims to understand a sign language video containing many glosses and learn a mapping $R:V\to G$ that recognizes the input video frame sequence V as a gloss sequence G. The overall model framework is shown in Figure 2. Given a video, a multi-layer convolutional network first extracts spatial features from each frame and generates various feature maps with diverse receptive fields. Then, a fusion module transforms each feature map to feature vector and concatenates all the multi-scaled features. Next, the sequence composed of fused features is learned by temporal module. Finally, the CTC decoder is used to align the sequence for model training and inference.

3.2. Spatial Encoder and Multi-Scale Features

In addition to RGB images, fine-grained details such as facial expressions, lips, and hand shapes are essential for sign language recognition. We leverage the inherent properties of convolutional neural networks (CNNs), which are particularly adept at local perception within images. As layers are stacked in a CNN, the receptive field progressively enlarges, which enables the network to transition from capturing minute details to extracting broader semantic features. We harness this capability by extracting visual features at various receptive field sizes, thus obtaining a range of granularities—from fine-grained details such as facial expressions and hand shapes to more coarse-grained features like overall body posture and gesture dynamics. This fusion of multi-scaled features ensures that our model captures both detailed and global visual information, essential for the nuanced interpretation required in cSLR.

We design stacked convolutional basic modules as a spatial encoder (SE) to extract features from frames of sign language videos. Each basic module comprises two convolutional layers, batch normalization, and rectified linear unit (ReLU) activation function, as shown in Figure 2. The spatial encoder’s last two basic modules yield two distinct feature maps with dimensions of $7\times 7$ and $3\times 3$, which are considered fine-grained (FG) and coarse-grained (CG) features, respectively. A larger feature map dimension corresponds to a narrower receptive field, incorporating a higher degree of detail-oriented features. Conversely, smaller dimensions encompass more abstract semantic information. Upon computation, the receptive fields for fine-grained and coarse-grained features are determined to be 31 and 67, respectively. We then propose a self-attention-based approach for fusing these features, leveraging the self-attention mechanism’s ability to capture dependencies across different scales.

3.3. Multi-Scale Features Fusion with Self-Attention

Feature maps are two-dimensional data that need to be converted into one-dimensional vectors for fusion and sequence modeling. Conventional techniques like global average pooling (GAP) or global max pooling (GMP) compress feature maps directly but often result in information loss [47]. Moreover, studies have indicated that the averaging operation in GAP excessively smoothes feature maps, thereby submerging information from crucial regions, whereas GMP is sensitive to outliers due to its maximization operation [48].

To address these issues, we propose a self-attention-based approach that dynamically integrates global information from feature maps. As depicted in Figure 3, the two-dimensional feature map is initially unfolded into a one-dimensional sequence, to which a randomly initialized special token $<cls>$ is appended at the beginning, serving as the label for the feature map. Subsequently, a two-layer Transformer encoder is employed to process the sequence, with the $<cls>$ token representing the feature map’s output as it encapsulates global information. Finally, the outputs of multi-scale feature maps are concatenated to achieve fusion. This method dynamically integrates global information from feature maps, offering a more flexible approach. The success of the Vision Transformer (ViT) model, which shares similarities with our method, provides a research foundation for our feature fusion strategy.

Specifically, given a video $V=\{p^1,p^2,\dots ,p^T\}\in {\mathbb{R}}^{T\times H\times W\times C}$ and $p^i\in {\mathbb{R}}^{H\times W\times C}$, where T denotes the number of frames in V, where T is the number of frames, and H, W, and C represent the height, width, and channels of each frame, respectively. Thus, the process of handling and fusing multi-scale features is illustrated as shown in Equations (1)–(4):

$$m_1^i,m_2^i=SE\left(p^i\right)$$

(1)

$$v_1^i=f_{trans\_enc}\left([m^{cls},Flatten\left(m_1^i\right)]\right)$$

(2)

$$v_2^i=f_{trans\_enc}([m^{cls},Flatten\left(m_2^i\right))$$

(3)

$$c^i=Concat(v_1^i,v_2^i)$$

(4)

where $SE$ is spatial encoder and $f_{trans\_enc}$ is Transformer encoder layer. In addition, $p^i\in {\mathbb{R}}^{224\times 224\times 3}$ denotes the ith frame image and $m_1^i\in {\mathbb{R}}^{7\times 7\times 256}$ and $m_2^i\in {\mathbb{R}}^{3\times 3\times 256}$ are the feature maps. The outputs $v_1^i$, $v_1^i\in {\mathbb{R}}^{1\times 256}$ represent the outcomes derived from the application of the Transformer encoder to process the sequences, which are both flattened and supplemented with the $<cls>$ token. Finally, by concatenating $v_1^i$ and $v_2^i$, we derive $c^i\in {\mathbb{R}}^{1\times 512}$, which serves as the image features for subsequent temporal modeling, providing a comprehensive representation that integrates both fine-grained and coarse-grained information.

3.4. Self-Attention Based on Euclidean Distance

In sequence modeling, learning the relationships between tokens and contextual information is crucial. The Transformer model employs a self-attention mechanism to capture the global context of sequences, demonstrating exceptional capabilities in processing text sequences. However, videos differ from sentences in that every token in a sentence can be converted into a word vector using embeddings, with finite vocabularies constraining the feature representation of words. In contrast, each frame of a video has an infinite number of representations within a specific space, resulting in a gap between the application of Transformers to videos and text.

First, we analyze the limitations of Transformers in processing video sequences. Then, we propose an improved approach to make them more adaptable for perceiving and learning contextual information within visual sequences, leveraging Euclidean distance for enhanced attention.

The self-attention in vanilla Transformer is formulated as Equation (5):

$$Attention(Q,K,V)=Softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$

(5)

The crucial part involves calculating the attention map by multiplying query $Q\in R^{t\times c}$ and key $K\in R^{t\times c}$. We assume $Q={[q_1,q_2,\dots ,q_t]}^T$ and $K={[k_1,k_2,\dots ,k_t]}^T$, so we can obtain the following inference:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q{K}^T=\left[\begin{array}{c}{q}_1^T\\ q_2^T\\ ⋮\\ q_t^T\end{array}\right]\left[\begin{array}{cccc}{k}_1& k_2& \cdots & k_t\end{array}\right]\\ \hfill =\left[\begin{array}{cccc}{q}_1^T{k}_1& q_1^T{k}_2& \cdots & q_1^T{k}_t\\ q_2^T{k}_1& q_2^T{k}_2& \cdots & q_2^T{k}_t\\ ⋮& ⋮& \ddots & ⋮\\ q_t^T{k}_1& q_t^T{k}_2& \cdots & q_t^T{k}_t\end{array}\right]\end{array}\end{array}$$

(6)

$${\alpha }_{i,j}=q_i^T{k}_j=\Vert q_i\Vert ·\Vert k_j\Vert ·cos{\theta }_{ij}$$

(7)

where, in Equations (6) and (7), ${\alpha }_{i,j}$ presents the value of row i and column j in the attention map, and ${\theta }_{ij}$ means the angle of vector $q_i$ and $k_j$. According to Equation (5), the $Q{K}^T$ necessitates softmax operation in the row direction. In each row, the query vector $q_i$ is the same, indicating that the attention weights are influenced by the magnitude and cosine similarity. However, our prior research reveals that an exceedingly large vector may disrupt the attention mechanism, causing it to focus excessively on a specific position while disregarding more global information (as shown in Figure 4).

$$EDAttention(Q,K,V)=Softmax\left(ED\right(Q,K\left)\right)V$$

(8)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill ED(Q,K)=\left[\begin{array}{cccc}<q_1,k_1>& <q_1,k_2>& \cdots & <q_1,k_t>\\ <q_2,k_1>& <q_2,k_2>& \cdots & <q_2,k_t>\\ ⋮& ⋮& \ddots & ⋮\\ <q_t,k_1>& <q_t,k_2>& \cdots & <q_t,k_t>\end{array}\right]\\ \hfill <q_i,k_j>=-\Vert q_i-k_j{\Vert }_2^2=-\Vert q_i{\Vert }_2^2-{\Vert k_j\Vert }_2^2+2q_i{k}_j\end{array}\end{array}$$

(9)

Therefore, to address the issue, we propose EDAttention, which replaces the dot product with the Euclidean distance between vectors as attention weights. Specifically, the attention weights are computed based on the Euclidean distance between the query vectors and key vectors (as shown in Equations (8) and (9)).

In the EDAttention mechanism, the attention score ${\alpha }_{i,j}$ is defined as the negation of the Euclidean distance between vectors $q_i$ and $k_j$, resulting in ${\alpha }_{i,j}\le 0$. Subsequent $softmax$ operation compresses ${\alpha }_{i,j}\in (0,1]$. Moreover, EDAttention achieves that greater distances between vectors correspond to lower similarity and, consequently, smaller attention weights, which aligns with analytical logic and intuitive perception. This approach ensures that attention is more evenly distributed across all positions in the sequence, allowing the model to fully capture the global information in long video sequences.

3.5. The Local Window in Self-Attention

In continuous sign language recognition, there are strict sequential correspondences between video clips and glosses, even if there is no definite boundaries to segment them. Consequently, we incorporate a local Transformer encoder layer before the model’s output, aiming to reinforce the perception of local information in the sequence following temporal modeling, thereby facilitating the alignment of local video clips with gloss semantics. As illustrated in Figure 5, we introduce a special masking operation before calculating the attention matrix and $softmax$ processing. This operation restricts the primary receptive range of each frame to a confined window and assigns negative infinity to the attention values of elements outside the window, effectively impeding interaction between distant content and the current frame.

This localized self-attention mechanism focuses more on individual video segments rather than the entire video. In conjunction with the improved Transformer, it composes the temporal modeling module for videos, capturing both local and global contexts. The efficacy of this approach is further substantiated by ablation experiments discussed in the subsequent section.

3.6. CTC Loss and CTC Decoder

The continuous sign language videos are unsegmented so that there are not the obvious boundaries between the glosses. Therefore, we introduce the connectionist temporal classification (CTC) to align the sequences. CTC aims to construct the map from frame sequence to gloss sequence even though these two sequences have different and various lengths, and it is applied for speech recognition first.

In our model, given a sequence $\mathbf{S}={\left\{s_i\right\}}_{i=1}^T$ generated from video by spatial–temporal modules, CTC endeavors to map it to the sign gloss sequence $\mathbf{G}={\left\{g_j\right\}}_{j=1}^L$. The objective of CTC is to maximize the sum of probabilities of all possible alignment paths between the input and target sequence. We introduce the special token $<BLANK>$ to model the transitions meticulously and eliminate the ambiguity caused by repeated glosses. In addition, we define the alignment path of the source sequence as $\pi ={\left\{{\pi }_t\right\}}_{t=1}^T$, where label ${\pi }_t$ denotes one of the gloss vocabulary included blank token. The probability of alignment path $\pi$ given the input sequence is represented as follows:

$$p\left(\pi \right|\mathbf{S})=\prod _{t=1}^{T}p\left({\pi }_t\right|\mathbf{S})=\prod _{t=1}^T{y}_{t,{\pi }_t}$$

(10)

We calculate the conditional probability of gloss sequence $\mathbf{G}$ as the sum of probabilities of all paths which can be mapped to $\mathbf{G}$:

$$p\left(\mathbf{G}\right|\mathbf{S})=\sum _{\pi \in {\mathcal{B}}^{-1}\left(\mathbf{G}\right)}p\left(\pi \right|\mathbf{S})$$

(11)

where $\mathcal{B}$ denotes the mapping operation that removes all $<BLANK>$ tokens and repeated words in the alignment path, and ${\mathcal{B}}^{-1}\left(G\right)=\left\{\pi \right|\mathcal{B}\left(\pi \right)=G\}$ is the inverse operation of $\mathcal{B}$. Eventually, CTC loss, which is as the objective function of the model, is defined as follows:

$${\mathcal{L}}_{CTC}=-logp\left(\mathbf{G}\right|\mathbf{S})=-log[\sum _{\pi \in {\mathcal{B}}^{-1}\left(\mathbf{G}\right)}p\left(\pi \right|\mathbf{S})]$$

(12)

The entire training process of MSF-EF is shown in Algorithm 1. During the training stage, we instruct the model by reducing the CTC loss gradually, and the CTC decoder is applied for target sequence prediction and inference. The CTC decoder is utilized to obtain the target sequence with the highest cumulative probability by the beam search algorithm.

Algorithm 1 Multi-scaled feature fusion–Euclidean Transformer (MSF-ET) for continuous sign language recognition.

Input: Sign language videos $V\in {\mathbb{R}}^{T\times 3\times 224\times 224}$ Output: Sequence of glosses $\mathbf{G}$ corresponding to the video V 1: set $i=0$; 2: while $i<\mathit{max}\_\mathit{epoch}$ do 3:     for $(V,\mathbf{G})$ in Dataset $\mathbb{D}$ do 4:         Initialize MSF-EF model parameters $\theta$ 5:         $m_1,m_2=f_{SE}\left(V\right)$ //Obtain spatial features; 6:         $m_1\leftarrow [<cls>,flatten\left(m_1\right)]$, $m_2\leftarrow [<cls>,flatten\left(m_2\right)]$ 7:         $v_1=f_{trans\_enc}\left(m_1\right)$, $v_2=f_{trans\_enc}\left(m_2\right)$ // Multi-granularity features processing; 8:         $c\leftarrow Concat\left([v_1,v_2]\right)$ // Feature fusion; 9:         $\mathbf{S}=f_{temp}\left(c\right)$ // Temporal sequence modeling; 10:       $\mathcal{L}=-logp\left(\mathbf{G}\right|\mathbf{S})=-log\left[{\sum }_{\pi \in {\mathcal{B}}^{-1}\left(\mathbf{G}\right)}p\left(\pi \right|\mathbf{S})\right]$ // CTC loss calculation; 11:       ${\theta }^{\prime }=\theta -\eta {\nabla }_{\theta }\mathcal{L}$ // Gradient backward; 12:       $\theta \leftarrow {\theta }^{\prime }$ // Parameter update. 13:     end for 14:     $i=i+1$ 15: end while

4. Experiments

4.1. Datasets and Metrics

We evaluate the proposed method on two mainstream continuous sign language recognition datasets: PHOENIX-2014 and CSL.

PHOENIX-2014 is the most popular benchmark for cSLR. All the videos are recorded from broadcast news and the split of these for Train, Dev and Test is 5672, 540, and 629, respectively. The dataset includes 9 different signers with a vocabulary size of 1295. All videos are transferred frame sequences at 25 fps.

CSL is a Chinese sign language dataset, and it contains 100 sign sentences with 178 words. All the videos are performed by 50 signers and repeated 5 times. Even though the vocabulary size is small, this dataset has diverse performance because of the signers’ various wear and action speeds.

In this task, we use word error rate (WER) as the metric of evaluating the performance of recognition. There are three types of mistakes of predicted sequence: substitutions (#sub), deletions (#del), and insertions (#ins), respectively. Thus, the WER from hypothesis to reference is defined as follows:

$$WER={\displaystyle \frac{\#sub+\#del+\#ins}{\#Totalwords}}.$$

(13)

4.2. Implementation Details

4.2.1. Network Details

The spatial encoder consists of a preprocess module and five basic modules, and the preprocess model includes a convolutional layer and max pooling. Each basic module contains two convolution layers with batch normalization and ReLU activation function. The epsilon and momentum of batch normalization are set to 1 × ${10}^{-5}$ and $0.1$, respectively. A max pooling operation is applied between the two convolution layers for downsampling. All convolutional layers are set with F3-S1-P1, where F, S, and P denote the filter size, stride, and padding size, respectively. We increase the channels module by module, and the output channels is unified to 256.

The fusion module is two vanilla Transformer encoder layers based on Vaswani et al. Specifically, each layer is set as 8 heads and 2048 hidden units, and the dropout rate is 0.1. As illustrated in Figure 3, we design one fusion module to process the different feature maps. In addition, the config of MSF-ET is similar to the fusion module.

In the inference stage, we utilize the CTC beam search algorithm to decode the sequence, and we set the beam size as 5.

4.2.2. Training Setup

During training, all the samples are limited to max length 300 frames. For data augmentation, all the frames in the video are first resized to $256\times 256$ and then randomly cropped to $224\times 224$ during training and center cropped to $224\times 224$ in testing. Specifically, for temporal augmentation, we first randomly repeat the sequence by 20% and then randomly remove 20% content. We utilize Adam to optimize our model, and the weight factors ${\lambda }_1$ and ${\lambda }_2$ are empirically set to be ${10}^{-4}$ and $0.05$, respectively. The initial learning rate is 5 × ${10}^{-5}$, and the learning rate is reduced by half per epoch after 40 epochs. In this experiment, a single Nvidia RTX 3090 with 24 G memory is used to accelerate model training and inference.

4.3. Ablation Study

4.3.1. Analysis of Multi-Scale Feature Fusion

In our proposed model, we introduce the multi-scale feature fusion method with self-attention to capture more comprehensive information in sign language videos. We specifically choose $3\times 3$ and $7\times 7$ feature maps to represent coarse-grained and fine-grained features, respectively.

To validate the effectiveness of using 3 × 3 and 7 × 7 feature maps for capturing coarse and fine details, we used class activation mapping (CAM) [49] to visualize the attention weights of feature maps at different scales, allowing us to observe the regions that the model focuses on at different feature scales. As shown in Figure 6, the 7 × 7 feature maps capture finer details such as facial expressions and hand shapes. These fine-grained features are crucial for distinguishing subtle variations in sign language gestures, providing detailed information that is essential for precise recognition. In contrast, Figure 6 shows that the 3 × 3 feature maps capture more coarse-level movements, such as overall body posture and hand trajectories. These coarse-grained features are essential for understanding the general structure and flow of sign language, providing a broader context that is necessary for recognizing overall gestures.

Table 1. Evaluation of various feature selections on PHOENIX-2014. The ↓ means the lower the better.

Spatial Features	Dev		Test
	del/ins	WER ↓	del/ins	WER ↓
Fine-grained features	13.5/4.0	32.9	14.1/3.7	33.4
Coarse-grained features	12.0/3.6	27.1	11.8/3.6	27.0
Fused features	7.4/3.3	22.4	7.3/3.2	22.6

compares the performance of different feature selections on the PHOENIX-2014 benchmark. For fine-grained features, we only utilize the 7 × 7 feature map with a receptive field of 31. For coarse-grained features, we use the 3 × 3 feature map. The fusion process is shown in Figure 3. The results demonstrate that using the proposed feature fusion method can achieve better performance compared to using single features, with WER reductions of 4.7 and 10.5, respectively. This indicates that the feature fusion method enhances the spatial representation of the sign language videos and facilitates the model’s understanding of sign language actions.

The visualizations support our experimental findings: fine-grained features, although detailed, also include more noise, making them less effective when used alone. Coarse-grained features provide a clearer semantic representation but miss finer details. By combining both types of features, our model leverages the strengths of each, resulting in a more robust understanding of sign language videos. This comprehensive approach ensures that both the macro (coarse) and micro (fine) aspects of sign language are effectively captured, leading to improved recognition performance.

4.3.2. Analysis of Feature Fusion Method

As for feature fusion, we introduce the self-attention mechanism to handle feature maps of different scales, instead of simple average pooling or max pooling. Furthermore, there are two conventional fusion methods: element-wise addition and concatenation. For cSLR, we conducted experiments to explore the effect of different fusion methods.

In

Table 2. Evaluation of different feature fusion methods on PHOENIX-2014.

Fusion Methods		Dev		Test
		del/ins	WER	del/ins	WER
Concatenation	max-pooling	14.6/3.3	26.7	14.4/3.9	27.1
	avg-pooling	12.1/4.9	25.9	12.8/3.6	26.2
	self-attention	7.4/3.3	22.4	7.3/3.2	22.6
Sum	max-pooling	13.7/3.7	26.5	13.5/4.0	26.9
	avg-pooling	12.8/4.2	26.1	12.2/3.7	25.8
	self-attention	7.6/3.7	22.8	7.4/4.1	23.1

, first, in the vertical comparison, the fusion methods of concatenation and element-wise addition have almost no significant impact on the performance, indicating that the information contained in both operations is almost consistent. Therefore, it is recommended to choose a more convenient method according to the specific task requirements. The results indicate that our proposed feature map processing method with the self-attention mechanism is significantly better than both pooling methods, with a reduction of 3.5–4 in the WER metric. This is because both pooling methods result in a loss of information [48]. Max pooling only retains the maximum value in a local region, discarding other information that may be valuable [47]. Global average pooling operates on the entire feature map, potentially ignoring detail information and reducing sensitivity to details. Our proposed method flattens the feature maps and uses a Transformer encoder to process them as a sequence, which can perceive global information while dynamically preserving details. In addition, it is more flexible compared to pooling operations. The success of the ViT model can also attest to the effectiveness of this approach to some extent.

4.3.3. Analysis of Attention with Euclidean Distance

For sequence modeling, we explain the shortcomings of the dot product attention mechanism when applied to video sequences. It is affected by the magnitude of feature vectors in the sequence, causing attention weights to overly focus on local segments and thus undermining the model’s ability to perceive the global context. Therefore, we use the Euclidean distance between feature vectors to measure attention in sequence modeling. In

Table 3. Evaluation of different attention weight calculation methods on PHOENIX-2014.

Spatial Features	Attention	Dev		Test
		del/ins	WER	del/ins	WER
Coarse-grained features	scaled-dot	12.3/4.8	28.3	12.7/4.1	28.5
	Euclidean	12.0/3.6	27.1	11.8/3.6	27.0
Fine-grained features	scaled-dot	16.1/2.9	34.7	15.3/3.5	34.5
	Euclidean	13.5/4.0	32.9	14.1/3.7	33.4
Fused features	scaled-dot	8.7/3.4	26.3	8.4/4.2	25.9
	Euclidean	7.4/3.3	22.4	7.3/3.2	22.6

, we observe that using Euclidean distance for sequence modeling in Transformer, whether with single or fused features, results in lower WER (1.5–3.9 lower) compared to the scale-dot product. The results demonstrate that our improvements to the Transformer are more effective in learning and understanding the semantics of continuous sign language gestures. Furthermore, we visualized the attention weight matrices of the Transformer using Euclidean distance and applied it to the same sample data as in Figure 4, as shown in Figure 7. Compared to Figure 4, it is evident from Figure 7 that, after using Euclidean distance to measure similarity, the attention weight distribution is more uniform, effectively alleviating the sparsity issue.

4.3.4. Analysis of Local Transformer

In the task of continuous sign language recognition, each gloss is associated with a specific segment in the continuous sign language video. To augment the model’s ability to learn local information within the sequence, we incorporated a local Transformer prior to the output layer. In order to compare its performance with 1D-CNN and examine the optimal size of the local window, we conducted experiments to investigate the differences in performance across various window sizes on PHOENIX-2014.

Based on the experimental results, it is evident that the absence of a network with local perception capabilities leads to a highly unsatisfactory final outcome. This finding underscores the significance of local temporal perception for cSLR tasks. The integration of a 1D-CNN contributes to an approximate reduction of three in the WER metric, as it leverages a smaller convolutional kernel for processing local regions within extended sequences, ultimately supplying local information for the final output. Utilizing a local Transformer further diminishes the WER, which can be ascribed to the 1D-CNN’s shared convolutional kernel that executes convolution operations using identical parameters on varying local information, thus yielding constrained information extraction. Conversely, the self-attention mechanism of the Transformer facilitates the dynamic assimilation of local sequences, demonstrating a more adaptable and resilient processing approach capable of capturing an abundance of local information. On the other hand, in

Table 4. Evaluation of last local layers and local size before output on PHOENIX-2014. The term “w/o” denotes the direct output of results without employing any network.

Last Layers	Local Size	Dev		Test
		del/ins	WER	del/ins	WER
w/o	-	10.3/3.2	28.9	10.5/3/4	29.3
1D-CNN	3	11.2/3.9	27.2	11.8/3.7	27.7
	5	7.9/3.8	26.1	7.8/3.7	25.9
	7	7.7/3.7	25.6	7.9/3.7	25.8
Local Transformer	3	8.2/3.1	24.7	7.6/4.2	24.2
	5	7.4/3.3	22.4	7.3/3.2	22.6
	7	8.5/4.7	25.0	7.7/3.7	24.6

, for the 1D-CNN, a larger convolutional kernel contributes to improved performance. However, with regards to the local Transformer, setting the local perception size to 5 yields the best results. If the size is too small, it may not be able to perceive sufficient information, whereas if it is too large, the performance might be adversely affected by the actions corresponding to adjacent glosses.

4.3.5. Analysis of Time Complexity and Inference Time Cost

Our model includes both spatial feature extraction using convolutional neural networks (CNNs) and temporal modeling using our multi-scaled feature fusion–Euclidean Transformer (MSF-ET) and involves several computational stages, each contributing to the overall complexity. The stage of spatial feature extraction employs a stack of convolutional layers. The time complexity for each convolution operation is generally $O\left(d^2{k}^2\right)$, where d is the dimension of the feature map, and k is the kernel size. In the feature fusion module, we utilize a self-attention mechanism across different feature maps generated by the spatial encoder. The complexity of the self-attention mechanism is $O(l^2·d)$, where l is the sequence length, and d is the dimensionality of each token. During the temporal stage, the Euclidean Transformer involves calculating distances between all pairs of input sequence elements, resulting in a complexity of $O(t^2·d)$.

We conduct additional analyses to measure the processing time. Our MSF-ET model processes each frame through several stages, including spatial feature extraction, feature fusion, and temporal encoding, as shown in Figure 8 In our current implementation on an Nvidia RTX 3090 GPU, the average processing time per 25 frames is approximately 96~134 milliseconds (ms). This includes convolution operations, feature fusion using self-attention, temporal encoding with our Euclidean Transformer, and the CTC decoding. Due to the parallel processing capabilities of attention mechanisms, inference time remains below 250 ms even when the number of video frames exceeds 200 (as shown in the following figure), enabling real-time inference for sign language videos that are under 200 FPS.

4.4. Comparison with Baselines

4.4.1. Evaluation on PHOENIX-2014

We conduct a comparative analysis of various state-of-the-art models to substantiate the efficacy of our research, as shown in

Table 5. Comparison of our proposed method with existing approaches on the PHOENIX-2014 dataset, where lower values indicate better performance. “RGB” and “Flow” in the table refer to frame images and optical flow, respectively. “SL” stands for supervised learning and “RL” represents reinforcement learning. The ↓ means the lower the better.

Models	Dev		Test
	del ↓/ins ↓	WER ↓	del ↓/ins ↓	WER ↓
HMM/CNN/RNN Based
CMLLR [50]	21.8/3.9	55.0	20.3/4.5	53.0
SubUNets [2]	14.6/4.0	40.8	14.3/4.0	40.7
Staged-Opt [3]	13.7/7.3	39.4	12.2/7.5	38.7
Re-sign [4]	-/-	27.1	-/-	26.8
LS-HAN [32]	-/-	-	-	38.3
CNN-HMM [19]	-/-	31.6	-/-	32.5
DenseTCN [16]	10.7/5.1	35.9	10.5/5.5	36.5
CNN-LSTM-HMM [20]	-/-	26.0	-/-	26.0
DNF(RGB) [37]	7.8/3.5	23.8	7.8/3.4	24.4
DNF (RGB + Flow) [37]	7.3/3.3	23.1	6.7/3.3	22.9
FCN [12]	-/-	23.7	-/-	23.9
SMC [7]	7.6/3.8	22.7	7.4/3.5	22.4
VAC [51]	7.9/2.5	21.2	8.4/2.6	22.3
mLTSF-Net [9]	8.5/3.3	23.8	8.3/3.1	23.5
CVT-SLR [52]	-/-	21.8	-/-	22.0
MSKA-SLR [53]	-/-	21.7	-/-	22.1
Transformer Based
REINFORCE (SL) [34]	5.7/6.8	39.7	5.8/6.8	40.0
REINFORCE (RL) [34]	7.3/5.2	38.0	7.0/5.7	38.3
SAFI [54]	16.6/1.8	31.7	15.1/1.7	31.3
SL-Transformers [18]	39.3/2.8	50.18	37.0/2.8	47.96
SL-Tramsformers(pre) [18]	13.5/5.7	26.7	13.8/6.4	27.62
PiSLTRc-R [55]	8.1/3.4	23.4	7.6/3.3	23.2
signBERT+ [56]	-/-	34.0	-/-	34.1
MSF-ET (Ours)	7.4/3.3	22.4	7.3/3.2	22.6

. We categorize the compared models into two major groups based on their distinct sequence modeling approaches: methods based on Transformer and those employing other network architectures. This is due to the weak performance of the original Transformer on cSLR tasks, which warrants a separate comparison. CMLLR and 1-Mio-Hands primarily utilize traditional HMM models to learn sign language videos. CNN-LSTM-HMM combines CNN-LSTM with HMM to exploit multi-stream data. DNF employs multimodal data comprising RGB and optical flow to enhance the capture of motion features. FCN establishes a fully convolutional network for spatial and temporal modeling. Finally, mLTSF-Net fuses temporal information based on local time similarity, thereby capturing the local consistent semantics of adjacent frames. In comparison to these methods, our proposed approach surpasses them to varying degrees without using additional data, achieving end-to-end training. The VAC model, which integrates VGG and BiLSTM, employs alignment learning to enhance its capabilities, achieving the current state-of-the-art results. Our evaluation metrics on the test set are slightly lower (+0.2 WER). Our approach focuses on the application of Transformer models in continuous sign language recognition (cSLR) tasks, optimizing and improving issues related to single visual features and self-attention mechanisms. The alignment learning and auxiliary learning methods used in VAC aim to strengthen the alignment between visual features and glosses, offering valuable insights for us to learn from and further improve our model’s performance.

Within the Transformer-based sequence modeling methods, it is worth noting that the self-attention-based approach consistently falls short compared to CNN or RNNs. Analyzing and addressing this phenomenon also constitutes one of the contributions of this work. As observed in Table 5, the application of Transformer to this task results in a WER metric of 30 or even higher. REINFORCE attempts to improve the encoder–decoder structure using reinforcement learning techniques. SL-Transformers employ pre-trained features from LSTM-HMM, reducing the WER from 50.18 to 26.7. In contrast to these models, our proposed method significantly lowers the WER metric, decreasing it by 15.6 points compared to the reinforcement learning approach, and even achieving a reduction of over 50% relative to the untrained SL-Transformers. The comparison with baseline models attests to the effectiveness of our proposed method, which not only achieves state-of-the-art performance among Transformer-based approaches but also rivals CNN/RNN models.

4.4.2. Evaluation on CSL

To demonstrate the robustness and effectiveness of our approach, we compared it with various baseline methods on a Chinese sign language dataset. In

Table 6. The evaluation results on CSL dataset.

Method	WER
LS-HAN [32]	17.3
HLSTM-attn [57]	10.2
CTF [13]	11.2
DenseTCN [16]	14.3
SF-Net [6]	3.8
FCN [12]	3.0
mLTSF-Net [9]	2.8
PiSLTRc-R [55]	2.8
SL-Transformer (implement)	3.7
MSF-ET (Ours)	2.7

, on the CSL dataset, our model achieved state-of-the-art (SOTA) results, albeit with a marginal improvement of 0.1 points in WER compared to CNN/RNN methods. The results demonstrate that our approach remains competitive in the domain of Chinese sign language recognition. Furthermore, to the best of our knowledge, there are no reported metrics for Transformer-based models on the CSL benchmark. Therefore, we replicate the SL-Transformer model’s WER on CLS, as illustrated in the penultimate row of Table 6. Employing the self-attention mechanism, our method reduces the WER by 23.3% compared to SL-Transformer on this benchmark, achieving the state-of-the-art for self-attention models.

5. Conclusions

In this paper, we develop an innovative method for multi-scale feature fusion in images, utilizing self-attention to preserve both low-level details and high-level semantic information. Furthermore, we improve vanilla Transformer by adopting Euclidean distance instead of scaled-dot product to measure attention weights, effectively mitigating the attention sparsity issue.

Our experimental results on German and Chinese sign language datasets demonstrate the competitive performance of our proposed model in comparison to existing methods, showcasing its versatility and effectiveness across different domains. The successful application of our approach to cross-modal tasks highlights its potential for further research and development in a wide range of applications, from natural language processing to computer vision.

6. Limitations and Future Work

Our study integrates multi-scale features based on self-attention to fuse visual information of different granularities and employs Euclidean distance to measure the attention mechanism, improving the sequence modeling capability of videos. However, there are still limitations in this study. First, the feature fusion method adds time complexity, which leads to a slower speed. Next, although the multi-scale fusion method based on self-attention is sufficiently flexible, it incurs significant computational overhead. This limitation prevented us from conducting experiments on a broader range of scales (e.g., 14 × 14 and 28 × 28 feature maps). Videos, compared to images, require substantially more computation. On the other hand, this study focuses on feature extraction and understanding, while the VAC model’s experience indicates that aligning visual features with glosses yields better results. Future research will incorporate alignment learning to further investigate continuous sign language recognition.