Feature Maps Need More Attention: A Spatial-Channel Mutual Attention-Guided Transformer Network for Face Super-Resolution

Abstract

Recently, transformer-based face super-resolution (FSR) approaches have achieved promising success in restoring degraded facial details due to their high capability for capturing both local and global dependencies. However, while existing methods focus on introducing sophisticated structures, they neglect the potential feature map information, limiting FSR performance. To circumvent this problem, we carefully design a pair of guiding blocks to dig for possible feature map information to enhance features before feeding them to transformer blocks. Relying on the guiding blocks, we propose a spatial-channel mutual attention-guided transformer network for FSR, for which the backbone architecture is a multi-scale connected encoder–decoder. Specifically, we devise a novel Spatial-Channel Mutual Attention-guided Transformer Module (SCATM), which is composed of a Spatial-Channel Mutual Attention Guiding Block (SCAGB) and a Channel-wise Multi-head Transformer Block (CMTB). SCATM on the top layer (SCATM-T) aims to promote both local facial details and global facial structures, while SCATM on the bottom layer (SCATM-B) seeks to optimize the encoded features. Considering that different scale features are complementary, we further develop a Multi-scale Feature Fusion Module (MFFM), which fuses features from different scales for better restoration performance. Quantitative and qualitative experimental results on various datasets indicate that the proposed method outperforms other state-of-the-art FSR methods.

1. Introduction

Face super-resolution (FSR), also known as face hallucination [1], is a technology for enhancing low-resolution (LR) face images into high-resolution (HR) ones. Due to low-cost imaging equipment and limited imaging conditions, face images often have a lower spatial resolution, which severely degrades the performance of most practical downstream applications such as face analysis and face recognition. Therefore, FSR has become increasingly popular in the computer vision and image processing fields, making it an important scientific tool [2].

Unlike general image super-resolution, FSR focuses on recovering pivotal facial structures. Although facial structures only occupy a small portion of the face, they are crucial in distinguishing different faces. The first FSR method, proposed by Baker and Kanade [1], triggered the upsurge of traditional FSR methods. Over time, various traditional techniques for FSR have been developed, including the interpolation approach [3], PCA [4], convex optimization [5], Bayesian approach [6], kernel regression [7], and manifold learning [8]. Traditional methods are limited by their shallow structure and representation abilities, making them incompetent in producing plausible facial images.

Alongside the rise of deep learning techniques [2,9,10], deep convolution neural networks (CNNs) [11,12,13,14,15] have achieved remarkable advancements in improving face image quality. The first CNN-based FSR method was introduced by Zhou et al. [11], greatly improving FSR performance compared with traditional FSR methods. To further explore facial information, Cao et al. [12] utilized reinforcement learning to capture the interdependency among facial parts. Zhang et al. [13] introduced super-identity loss to assist the network in generating more accurately identified super-resolution face images. In contrast to the FSR methods above, which recover face images directly, Huang et al. [14] used the wavelet transform to project face images into wavelet spaces to capture rich contextual information, while Wang et al. [15] applied the Fourier transform to obtain an image-size receptive field for capturing global facial structure.

Inspired by the great success of generative adversarial networks (GANs) in the image processing field [16,17,18], Yang et al. [19] introduced a collaborative suppression and replenishment framework based on GANs. Gao et al. [20] proposed a conditional generative model based on the diffusion model, which replaces the U-Net in super-resolution to capture complex details and fine textures. Addressing the fact that GAN-based methods require greater computational resources, PCA-SRGAN [21] uses Principal Component Analysis decomposition, while SPGAN [22] employs supervised pixel-wise loss to ease the GAN training process.

Due to the highly structured nature of the human faces, many FSR methods leverage facial priors, e.g., face landmarks and parsing maps, to enhance the reconstruction performance. Chen et al. [23] utilized facial parsing maps to guide the end-to-end FSR convolution network. Meanwhile, Bulat et al. [24] combined GANs with a well-designed heatmap loss to constrain the face structure between HR and super-resolved (SR) face images. To better capture sharp facial structures in face images with large pose variations, Hu et al. [25] introduced 3D facial priors instead of the commonly used 2D ones. Considering the challenge of estimating priors from degraded LR face images directly, DIC [26] used an iterative process in which FSR and prior estimation were performed repeatedly to enhance FSR performance.

Recently, the attention mechanism has emerged as a new trend in computer vision tasks [27,28,29,30,31]. Chen et al. [32] devised a face attention unit to capture facial structure information, while Lu et al. [33] designed an external–internal split attention group to reconstruct clear facial images. Furthermore, the performance of transformers has already been proven and widely applied in computer vision tasks, such as image recognition [34,35] and image restoration [28,36,37]. The core of the transformers is a self-attention mechanism that can capture both long- and short-range correlations between words/pixels [38]. Therefore, transformer-based methods have gained dramatic attention for their high ability to capture both local and global dependencies.

While transformer-based methods have led to significant improvements in FSR performance, they have a number of limitations that need to be addressed. First, there is a lack of sufficient discussion regarding the inner feature maps. As shown in Figure 1b, without guidance, the features of the inner feature maps may not always be detail-rich, and may even be buried in gray. This restricts the transformer block to selecting only a limited number of feature maps based on the self-attention matrix (Figure 1c), while leaving others untouched. On the contrary, from Figure 1d it can be seen that utilizing a guiding block to enhance essential facial components results in more correlated output feature maps, benefiting the “one-to-many” FSR problem [39] and yielding superior outcomes (Figure 1e). Therefore, it is crucial to have a guiding block that can identify the essential facial component for inner feature maps in transformer-based methods. Second, most previous transformer-based methods [28,37] have utilized the same transformer structure for different feature layers. However, the transformer structure on high-level features cannot be thoroughly applied to low-level features [40], and vice versa, otherwise resulting in unsatisfactory FSR performance. Lastly, the usual spatial-wise transformers are limited to position-specific windows, meaning that the partition strategy may potentially alter the structure of the facial image [15].

Based on the above analysis, we propose a spatial-channel mutual attention-guided transformer network for face super-resolution, which explores the potential of inner feature maps for reconstructing plausible face images. The proposed method is a multi-scale connected encoder–decoder network. In the encoder–decoder branches, a Spatial-Channel Mutual Attention-guided Transformer Module (SCATM) is carefully designed to extract more detailed features by enhancing the relationship between inner feature maps. It is composed of a Spatial-Channel Mutual Attention Guiding Block (SCAGB) and a Channel-wise Multi-head Transformer Block (CMTB). Among them, SCAGB aims to guide the transformer block in identifying the essential facial component. SCAGB on the top layer (SCAGB-T) aims to guide and promote both local facial details and global facial structures, while SCAGB on the bottom layer (SCAGB-B) seeks to identify the crucial encoded features. Unlike the usual spatial-wise transformersm which are limited to position-specific windows, the CMTB utilizes feature map channels to achieve image-size receptive fields. The combination of SCAGB and CMTB is complementary, and can simultaneously promote both local facial details and global facial structures. Meanwhile, unlike pyramid super-resolution networks [41,42], which progressively reconstruct high-resolution images, we further develop a Multi-scale Feature Fusion Module (MFFM) which fuses features from all layers, making for network flexibility and better restoration performance.

Figure 1. Visual analysis of the inner feature maps trained with/without guiding blocks for transformer-based FSR methods: (a) the input face image; (b1) the inner feature maps without guiding blocks; (b2) the inner feature maps with guiding blocks; (c1) the self-attention heatmap without guiding blocks; (c2) the self-attention heatmap with guiding blocks; (d1) the correlation [43] between input and output feature maps without guiding blocks (please note that for fair comparison five different images are tested here instead of one); (d2) the correlation between input and output feature maps with guiding blocks; (e) the output images (the top one trained without guiding blocks and bottom one trained with guiding blocks).

Figure 1. Visual analysis of the inner feature maps trained with/without guiding blocks for transformer-based FSR methods: (a) the input face image; (b1) the inner feature maps without guiding blocks; (b2) the inner feature maps with guiding blocks; (c1) the self-attention heatmap without guiding blocks; (c2) the self-attention heatmap with guiding blocks; (d1) the correlation [43] between input and output feature maps without guiding blocks (please note that for fair comparison five different images are tested here instead of one); (d2) the correlation between input and output feature maps with guiding blocks; (e) the output images (the top one trained without guiding blocks and bottom one trained with guiding blocks).

In summary, the main contributions of this work are four-fold:

• We devise a spatial-channel mutual attention-guided transformer network for face super-resolution. To the best of our knowledge, this is the first paper to explore the potential of inner feature maps in reconstructing plausible face images in the transformer-based FSR area.
• We carefully design a Spatial-Channel Mutual Attention-guided Transformer Module (SCATM) to extract more detailed features by enhancing the relationship between inner feature maps. Thanks to its powerful modeling ability, both local facial details and global facial structures can be fully explored and utilized.
• We propose an elaborately designed Multi-scale Feature Fusion Module (MFFM) to fuse multi-scale features during the reconstruction process. This module is crucial in enabling our method to acquire a wide range of features, which in turn enhances the quality of the restoration performance.
• We conduct experiments to confirm the effectiveness of the proposed method. The results of our experiments, conducted on two frequently used benchmark datasets (CelebA [44] and Helen [45]) demonstrate that our method surpasses others in terms of both visual outcomes and quantitative measurements.

2. Proposed Method

Considering the vital role of inner feature maps in identifying the essential facial component for better FSR performance, we develop a spatial-channel mutual attention-guided transformer network for FSR, which is the first study to explore the potential of inner feature maps in reconstructing plausible face images in the transformer-based FSR area.

To better elaborate the proposed method, we divide the method into four subsections. In the first subsection, we briefly revisit the difference between feature maps and feature spaces, which is the key foundation of the proposed method. Then, we take an overall view of the architecture of the proposed method. Next, we focus on the main part of the proposed method, the Spatial-Channel Mutual Attention-guided Transformer Module (SCATM), which contains a Spatial-Channel Mutual Attention Guiding Block (SCAGB) and a Channel-wise Multi-head Transformer Block (CMTB). The combination of SCAGB and CMTB is complementary, and can simultaneously promote both local facial details and global facial structures. Finally, we introduce the Multi-scale Feature Fusion Module (MFFM), which fuses features from all layers for network flexibility and better restoration performance.

2.1. Revisiting Feature Maps and Feature Spaces

Feature maps and feature spaces are similar in several ways: CNN-based FSR methods utilize complex convolution layers to project LR images into inner “feature maps” and then into HR ones; on the other hand, manifold learning-based FSR methods, which belong to the traditional FSR method, project LR images into “feature spaces” and then into HR ones, assuming that LR and HR spaces share the same local geometry [46]. Many traditional FSR methods have been developed based on manifold learning [47,48,49], focusing on further enhancing the LR and HR space relationship. However, with the rise of CNNs, manifold learning has gained less attention due to the complicated nature of CNN structures, which are difficult to deliberate on. Yang et al. [50] proposed a manifold localized deep external compensation (MALDEC) network that references online big data to provide an accurate localization and mapping to the HR manifold for image super-resolution. Menon et al. [51] traversed the HR manifold spaces to search for images that suit the original LR image, then feed the downscaling loss to a generative model for image reconstruction. Chen et al. [52] proposed a homogenization projection in LR space and HR space to formulate FSR in a multi-stage framework. Guo et al. [39] claimed that limiting mapping spaces would benefit image super-resolution, and introduced a closed-loop dual regression network (DRN) containing an additional constraint. The above methods try to merge traditional manifold learning with CNN-based methods; however, they overlook the key role of inner feature maps (Figure 1), limiting their image super-resolution performance. Therefore, how to effectively handle inner feature maps is vitally important for a high-quality image reconstruction process.

2.2. Overview

The proposed method, illustrated in Figure 2, is a multi-scale connected symmetrical hierarchical network containing three stages: encoding, bottleneck, and decoding. The encoding stage aims to extract and promote both local facial details and global facial structures, while the bottleneck stage is designed to optimize the encoded low-level features. Finally, the decoding stage is introduced for multi-scale feature fusion and image reconstruction. For convenience of description, we denote ${\mathit{I}}_{\mathrm{LR}}$, ${\mathit{I}}_{\mathrm{SR}}$, and ${\mathit{I}}_{\mathrm{HR}}$ as the low-resolution (LR) images, super-resolved (SR) images, and ground-truth high-resolution (HR) images, respectively.

(1) Encoding Stage: The encoding stage aims to extract and promote both local facial details and global facial structures. First, a $3\times 3$ convolution layer is applied to extract shallow features. Here, we suggest using 32 output channels for optimal performance. It is essential to keep in mind that the number of output channels should exceed the number of input channels, while an excessive number of output channels may lead to a significant increase in computation complexity and a decrease in feature information storage effectiveness [53]. Afterwards, the extracted shallow features are passed through three encoding stages. Each stage contains a carefully designed Spatial-Channel Mutual Attention-guided Transformer Module-Top (SCATM-T) and a downsampling block. The SCATM-T contains a Spatial-Channel Mutual Attention Guiding Block-Top (SCAGB-T) and a Channel-wise Multi-head Transformer Block (CMTB), which are discussed in the following subsection. The downsampling block consists of three layers: a $3\times 3$ convolutional layer with a stride of 2, followed by a LeakyReLU activation function layer, then another $3\times 3$ convolutional layer with a stride of 1. Please note that the first convolutional layer doubles the feature channels while simultaneously halving the feature map size.

(2) Bottleneck Stage: The bottleneck stage contains a large number of encoded feature maps, though each map is relatively small compared to those in the encoding stage. To better utilize these feature map features in the decoding stage, we introduce the Spatial-Channel Mutual Attention-guided Transformer Module-Bottom (SCATM-B). Unlike the SCATM-T in the encoding stage, the guiding blocks in SCATM-B aim to further enhance the low-level encoded features. With the help of SCATM-Bs, the model can focus on a greater variety of facial structures and continuously strengthen different facial features.

(3) Decoding Stage: The decoding stage aims to reconstruct high-quality face images based on the previously extracted and refined multi-layer features. In the decoding stage, the low-level features are first fed to the upsampling block, which contains a $6\times 6$ transposed convolutional layer with a stride of 2 followed by a LeakyReLU activation function layer and a $3\times 3$ convolution layer with a stride of 1. The feature channels are halved and the feature map size is doubled in the first transposed convolutional layer, which is the opposite of the downsampling process in the encoding stage. Afterward, the upsampled features are combined with features from other scales by the Multi-scale Feature Fusion Module (MFFM), extending the network flexibility and resulting in better restoration performance. Then, the well-combined features are fed to the SCATM-T for further image detail enhancement. Lastly, a $3\times 3$ convolutional layer is utilized to convert the learned feature maps into the output face image ${\mathit{I}}_{\mathrm{Out}}$. The final SR face image output is ${\mathit{I}}_{\mathrm{SR}}={\mathit{I}}_{\mathrm{Out}}+{\mathit{I}}_{\mathrm{LR}}$. Please note that the LR face images in the paper have already been upsampled to the same size as the HR ones by bicubic interpolation.

Additionally, to optimize the FSR performance, the proposed model is supervised by minimizing the following pixel-level loss function:

$$\mathcal{L}=\frac{1}{N}\sum _{i=1}^N{∥{\mathit{I}}_{\mathrm{SR}}^i-{\mathit{I}}_{\mathrm{HR}}^i∥}_1$$

(1)

where N denotes the number of training images and ${\mathit{I}}_{\mathrm{SR}}^i$ and ${\mathit{I}}_{\mathrm{HR}}^i$ are the i-th SR and ground-truth HR face image in the training dataset, respectively.

2.3. Spatial-Channel Mutual Attention-Guided Transformer Module (SCATM)

The SCATM is the central part of the proposed method, which contains a Spatial-Channel Mutual Attention Guiding Block (SCAGB) and a Channel-wise Multi-head Transformer Block (CMTB). Based on the need for multi-scale feature maps to apply different suitable blocks for better feature extraction and enhancement, SCAGB is further divided into two distinct blocks: the Spatial-Channel Mutual Attention Guiding Block-Top (SCAGB-T) in the encoding–decoding stage, and the Spatial-Channel Mutual Attention Guiding Block-Bottom (SCAGB-B) in the bottleneck stage. SCATM-T aims to promote both local facial details and global facial structures, while SCATM-B seeks to optimize the encoded low-level features. Moreover, unlike the usual spatial-wise transformers, which are limited to position-specific windows, the CMTB utilizes feature map channels to achieve image-size receptive fields. The combination of SCAGB and CMTB is complementary, and can simultaneously promote both local facial details and global facial structures.

2.3.1. Spatial-Channel Mutual Attention Guiding Block-Top (SCAGB-T)

The SCAGB-T aims to locate and guide both local facial details and global facial structures for the following CMTB, the detailed architecture of which is shown in Figure 3a. First, the Hourglass Block [54], which has already proven its effectiveness in generating spatial attention maps [41], is utilized to capture facial landmark features such as the eyes, nose, and mouth. Moreover, aiming to guide the weights of different feature map channels, the channel attention (CA) network [29] is applied to select and pay more attention to feature map channels that are rich in features. Thanks to a carefully designed structure that mutualizes spatial and channel attention wisely, the SCAGB can guide the following CMTB to capture the essential part of the face images of all channels for better reconstruction results with more details. Afterwards, a $3\times 3$ convolutional layer followed by a sigmoid function is applied to generate the spatial-channel mutual attention map. Then, the input feature maps are multiplied element-wise by the attention map and fed to the subsequent transformer block with better extracted spatial features and promoted channel information. Aiming to eliminate the gradient vanishing problem, the residual connection with a full connection layer is applied between the input feature maps and the output of the channel attention network.

2.3.2. Spatial-Channel Mutual Attention Guiding Block-Bottom (SCAGB-B)

Unlike the above SCAGB-T, the SCAGB-B in the bottleneck stage aims to guide the enhancement of the low-level encoded features. The bottleneck stage contains a large number of encoded feature maps; however, each map is relatively small compared to the ones in the encoding stage. Therefore, it is essential to introduce a dynamic selection mechanism that allows each neuron to adjust its receptive field size adaptively. The dynamic selection mechanism introduced here, which is the major difference between SCAGB-T and SCAGB-B, is the selective kernel (SK) network [55] shown in Figure 3b. Given a sequence of feature maps, the SK network firstly conducts two convolution layers with different respective fields, followed by a batch normalization layer and a ReLU layer. The calculated upper and lower inner feature maps are denoted as $\mathbf{U}$ and $\mathbf{V}$, respectively. Then, the feature maps are integrated via an element-wise summation and sent through the global average pool (GAP) layer to generate channel-wise statistics with different respective fields. Afterwards, two full connection layers are applied to the inner feature maps to enable the guidance for the adaptive selections. Finally, a soft attention layer across different channels is applied to adaptively select the information from different respective fields. Assuming that the upper and lower input of the Select layer in Figure 3b is $F\left(\mathbf{U}\right)$ and that $G\left(\mathbf{V}\right)\in {\mathbb{R}}^{C\times 1}$, where $F(·)$ and $G(·)$ denote the previous inner feature map process and C denotes the number of channels of the inner feature map, the output weight is

$${\mathbf{w}}_c^{upper}=\frac{e^{F_c\left(\mathbf{U}\right)}}{e^{F_c\left(\mathbf{U}\right)}+e^{G_c\left(\mathbf{V}\right)}},{\mathbf{w}}_c^{lower}=\frac{e^{G_c\left(\mathbf{V}\right)}}{e^{F_c\left(\mathbf{U}\right)}+e^{G_c\left(\mathbf{V}\right)}},$$

(2)

where c in ${\mathbf{w}}_c^{upper}$ denotes the c-th element of the ${\mathbf{w}}^{upper}$, likewise ${\mathbf{w}}_c^{lower}$, $F_c\left(\mathbf{U}\right)$, and $G_c\left(\mathbf{V}\right)$.

The final attention maps of the SK network are obtained through the attention weights on the inner feature maps from various respective fields:

$${\mathbf{A}}_c={\mathbf{w}}_c^{upper}\times {\mathbf{U}}_c+{\mathbf{w}}_c^{lower}\times {\mathbf{V}}_c,{\mathbf{w}}_c^{upper}+{\mathbf{w}}_c^{lower}=1,$$

(3)

where $\mathbf{A}=\left[\begin{array}{cccc}{\mathbf{A}}_1,& {\mathbf{A}}_2,& \dots ,& {\mathbf{A}}_C\end{array}\right]$ denotes the output attention maps, ${\mathbf{A}}_c\in {\mathbb{R}}^{H\times W}$, and H and W denote the height and width of the feature maps, respectively.

2.3.3. Channel-Wise Multi-Head Transformer Block (CMTB)

After preprocessing the inner feature maps with the guiding blocks, there is still a high demand to effectively aggregate the previous feature information from various channels for high-quality face image restoration. Moreover, the usual spatial-wise transformers are limited to position-specific windows, where the partition strategy may potentially alter the structure of the facial image [15]. Therefore, we introduce the Channel-wise Multi-head Transformer Block (CMTB). It can achieve image-size receptive fields based on channels instead of position-specific windows and is more computation-friendly, making it a good match for the previous guiding blocks. As depicted in Figure 4, CMTB consists of a Channel-wise Multi-head Self-attention Network (CMSN) and a Gated-Dconv Feed-Forward Network (GDFN). The CMSN is the primary component of the CMTB, while the GDFN aims to encode information from spatially neighboring pixel positions to effectively learn local image structures.

The CMTB can achieve image-size receptive fields based on feature map channels instead of position-specific windows. Given feature maps ${\mathbf{X}}_{in}\in {\mathbb{R}}^{H\times W\times C}$ as the input of the CMSN, which are reshaped into tokens $\mathbf{X}\in {\mathbb{R}}^{HW\times C}$ based on channels (where H, W, and C, respectively denote the height, width, and channel numbers of the feature maps), then $\mathbf{X}$ is linearly projected to achieve three different matrices: query $\mathbf{Q}\in {\mathbb{R}}^{HW\times C}$, key $\mathbf{K}\in {\mathbb{R}}^{HW\times C}$, and value $\mathbf{V}\in {\mathbb{R}}^{HW\times C}$:

$$\mathbf{Q}=\mathbf{X}{\mathbf{W}}^{\mathbf{Q}},\mathbf{K}=\mathbf{X}{\mathbf{W}}^{\mathbf{K}},\mathbf{V}=\mathbf{X}{\mathbf{W}}^{\mathbf{V}},$$

(4)

where ${\mathbf{W}}^{\mathbf{Q}}$, ${\mathbf{W}}^{\mathbf{K}}$, and ${\mathbf{W}}^{\mathbf{V}}\in {\mathbb{R}}^{C\times C}$ are learnable parameters (biases are omitted here for simplification). Afterwards, $\mathbf{Q}$, $\mathbf{K}$ and $\mathbf{V}$ are split into N heads along the channel dimension: $\mathbf{Q}=\left[\begin{array}{ccc}{\mathbf{Q}}_1,& \dots ,& {\mathbf{Q}}_N\end{array}\right]$, $\mathbf{K}=\left[\begin{array}{ccc}{\mathbf{K}}_1,& \dots ,& {\mathbf{K}}_N\end{array}\right]$, $\mathbf{V}=\left[\begin{array}{ccc}{\mathbf{V}}_1,& \dots ,& {\mathbf{V}}_N\end{array}\right]$, where the dimension of each head is $d=C/N$. Therefore, the self-attention matrix for $hea{d}_i$ is

$${\mathbf{A}}_i=softmax\left({\sigma }_i{\mathbf{K}}_i^T{\mathbf{Q}}_i\right),hea{d}_i={\mathbf{V}}_i{\mathbf{A}}_i,$$

(5)

where ${\mathbf{K}}_i^T$ denotes the transposed matrix of ${\mathbf{K}}_i$. By implementing this reshape strategy, the size of the generated attention map will be $d\times d$ instead of $HW\times HW$, which greatly cuts down the computational complexity. Moreover, a learnable parameter ${\sigma }_i\in {\mathbb{R}}^1$ is introduced to further extend the flexibility of the network. Subsequently, the outputs of N heads are fed to the concatenate layer followed by a full connection layer, and the resulting attention matrix is then added with the values from the position embedding generator:

$$CMSN\left(\mathbf{X}\right)=(\underset{i=1}{\overset{N}{Concat}}\left(hea{d}_i\right))\mathbf{W}+f_p\left(\mathbf{V}\right)$$

(6)

where $\mathbf{W}\in {\mathbb{R}}^{C\times C}$ are learnable parameters and $f_p(·)$ represents the position embedding generator, which aims to encode the position information of different channel dimensions. It contains a $3\times 3$ depth-wise convolution layer with a stride of 1 followed by a GELU layer [56] and another $3\times 3$ depth-wise convolution layer with a stride of 1. Finally, the output feature maps ${\mathbf{X}}_{out}\in {\mathbb{R}}^{H\times W\times C}$ can be obtained by reshaping the result of Equation (6).

Moreover, we introduce GDFN [57] to encode information from spatially neighboring pixel positions in order to effectively learn local image structures. Given feature maps ${\mathbf{X}}_{in}\in {\mathbb{R}}^{H\times W\times C}$ as the input of the GDFN, the output ${\mathbf{X}}_{out}\in {\mathbb{R}}^{H\times W\times C}$ can be obtained by

$$\widehat{\mathbf{X}}={\mathbf{H}}_{dconv}^{3\times 3}\left({\mathbf{H}}_{fc}\left({\mathbf{X}}_{in}\right)\right),{\mathbf{X}}_{out}={\mathbf{H}}_{fc}(\widehat{\mathbf{X}}·\sigma \left(\widehat{\mathbf{X}}\right)),$$

(7)

where ${\mathbf{H}}_{dconv}^{3\times 3}(·)$ and ${\mathbf{H}}_{fc}(·)$ denote the $3\times 3$ depth-wise convolution layer and the full connection layer, respectively, while $\sigma (·)$ represents the GELU nonlinearity.

The SCAGB distills and guides the key features from inner feature maps, while the CMTB further aggregates and refines the previous feature information. These two blocks complement each other and gradually enhance the relationship between the inner feature maps. Benefiting from the SCAGB and CMTB, the SCATM can simultaneously promote both local facial details and global facial structures, offering a promising solution to the challenging face image reconstruction task.

2.4. Multi-Scale Feature Fusion Module (MFFM)

Successive pyramid super-resolution networks [41,42] have already proven the importance of multi-scale feature information in the image reconstructing process; however, the pyramid methods mentioned above progressively reconstruct high-resolution images from neighboring layers, ignoring the others. Aiming to further utilize the multi-scale feature information and to provide the network with better feature representation capabilities, the Multi-scale Feature Fusion Module (MFFM), the details of which shown on the bottom side of Figure 1, is introduced here. First, the multi-scale feature map sizes are unified to the same size as the target feature map layer. Considering that the magnification scale between different layers is always $2^n(n\in \mathbb{Z})$, a $3\times 3$ convolution layer with a stride of 2 and a $6\times 6$ transposed convolution layer with a stride and padding of 2 are introduced for /2 downscaling and ×2 upscaling processes, respectively. Moreover, for a larger magnification scale such as /4 or ×4, double $3\times 3$ convolution or $6\times 6$ transposed convolution layers, etc., would be applied. After unifying all required feature map sizes, the multi-scale feature maps are concatenated to pass through a full connection layer, followed by the channel attention network, the details of which can be viewed in Figure 3a. Finally, the well-handled multi-scale feature information is added to the target feature map layer in the encoding stage.

3. Experiments

3.1. Dataset and Metrics

In our experiments, the model was trained on the CelebA [44] dataset and evaluated on the CelebA and Helen [45] datasets along with the real face images. In the data preprocessing phase, images were simply cropped to a size of $128\times 128$ based on their center point and then treated as the ground truth. Afterwards, $16\times 16$ LR face images were obtained from the ground truth images with a /8 downscaling bicubic operation. It should be noted that the model does not require any additional facial landmarking on the dataset for training. Then, we trained the model on 18,000 face images from the CelebA dataset and evaluated its performance on 1000 face images from the same dataset along with 50 face images from the Helen dataset. Moreover, to prove the flexibility of the model, we directly evaluated the model on the Helen dataset and real face images with the model trained on CelebA. It should be noted that the proposed deep learning method cannot be trained with a single image. Thus, sufficient face images must be provided should anyone want to train it on their own dataset.

To objectively evaluate the quality of the SR results, three image quality assessment metrics are employed here: the Peak Signal-to-Noise Ratio (PSNR) [58], Structural Similarity (SSIM) [59], and Learned Perceptual Image Patch Similarity (LPIPS) [60].

3.2. Implementation Details

All experiments were conducted in PyTorch [61] with an GeForce RTX 4090 24 GB graphic card made by NVIDIA from Santa Clara, CA, USA. The proposed model was optimized by Adam with $\beta 1=0.9$, $\beta 2=0.99$, and a learning rate of $2\times {10}^{-4}$.

3.3. Ablation Studies

To evaluate the effectiveness of different modules in the model, we conducted a series of ablation studies on the CelebA test sets for ×8 SR.

(1) Study on SCATM-T: The SCATM-T, which consists of an SCAGB-T and a CMTB, aims to extract and promote both local facial details and global facial structures. It is the first attempt to explore the potential of inner feature maps in reconstructing plausible face images in the transformer-based FSR area. To verify the effectiveness of SCATM-T and its components, we designed three test models by removing different module parts. The test results are shown in

Table 1. Ablation study of the components in the proposed SCATM-T. Please note that the best results are emphasized with bold in the experiment part for better visualization.

SCAGB-T	CMTB	PSNR↑	SSIM↑	LPIPS↓
×	×	27.30	0.7833	0.2047
×	✓	27.55	0.7897	0.1831
✓	×	27.54	0.7891	0.1839
✓	✓	27.63	0.7905	0.1797

.

From the table, it can be observed that:

(a)

The performance of SCATM-T without any components (i.e., removing SCATM-T) decreased dramatically. This is because the proposed model structure is much shallower without the SCATM-T, making it challenging to refine features. Moreover, without the processed fine-detailed feature maps from the SCATM-T, the Multi-scale Feature Fusion operation Module (MFFM) is greatly affected.

(b)

The quantitative metrics of SCATM-T with one single component inside are better than the no-component one above, demonstrating that both SCAGB-T and CMTB can enhance the representation ability of the model. However, SCATM-T with only SCAGB-T cannot guide anything, while SCATM-T with only CMTB cannot focus on feature map parts that are rich in features, limiting its performance.

(c)

Equipped with both of the carefully designed components SCAGB-T and CMTB, SCATM-T achieves the best performance in terms of all evaluation matrices, which proves that the combination of SCAGB-T and CMTB is complementary and can simultaneously promote both local facial details and global facial structures.

(2) Study on SCATM-B: The SCATM-B, which contains an SCAGB-B and a CMTB, aims to enhance the low-level encoded features. Due to their comparable inner structures, we conducted similar experiments to the ones reported in the “Study on SCATM-T” part, the results of which are shown in

Table 2. Ablation study of the components in the proposed SCATM-B.

SCAGB-B	CMTB	PSNR↑	SSIM↑	LPIPS↓
×	×	27.50	0.7877	0.1897
×	✓	27.60	0.7902	0.1811
✓	×	27.59	0.7899	0.1817
✓	✓	27.63	0.7905	0.1797

. Similar observations and conclusions to those made in the previous section can be derived. However, it is worth noting that the performance of the model without SCATM-B is better than that without SCATM-T. This is because the SCATM-T and the MFFM are more complementary compared with the relationship between the SCATM-B and the MFFM; with one removed, the performance of others will also be greatly impacted.

Moreover, we evaluated the influence on model performance of the number of SCATM-Bs, the results of which are shown in

Table 3. Performance comparison of different SCATM-B numbers in the proposed method.

SCATM-B Numbers	PSNR↑	SSIM↑	LPIPS↓
0	27.50	0.7877	0.1897
2	27.57	0.7900	0.1823
4	27.63	0.7905	0.1797
6	27.62	0.7903	0.1804

. From the table, it can be observed that the model performs poorly without any SCATM-Bs, which proves that the SCATM-B plays a vital role in the model. Meanwhile, the performance of the model rises with the increase in the number of SCATM-Bs in a particular range. However, as the number of SCATM-Bs increases and exceeds four, the evaluation matrix change rate slows down and even decreases slightly. Additionally, the model size increases, which leads to an upsurge in the computational complexity of the model. Therefore, the number of SCATM-Bs is set to four for a good balance between model size and performance.

(3) Study on MFFM: MFFM is specially designed to further utilize the multi-scale feature information for better FSR performance. In this part, we design three different test models to demonstrate the effectiveness of the MFFM, the results of which are shown in

Table 4. Performance comparison of different approaches to the multi-scale feature fusion process.

Approaches	PSNR↑	SSIM↑	LPIPS↓
Not Applied	27.54	0.7885	0.1873
Only Add	27.58	0.7892	0.1838
Only Concat	27.59	0.7895	0.1832
Our MFFM	27.63	0.7905	0.1797

.

From the table, it can be observed that: (a) The model without the MFFM performs the worst, which proves the importance of multi-scale features in the image reconstruction process. (b) An add operation or a concatenate layer to fuse multi-scale features does benefit the performance of the FSR model; however, they are too simple to take responsibility for the multi-scale feature fusion part. (c) With the carefully designed MFFM, the model achieves the best performance in terms of PSNR, SSIM, and LPIPS, which proves that a suitable feature fusion strategy such as MFFM can benefit the image reconstruction process.

3.4. Comparison with the State-of-the-Art

To verify the superiority of the proposed method, in this section we compare our method with several state-of-the-art methods, including two GAN-based methods (SRResNet [16] and RCAN [29]), three attention-based methods (SPARNet [32], SISN [33], and IGAN [30]), and two pioneering transformer-based methods (SwinIR [37] and Uformer [28]) on the CelebA and Helen datasets along with real face images. Moreover, bicubic interpolation is applied as the baseline. All models were trained using the same CelebA dataset for fair comparison. Quantitative results are tabulated in

Table 5. Quantitative comparisons for $\times 8$ SR on the CelebA and Helen test sets.

Methods	CelebA			Helen
	PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓
Bicubic	23.44	0.6180	0.5900	23.79	0.6739	0.5254
SRResNet [16]	26.08	0.7502	0.2131	25.47	0.7828	0.2308
IGAN [30]	26.99	0.7801	0.2201	26.37	0.7996	0.2245
RCAN [29]	26.99	0.7796	0.2249	26.39	0.7965	0.2359
SISN [33]	26.85	0.7738	0.2337	26.33	0.7974	0.2322
SPARNet [32]	26.95	0.7794	0.2211	26.38	0.7953	0.2314
SwinIR [37]	27.15	0.7850	0.2162	26.48	0.7917	0.2413
Uformer [28]	27.33	0.7884	0.2040	26.67	0.8009	0.2063
Ours	27.63	0.7905	0.1797	26.97	0.8069	0.1945

.

(1) Comparison on CelebA dataset: Quantitative comparisons with other methods on the CelebA dataset are shown in Table 5. According to the table, the proposed method outperforms other competitive methods in terms of PSNR, SSIM, and LPIPS, which proves that the proposed method has the advantage of recovering accurate and realistic face details. Moreover, we provide some test images from the CelebA dataset for visual comparisons, shown in Figure 5. From the first two samples in Figure 5, it can be seen that our method can better restore nose contours and eye details compared to other state-of-the-art methods, while avoiding unpleasant artifacts. This is due to the combination of SCAGB and CMT, which complement each other and can simultaneously enhance both local facial details and global facial structures. Furthermore, despite all the methods mentioned above achieving satisfactory evaluation metrics on the last sample, the visual results are not very compelling due to the inability to reconstruct eyeglasses. Therefore, adding face identifications in the FSR process that can help to identify lost details such as eyeglasses in the LR image represents a good opportunity for future work.

(2) Comparison on Helen dataset: In this part, we directly evaluate the model on Helen datasets with the model trained on CelebA to prove the flexibility of the model. Quantitative and visual comparisons with other methods on the Helen dataset are shown in Table 5 and Figure 6, respectively. According to the results, our proposed method has the advantage of recovering facial images both quantitatively and qualitatively, which demonstrates the robustness and stability of our method. However, it is worth noting that the methods mentioned above showed varying degrees of performance decrease on the Helen test set as compared to the CelebA test set. Therefore, digging into the inner differences among various datasets is a good choice for enhancing the generality of FSR methods, especially for real face image restoration.

(3) Comparison on real face images: Due to the fact that real face images are captured from a variety of complex environments that the aforementioned CelebA dataset cannot simulate, restoring face images from the real world is always a considerable challenge. Aiming to validate the effectiveness of our proposed method in real-world face images, we conducted experiments on low-quality real-world face images collected from the classic “I Love My Family” TV series. This is a popular sitcom made in the 1990s that suffers severe issues with low resolution due to the use of outdated imaging equipment, making it perfect for testing. Experiments were conducted with the aim of reconstruct more detailed facial images with appealing facial structures, the reconstructed results of which are shown in Figure 7. Benefiting from the guiding blocks and transformers in the proposed method, which complement each other and can simultaneously enhance both local facial details and global facial structures, our method achieves appealing performance with reasonable results compared to other state-of-the-art methods.

3.5. JPEG Artifacts Analysis

To further prove the robustness of the proposed method on FSR tasks, we introduced JPEG artifacts to blur face images and tested these images without training another network based on JPEG artifacts. The images with JPEG artifacts were generated using the “JPEG Artifact Generator v1.0” [62] with the “Base JPEG compression” set to 0.2. We also selected two representative methods, Uformer and IGAN, as the comparative methods and the bicubic method as the baseline. The results are shown in Figure 8.

As the figure illustrates, all models, including ours, suffer performance degradation due to the lack of training with JPEG artifacts. However, our proposed method leveraging guiding blocks to preserve face features still manages to outperform the others. This is particularly evident in the fourth-row images.

3.6. Model Complexity Analysis

The proposed method has proven its superior ability in quantitative and qualitative FSR performance based on the previous experiments. In this section, we compare the model performance, size, and execution time of our approach with other state-of-the-art methods. The results are shown in Figure 9. From the figure, it can be observed that our method achieves the best quantitative results while maintaining comparable execution time and model size, which makes it a possible choice for FSR tasks.

4. Conclusions

In contrast to existing transformer-based approaches that focus on introducing sophisticated structures, we propose a spatial-channel mutual attention-guided transformer network for face super-resolution. This is the first study to explore the potential of inner feature maps for reconstructing plausible face images in the transformer-based FSR area. The proposed method is a multi-scale connected encoder–decoder network. The primary component of the network is a Spatial-Channel Mutual Attention-guided Transformer Module (SCATM), which is composed of a Spatial-Channel Mutual Attention Guiding Block (SCAGB) and a Channel-wise Multi-head Transformer Block (CMTB). The SCATM on the top layer (SCATM-T) aims to promote both local facial details and global facial structures, while the SCATM on the bottom layer (SCATM-B) seeks to optimize the encoded low-level features. Unlike the usual spatial-wise transformers, which are limited to position-specific windows, the CMTB utilizes feature map channels to achieve image-size receptive fields. Moreover, we develop a Multi-scale Feature Fusion Module (MFFM), which fuses features from different scales for better restoration performance. Extensive experiments on both simulated and real-world datasets demonstrate that the proposed method can achieve state-of-the-art performance.