Ancient Painting Inpainting with Regional Attention-Style Transfer and Global Context Perception

Abstract

Ancient paintings, as a vital component of cultural heritage, encapsulate a profound depth of cultural significance. Over time, they often suffer from different degradation conditions, leading to damage. Existing ancient painting inpainting methods struggle with semantic discontinuities, blurred textures, and details in missing areas. To address these issues, this paper proposes a generative adversarial network (GAN)-based ancient painting inpainting method named RG-GAN. Firstly, to address the inconsistency between the styles of missing and non-missing areas, this paper proposes a Regional Attention-Style Transfer Module (RASTM) to achieve complex style transfer while maintaining the authenticity of the content. Meanwhile, a multi-scale fusion generator (MFG) is proposed to use the multi-scale residual downsampling module to reduce the size of the feature map and effectively extract and integrate the features of different scales. Secondly, a multi-scale fusion mechanism leverages the Multi-scale Cross-layer Perception Module (MCPM) to enhance feature representation of filled areas to solve the semantic incoherence of the missing region of the image. Finally, the Global Context Perception Discriminator (GCPD) is proposed for the deficiencies in capturing detailed information, which enhances the information interaction across dimensions and improves the discriminator’s ability to identify specific spatial areas and extract critical detail information. Experiments on the ancient painting and ancient Huaniao++ datasets demonstrate that our method achieves the highest PSNR values of 34.62 and 23.46 and the lowest LPIPS values of 0.0507 and 0.0938, respectively.

1. Introduction

Ancient paintings, as a form of art that integrate architecture with color and form, are subject to damage in their local regions due to environmental influences, major natural disasters, and the risk of intentional destruction [1]. However, because of influences from the environment, significant calamities of nature, and the risk of deliberate damage, this treasure trove of artistic legacy has become exceedingly delicate. Traditional ancient painting inpainting works [2] are mainly done by experienced experts using manual reproduction methods. This approach relies on the expertise of the restorer but is also inefficient. More importantly, there may be subjective misunderstanding of the contents of ancient paintings, leading to irreversible damage to them. Ancient painting inpainting has recently emerged as a fundamental research topic in computer vision [3].

Since the advent of deep learning [4], numerous studies in image processing and computer vision have leveraged its powerful capabilities for feature extraction and learning, achieving groundbreaking results. The integration of deep learning technology into the domain of image inpainting has addressed the limitations of traditional methods, which often fail to consider comprehensive image information. Among these advancements, generative adversarial networks (GANs [5]) stand out due to their distinctive adversarial approach and superior generative capabilities. They have been extensively applied across various image inpainting applications. GAN-based image inpainting technology [6], which combines a potent generative network with a discriminative network, has demonstrated significant advantages. The generative network is tasked with creating realistic image content, while the discriminative network assesses the authenticity of the output. This adversarial training significantly enhances the quality and precision of the details in the restored images. Because the U-Net [7] network’s excellent feature fusion can learn different levels of information in an image, researchers have tried to apply it to image inpainting tasks and have achieved excellent inpainting results. In addition, Ou et al. [8] used GAN for image refinement, combined with the use of the TranMSMrmer [9] module to restore refinement features for superior performance in image recovery tasks. Abderrazak et al. [10] proposed an antagonistic generation network based on universal multi-core filters to extract more relevant and discriminating features related to image tasks. Sun and others [11] proposed an edge-guided ancient painting inpainting method using edge image guidance network training through self-attention to improve the attention to edge and detail features. Furthermore, Mardieva et al. [12] demonstrated significant improvements in image quality using a lightweight image super-resolution approach with deep residual feature distillation, which is relevant to enhancing performance in image restoration tasks.

Ancient painting inpainting requires a delicate balance between preserving the original artistic integrity and addressing the damages incurred over time. While considerable progress has been made in the field, existing GAN-based methods often struggle to effectively capture intricate details and maintain semantic coherence, especially in severely damaged or missing regions. To address these challenges, we propose a novel ancient painting inpainting network, RG-GAN, which introduces three key modules—RASTM (Regional Attention-Style Transfer Module), MFG (Multi-scale Fusion Generator), and GCPD (Global Context Perception Discriminator)—each designed to target specific limitations of prior inpainting methods. Firstly, RASTM improves the preservation of stylistic consistency by incorporating regional attention mechanisms, ensuring that transferred styles match the original artistic context more closely. Secondly, MFG enhances the reconstruction of fine-grained details across various scales, addressing the challenge of perceptual inconsistency between the inpainted areas and the original artwork. Finally, the GCPD further strengthens the model’s ability to capture global contextual relationships, enabling a more accurate and detailed inpainting, even in areas with extensive damage. By explicitly addressing these gaps, RG-GAN offers a more comprehensive and nuanced solution to ancient painting inpainting. The experimental results on both the ancient paintings and the ancient Huaniao++ datasets demonstrate that our method achieves superior semantic coherence and detail restoration compared to existing approaches.

The contributions of this article can be summarized as follows:

• To achieve an accurate match between the features of the missing region and the normal region, a regional attention-style transfer module is designed, which uses an attention mechanism to transfer the feature distribution of the content in the normal area to the missing area, so as to achieve fine-grained alignment between the content in the missing area and the content in the normal area, thus achieving complex style transfer while maintaining the authenticity of the content.
• A new Multi-scale Fusion Generator (MFG) is designed. The encoder uses the multi-scale residual downsampling module to reduce the size of the feature map and at the same time effectively extracts and integrates the features of different scales, which provides a rich feature representation for the network. Then, similar semantic features of different scales extracted from the encoder are fused, and information interaction is carried out through a skip connection.
• We propose the Multi-scale Cross-layer Perception Module (MCPM) that leverages the deep semantic features across multiple scales from the encoder, enriching the feature representation of the inpainted areas. By integrating multi-level semantic information from the encoder, the decoder can fully exploit the encoder’s rich features, leading to more coherent and detailed inpainting details.
• We propose the Global Context Perception Discriminator (GCPD) that uses the feature extractor and global attention mechanism designed in order to deepen the feature extraction process so that the network can capture more subtle and complex features of ancient painting images. At the same time, it can effectively retain the information of channels and spatial dimensions, thus strengthening cross-dimensional information interaction. The design of the GCPD significantly improves the discriminator’s ability to identify specific spatial areas and extract critical detail information.

2. Related Work

2.1. Ancient Painting Inpainting

Ancient painting is not only of great artistic value, it is also the testimony of history and culture, recording the development of human society and reflecting the social landscape of different historical periods [13]. The preservation of these precious works of art is important because they are an irreplaceable reserve of civilization and cultural diversity. Traditional ancient painting inpainting methods have a long history, which relies mainly on the hand-craftsmanship of restorers and their deep understanding of materials and techniques. The inpainting process usually involves careful examination of the painting, as well as cleaning, decontaminating, coloring, repairing damage to the canvas or paper, etc. These methods emphasize respect for the original. Traditional inpainting methods have accumulated a wealth of experience in practice, but there are certain limitations, such as the high demand for inpainting skills and the length of time needed for the inpainting process.

As deep learning has swiftly advanced [14,15], it has demonstrated an unmatched ability to revitalize the domain of ancient painting inpainting [3]. Ancient painting inpainting requires not only the inpainting of their physical form but also the preservation of their artistic value and historical information. Deep learning enables the intelligent inpainting of missing regions [16]. The filled content should not only integrate visually with the original but also maintain a high degree of consistency in artistic style and historical context [8]. Refs. [3,17] utilized a neural network to identify and understand complex features in ancient paintings, such as textures, edges, and styles, to perform in-depth analysis and highly accurate inpainting of ancient paintings. In addition, generative adversarial networks (GANs [5]) have proven to be adept at the inpainting of ancient paintings. Comprising a generator that produces the reconstructed image and a discriminator that assesses the image’s genuineness, GANs employ an adversarial training approach. This dynamic balance ensures the restored paintings are visually indistinguishable from their authentic counterparts [18]. Additionally, ancient paintings often have fine lines and complex patterns. GAN can learn these detailed features and generate them seamlessly based on the original painting for the inpainting part [19]. Achieving such high-quality inpainting is challenging with conventional methods. The evolution of ancient painting inpainting methods exemplifies a transition from artisanal skills to the incorporation of contemporary technological advancements.

2.2. Multi-Scale Fusion

Multi-scale fusion [20] plays a pivotal role in image inpainting within the framework of generative adversarial networks (GANs). This approach operates at various resolution levels to comprehensively reconstruct both the structure and content of an image. At a coarser scale, the network initially captures and reconstructs the fundamental framework and principal elements of the image, ensuring macroscopic accuracy and consistency of the inpainting results [21]. Subsequently, during downscaling, the network concentrates on the recovery of subtle image features such as edges, textures, and local patterns. These detailed features are essential for enhancing the authenticity of the restored images [22]. In image inpainting, multi-scale fusion typically encompasses a spectrum of features from coarse to fine, facilitating multi-level processing [23]. At the lower levels, the focus shifts towards the meticulous reconstruction of details and textures. The advantage of multi-scale fusion lies in its ability to balance the significance of global and local features, thereby generating richer and more realistic image outputs. The application of multi-scale fusion in GAN-based image inpainting not only significantly enhances the visual quality and detail richness of the restored images but also contributes to improved model stability and training efficiency. This is particularly beneficial when addressing complex and large-scale image inpainting tasks [24].

3. RG-GAN

3.1. Overview

RG-GAN is tailored to restore images with a high degree of fidelity, particularly focusing on ancient paintings where the preservation of historical and artistic integrity is paramount. As shown in Figure 1, RG-GAN consists of a multiscale fusion generator and a global context perception discriminator. The encoder of the generator is responsible for converting damaged or incomplete ancient images into a set of feature representations. The decoder of the generator then uses it to reconstruct the image and fill in the missing areas [25]. It ensures that the high-level semantic features captured by the encoder are effectively transmitted and maintained during the reconstruction process, thus solving the problems of semantic incoherence and missing high-level feature information. The discriminator is used to identify subtle differences between the inpainting results and the real ancient painting images [26] and provide feedback to the generator to guide it to generate more realistic and consistent images, further improving the quality and artistry of the inpainting results.

3.2. Multi-Scale Fusion Generator

In image inpainting, the U-Net architecture typically faces challenges, particularly when processing images with missing areas [27]. This is because the skip connections have almost no input signal in the center of the missing area, making it difficult to effectively pass texture information to the decoder, which can result in texture blurring and structural distortion in the repaired image. Additionally, the U-Net model has deficiencies in both global semantic consistency and local feature continuity, leading to semantically incoherent inpainting results.

In this paper, we propose an innovative multi-scale fusion generator called the Multi-scale Fusion Generator (MFG). By designing a regional attention-style transfer module, it realizes fine-grained alignment between the content in the missing area and the content in the normal area, thus achieving complex style transfer while maintaining the authenticity of the content. This enhancement improves the quality of information passed through the skip connections. The MFG further leverages multi-scale fusion technology to progressively integrate filled high-level features with low-level features. This approach ensures that basic features can access and integrate semantic insights from advanced-level features, thereby improving the conveyance and application of feature data. Additionally, the MFG decoder outputs a repaired image at each scale, which is used to calculate pyramid losses, optimizing the filling effect on the missing area at each scale. The multi-scale fusion generator, which combines regional attention-style transfer modules, can adapt to various image inpainting scenarios, thereby improving the quality of inpainting.

As shown in Figure 1, MFG consists of six layers of multi-scale residual downsampling (MRD), which reduces the size of the input feature map from 128 × 128 to 4 × 4. The multi-scale residual downsampling module is shown in Figure 2. The decoder part also contains a six-layer structure and interacts with the encoder through a connection. The MRD module consists of two branches. The left branch uses the 1 × 1 convolution module and the 3 × 3 depth convolution module to extract fine-grained features, while the right half uses the depth convolution module to improve the inpainting function of ancient painting images while reducing the model parameters. Finally, the output features of the two branches are fused, effectively extracting and integrating features of different scales, providing rich feature representation for the network and reusing input features when establishing residuals. MRD can be expressed as:

$$\mathrm{X}1={Conv}_1\left({Conv}_1\left({Conv}_3\left({Conv}_1\left(\mathrm{F}\right)\right)\right)\right)\oplus \mathrm{F},$$

(1)

$$\mathrm{X}2={PWConv}_1\left({DWConv}_3\left(F\right)\right),$$

(2)

$$\mathrm{X}={Conv}_1(Cat(\mathrm{X}1,\mathrm{X}2)),$$

(3)

where DWConv₃ refers to the 3 × 3 depth-wise convolution, PWConv₁ denotes the 1 × 1 pointwise convolution, Conv₃ and Conv₁ represent the standard 3 × 3 and 1 × 1 convolutions, respectively, Cat is the concatenation operation, and ⊕ indicates element-wise addition.

3.3. Multi-Scale Cross-Layer Perception Module

Regional Attention-Style Transfer Module (RASTM): To more accurately match the characteristics of the missing area and the normal area, it is ensured that the style of the missing image area is consistent with that of the undamaged image. This paper designs a Regional Attention-Style Transfer Module (RASTM), as shown in Figure 3. Firstly, the input feature F is multiplied by the Mask to obtain the missing area content, and the input feature F is multiplied by the 1-Mask to obtain the normal area style content. Then, the attention mechanism is used to transfer the feature distribution of the content in the normal area to the missing area, in order to achieve fine-grained alignment between the content in the missing area and the content in the normal area. We then ensure that the contents of the missing area and the normal area are more accurately matched in different spatial positions in order to realize complex style transfer while maintaining the authenticity of the content.

Conventional multi-scale fusion mechanisms often convey feature data from only a single layer of the encoder to the decoder, thereby limiting the low-level features’ ability to discern semantic cues from higher-level features. This limitation leads to discontinuity in both the features and the semantic context within the restored area. To overcome this limitation, we adopt an improved multi-scale fusion mechanism that enhances the transfer of feature maps of missing areas filled by the Regional Attention-Style Transfer Module (RASTM). This enables the decoder to use the encoder’s multi-layer features more effectively, thereby improving the lack of high-level feature information in the inpainting results.

Multi-scale Cross-layer Perception Module (MCPM): As shown in Figure 4, the MCPM progressively incorporates the already filled high-level features into the low-level features, thereby improving feature integration. For example, let F1 represent high-level features in the encoder and F2 and F3 represent mid-level features. After processing by RASTM, SF1, SF2, and SF3 are obtained. Then, SF1 is sampled and fused with SF2. The feature is refined using 1 × 1 convolution and 3 × 3 convolution operations, followed by Softmax extraction to obtain DF2, which is then passed to the decoder and continues to be sampled to blend with subsequent feature maps. This skip connection not only transmits information from a single feature map but also facilitates the low-level feature’s perception of the semantic information of the high-level feature.

Firstly, the continuous features are calculated by the regional attention-style transfer module as follows:

$$\mathrm{SF}1=\mathrm{RASTM}\left(\mathrm{F}1\right),$$

(4)

$$\mathrm{SF}2=\mathrm{RASTM}\left(\mathrm{F}2\right),$$

(5)

$$\mathrm{SF}3=\mathrm{RASTM}\left(\mathrm{F}3\right),$$

(6)

where RASTM is the Regional-Attention Style Transfer Module, which uses attention mechanisms to enhance specific regional features during style transfer, and where SF1, SF2, and SF3 are the features obtained from the RASTM.

Secondly, fusion features are obtained using skip connections and convolution operations as follows:

$$\mathrm{DF}2=Softmax\left({Conv}_3\right({Conv}_1(\mathrm{SF}2\oplus \mathrm{Up}\left(\mathrm{SF}1\right))\left)\right),$$

(7)

$$\mathrm{DF}3=Softmax\left({Conv}_3\left({Conv}_1(\mathrm{SF}3\oplus \mathrm{Up}\left(\mathrm{SF}2\right))\right)\right),$$

(8)

where ⊕ denotes element-wise addition at the pixel level, Up refers to upsampling, and Softmax is the activation function used to normalize the output into a probability distribution.

3.4. Global Context Perception Discriminator

In generative adversarial networks, the discriminator is critical in evaluating the authenticity of the generated image and in providing feedback to guide the generator in creating more realistic images. To improve this process, a Global Context Perception Discriminator (GCPD) is proposed. It utilizes a global attention mechanism to preserve information across channels and spatial dimensions, reinforcing the importance of cross-dimensional interactions. This enables the discriminator to more accurately understand and evaluate inpainted images, thereby enhancing the model’s performance in diverse inpainting tasks.

The structure design of GCPD is shown in Figure 5, which consists of several key parts: feature extractor, global attention module (GAM), convolution module, and full connection layer FC. Specifically, the feature extractor is composed of two branches—the left branch is composed of three continuous 3 × 3 convolution layers, which are used to extract initial visual features from the input ancient painting images, and the right branch further deepens the feature extraction process by using 3 × 3 and 1 × 1 convolution so that the network can capture more subtle and complex ancient painting image features. Then, the features extracted from the two branches are fused, and the number of channels is adjusted by a 1 × 1 convolution. Then, the global attention mechanism is applied to capture the key multi-dimensional features of the image and retain the information of the channel and spatial dimensions. This design not only reduces the model size of the convolutional neural network but also improves its performance, making GCPD more efficient and accurate in evaluating generated images.

As depicted in Figure 6, the global attention module (GAM) operates by sequentially integrating channel attention with spatial attention. It initiates the process with a three-dimensional configuration designed to preserve the volumetric features of the image through channel attention components. Then, the inter-dimensional channel and spatial correlations are amplified through the application of a multi-layer perceptron (MLP). This enhancement is fused with the initial input features via a residual link, which serves to boost the feature’s descriptive power.

In the spatial attention section, two 7 × 7 convolution layers are used to effectively capture and blend spatial information. GAM is designed to accurately identify and emphasize key channel and spatial features in images, providing a richer and more accurate representation of features for subsequent image processing tasks, which can be expressed as:

$$\mathrm{F}1=\sigma (InversePermute(MLP\left(\mathrm{Permute}\right(F\left)\right)\left)\right)\odot F,$$

(9)

$$\mathrm{F}2=\sigma \left({Conv}_7\left({Conv}_7\left(\mathrm{F}1\right)\right)\right)\odot \mathrm{F}1,$$

(10)

where $\sigma$ is the sigmoid activation function, whose mathematical definition is $\sigma \left(x\right)={\displaystyle \frac{1}{1+e^{-x}}}$, MLP refers to a shared perception machine, and Permute and InversePermute are used to rearrange the order of feature dimensions. Specifically, Permute reorders the dimensions of the input tensor to facilitate certain operations, and InversePermute restores the original dimension order after processing. The symbol ⊙ represents an element-by-element (or pixel-wise) multiplication, and ${Conv}_7$ denotes convolutions with a 7 × 7 kernel size.

3.5. Loss Function

Within the framework of the RG-GAN, the design of the loss function includes several components due to the innovative integration of a regional attention style transfer and multi-scale fusion technology, as well as the multi-scale color repairing of the image using the decoder output. Specifically, the loss function consists of several key components, including adversarial and reconstruction losses.

3.5.1. Adversarial Loss

In GAN training, the discriminator may not be updated after the optimal state is reached, resulting in erratic training. To solve this problem, WGAN [28] was introduced, which uses weight clipping to stabilize training. However, this approach has its limitations because it relies on a fixed range of weights, limiting the expressive power of the network. Further proposed is WGAN-GP [29], which enhances the stability of the network by optimizing the target function by setting separate penalty terms for each sample. In this paper, WGAN-GP is used to calculate the adversarial loss to achieve more stable and efficient image generation training, balancing the generator and discriminator to promote high-quality image generation.

$$L_{\mathrm{adv}}=\mathbb{E}\left[\mathrm{D}\left(\mathrm{x}\right)\right]-\mathbb{E}\left[\mathrm{D}\left(\mathrm{G}\left(\mathrm{z}\right)\right)\right]+{\lambda }_{\mathrm{gp}}·L_{\mathrm{gp}},$$

(11)

where $\mathbb{E}$ denotes the expectation operator, $\mathrm{x}$ represents the real image, $\mathrm{z}$ denotes the image to be repaired, $\mathrm{D}\left(\mathrm{x}\right)$ is the output of the discriminator function applied to real data $\mathrm{x}$, and $\mathrm{D}\left(\mathrm{G}\right(\mathrm{z}\left)\right)$ is the output of the discriminator when applied to the generated data $\mathrm{G}\left(\mathrm{z}\right)$, with $\mathrm{G}$ being the generator. The term $L_{\mathrm{gp}}$ is the gradient penalty, which involves the gradient of the discriminator’s output with respect to the input data, serving to enforce smoothness and regularization in the training process.

3.5.2. Reconstruction Loss

This paper introduces the reconstruction loss to ensure that the details of the inpainting area are preserved while maintaining the original semantics. The reconstruction loss is defined as:

$$L_{\mathrm{rec}}={\mathbb{E}}_{x_{\mathrm{a}}}{∥x^{\mathrm{a}}-\mathrm{G}\left(x^{\mathrm{a}},\mathrm{a}\right)∥}_1,$$

(12)

where ${∥·∥}_1$ denotes the $L_1$ norm, which suppresses the blurring effect of the image, $\mathrm{a}$ is the original property label, and $x^{\mathrm{a}}$ is the reconstructed image.

3.5.3. Total Loss Function

$$L_{\mathrm{total}}=u_1{L}_{\mathrm{adv}}+u_2{L}_{\mathrm{rec}},$$

(13)

where $L_{\mathrm{adv}}$ is the adversarial loss, $L_{\mathrm{rec}}$ is the reconstruction loss, and $u_1$ and $u_2$ are the weights of the corresponding loss functions. The weights are defined as follows: $u_1>0$ and $u_2>0$ (indicating that neither $L_{\mathrm{adv}}$ nor $L_{\mathrm{rec}}$ can be excluded from $L_{\mathrm{total}}$).

4. Experiments

4.1. Datasets

In this paper, we build two thematic datasets of ancient paintings for the study of ancient painting inpainting. The first dataset consists of 5798 landscape-themed ancient paintings from several historical periods, each of which is a uniform size of 256 × 256 pixels. The second dataset contains 4638 ancient paintings of flower and bird subjects, also from different historical periods and styles, unifying each picture to 256 × 256 pixels. We divide the two ancient datasets into training and validation sets by 8:2, as shown in

Table 1. Distribution of different datasets.

Ancient Painting Dataset		Ancient Huaniao++ Dataset
Total	5798	Total	4638
Training set	4638	Training set	3710
Validation set	1160	Validation set	928
Size	256 × 256	Size	256 × 256

. Finally, a particular script is used to generate a mask image, resize the image to the specified size, and automatically generate multiple masks, pairing each image with a different mask image, resulting in an image corresponding to a mask image.

4.2. Experimental Setup

The environment of this experiment is NVIDIA GTX4090, torch2.0.0, CUDA11.8 running video card and 24 GB running memory. In this experiment, Adam’s algorithm was used to optimize the inpainting model parameters, with a hyperparameter of 0.00001, a learning rate of 0.0001, batch size of 16, and 400 epochs.

4.3. Evaluation Metrics

In the conducted experiment, a suite of metrics was employed to assess the model’s efficacy, including the mean absolute error (MAE) for measuring the average magnitude of the error, the peak signal-to-noise ratio (PSNR) to evaluate the quality of the reconstruction, the structural similarity index measure (SSIM) for gauging the similarity in structural patterns between the model’s output and the reference, and the learned perceptual image patch similarity (LPIPS) to quantify perceptual similarity based on learned features.

MAE is expressed by

$$MAE={\displaystyle \frac{1}{MN}}\sum _{i=0}^{M-1}\sum _{j=0}^{N-1}{\parallel I(i,j)-K(i,j)\parallel }_1,$$

(14)

where I(i,j) and K(i,j) are the pixel values of images I and K at position (i,j) in the original and reconstructed images, respectively.

PSNR is expressed by

$$MSE={\displaystyle \frac{1}{MN}}\sum _{i=0}^{M-1}\sum _{j=0}^{N-1}{[I(i,j)-K(i,j)]}^2,$$

(15)

$$PSNR=10·{log}_{10}\left({\displaystyle \frac{MA{X}_I^2}{MSE}}\right),$$

(16)

where $MA{X}_I$ is the maximum possible value of the image pixel.

SSIM can expressed by

$$SSIM=\left[l{(x,y)}^{\alpha }c{(x,y)}^{\beta }s{(x,y)}^{\nu }\right]={\displaystyle \frac{\left(2{\mu }_x{\mu }_y+C_1\right)\left(2{\tau }_{xy}+C_2\right)}{\left({\mu }_x^2+{\mu }_y^2+C_1\right)\left({\tau }_x^2+{\tau }_y^2+C_2\right)}},$$

(17)

where l(x,y), c(x,y), and s(x,y) respectively represent the similarity of luminance, contrast, and structure. Additionally, $u_x$ and $u_y$ are the average values of x and y respectively, ${\tau }_{\mathrm{x}}$ and ${\tau }_{\mathrm{y}}$ are their standard deviations, ${\tau }_{\mathrm{xy}}$ is their covariance, and $C_1$ and $C_2$ are constants to maintain stability.

LPIPS is expressed by

$$LPIPS(x,y)=\sum _l{w}_l·{∥f_l\left(x\right)-f_l\left(y\right)∥}_2^2,$$

(18)

where $f_l$(x) and $f_l$(y) are features extracted from images x and y through the pre-trained deep neural network at layer l, $∥·∥$ is between $f_l$(x) and $f_l$(y) in Euclidean distance, $w_l$ is the weight of the l-th layer, and the summation is performed on the selected l-layer feature layer.

4.4. Experiments on the Ancient Painting Dataset

4.4.1. Quantitative Comparison

In this paper, we compare four models: PI [30], RFR [31], EC [32], and FcF [33]. As can be seen from

Table 2. Comparison of different methods applied to the ancient painting dataset.

Model	PSNR↑	SSIM↑	MSE↓	LPIPS↓
PI [29]	32.41	0.892	0.0009	0.0762
RFR [30]	33.42	0.921	0.0008	0.0720
EC [31]	33.53	0.913	0.0007	0.0523
FcF [33]	33.72	0.914	0.0008	0.0532
RG-GAN (Ours)	34.62	0.930	0.0005	0.0507

, our RG-GAN model has a PSNR value of 34.62, which is the highest among the models. This indicates that our model has achieved the best quality in reconstructing and restoring images, demonstrating a superior ability to recover the original information of ancient images more accurately. Additionally, the RG-GAN model has an LPIPS value of 0.0507, which is the lowest among all models, indicating that it has the smallest perceptible difference. This is because, by combining Regional Attention-Style Transfer and multi-scale cross-layer fusion technology, this model can realize complex style transfer while maintaining the authenticity of the restored image content and at the same time enable the decoder to comprehensively utilize the feature information from different levels, thus improving the inpainting quality problem caused by the lack of high-level semantic features. In addition, this model also proposes a new Global Context Perception Discriminator, which enhances the discriminator’s ability in spatial region location and detail information extraction through the global attention mechanism and significantly improves the performance of the model in various inpainting tasks.

From the results of the comparison of the five methods on the PSNR indicator in Figure 7, it can be observed that RG-GAN has achieved the best results during training compared to other methods. The PSNR and SSIM metrics of our method reached 34.37 and 0.928, respectively, the highest values in all models, and the MSE and LPIPS metrics achieved the lowest values. This is because the model combines stable field operators and multi-scale cross-layer fusion techniques. We have solved the problem of the U-Net model dealing with invalid pixel transfer in missing areas. This enables the decoder to leverage feature information from different levels. This improves the inpainting quality problems caused by a lack of high-level semantic features. In addition, this paper proposes a new Global Context Perception Discriminator, which enhances the ability of the global attention mechanism in spatial localization and detail extraction and improves the performance of the model in various inpainting tasks.

4.4.2. Qualitative Comparison

Figure 8 shows the visualization results of different methods on landscape painting datasets. As can be seen, Figure 8c,d have lost details and exhibit blurred edges during image inpainting, resulting in poor inpainting results. Figure 8e,f show incoherent inpainting results in the missing areas of the image during the inpainting process. In contrast, our method produces higher-quality inpainting images compared to other methods. The preservation of edge details is particularly prominent in optimal inpainting.

4.5. Experiments on the Ancient Huaniao++ Dataset

4.5.1. Quantitative Comparison

To verify the generalization of our method, we compared it with four mainstream models on the ancient Huaniao++ dataset. The experimental results are shown in

Table 3. Comparison of different models on the ancient Huaniao++ dataset.

Model	PSNR↑	SSIM↑	MSE↓	LPIPS↓
PI [30]	19.15	0.723	0.0182	0.1896
RFR [31]	20.12	0.745	0.0151	0.1637
EC [32]	19.52	0.724	0.0154	0.1756
FcF [33]	21.34	0.793	0.0095	0.1874
RG-GAN (Ours)	23.46	0.857	0.0040	0.0938

. This model achieves the highest PSNR and SSIM values, 23.46 and 0.857, respectively, indicating that our RG-GAN excels at maintaining image structures and textures similar to the original. The repaired image is visually closer to the original and has the lowest MSE and LPIPS values. Our model’s performance on the ancient flowers and birds image dataset has proven to be superior to other mainstream image inpainting models.

As can be seen from the visualization of the results on the ancient Huaniao++ dataset using the five methods in Figure 9, our RG-GAN model has the highest PSNR value. The increasing gap with other models fully illustrates the excellent performance of our method in image inpainting and significantly improves the inpainting task for ancient paintings. Furthermore, our RG-GAN, with a model size of 16.7 M, processes each 256 × 256 image in approximately 72 ms. By comparison, the RFR [31] model, which has a larger size of 33 M, takes 95 ms for the same image size. This highlights the efficiency of our approach, as it not only reduces computational overhead but also delivers faster inference times, making it well-suited for real-world applications.

4.5.2. Qualitative Comparison

Figure 10 shows the visualization results of different methods on the ancient Huaniao++ dataset. As can be seen from Figure 10, there is a certain degree of ambiguity in the inpainting results of Figure 10c–f in terms of improving the fuzziness of the repaired images; however, there is a loss of detail features. In contrast, our method produces better results than other image inpainting methods, showing a semantically consistent repair for missing regions and excellent inpainting results in image detail and edge texture.

4.6. Ablation Study

Table 4. Ablation study of different models on the ancient painting dataset.

Model	PSNR↑	SSIM↑	MSE↓	LPIPS↓
Baseline	30.56	0.890	0.0065	0.0558
w/o RASTM	33.75	0.920	0.0024	0.0532
w/o MCPM	32.24	0.901	0.0042	0.0546
w/o GCPD	33.43	0.912	0.0032	0.0538
RG-GAN (Ours)	34.62	0.930	0.0005	0.0507

illustrates the impact of various models on the inpainting of ancient paintings.

Effectiveness of the Regional Attention-Style Transfer Module (RASTM). As observed in the second row of Table 4, the Regional Attention-Style Transfer Module uses an attention mechanism to transfer the feature distribution of the missing area content and the normal area style content, in order to achieve fine-grained alignment between the missing area content and the normal area content and to realize complex style transfer while maintaining the authenticity of the content. Consequently, the PSNR and SSIM metrics increased by 0.87 and 0.010, respectively. Figure 11c demonstrates that there is local fuzziness in image inpainting when the RASTM module is omitted.

Effectiveness of the Multi-scale Cross-layer Perception Module (MCPM). This paper introduces the Multi-scale Cross-layer Perception Module designed to ensure that skip connections are not solely reliant on a single-scale feature but also incorporate multiple levels of semantic information from the encoder. This allows the decoder to fully exploit the deep semantic features of the encoder, thereby enhancing the semantic coherence and detailed richness of the repaired images. Figure 11d indicates that there is no semantic inconsistency in the image missing regions when the MCPM module is employed. Additionally, the PSNR and SSIM indicators are improved by 2.38 and 0.029, respectively.

Effectiveness of the Global Context Perception Discriminator (GCPD). The Global Context Perception Discriminator (GCPD) addresses the shortcomings of existing models in capturing details during the inpainting of ancient paintings. By utilizing a global attention mechanism, it effectively retains information across channel and spatial dimensions, enhancing cross-dimensional information interaction and significantly improving model inpainting. As depicted in Figure 11e, the PSNR and SSIM metrics increased by 1.19 and 0.018, respectively, for image inpainting without the GCPD module.

Figure 11f displays the image inpainting effect of our method. The repaired image produced by our method closely resembles the original, validating the effectiveness of our method.

5. Conclusions

To address the challenges in ancient painting inpainting, this paper proposes a novel ancient painting inpainting method with multi-scale fusion and global context perception to counteract the damage suffered by ancient paintings over time. Firstly, to deal with the inconsistency between the styles of missing areas and those of non-missing areas, a Regional Attention-Style Transfer Module is proposed in this paper to achieve fine-grained alignment between the contents of missing areas and normal areas and to achieve complex style migration while maintaining the authenticity of the contents. By providing reasonable initial values to the missing areas of the feature map for the decoder, semantic consistency and feature continuity are ensured, thus improving the accuracy of the inpainting. Secondly, to address the lack of detailed information extraction, we design a multi-scale fusion mechanism. This mechanism enhances the feature representation of the filled area by gradually refining the method, enabling the decoder to fully utilize the multi-level semantic information from the encoder, effectively improving the semantic coherence and detailed richness of the repaired image. Additionally, a Global Context Perception Discriminator is proposed to overcome the limitations of existing models in detail capture. Utilizing the global attention module, the discriminator’s ability to locate spatial areas and extract critical detail information is enhanced, further improving the overall performance of the model. Experiments on the ancient paintings and ancient Huaniao++ datasets demonstrate that our method achieves the peak PSNR values of 34.62 and 23.46, along with the lowest LPIPS scores of 0.0507 and 0.0938, respectively. In the future, we are working on finer ancient painting inpainting methods to achieve complete inpainting of detailed textures.