Building Facade-Completion Network Based on Dynamic Convolutional GAN

Abstract

Building facade completion is an important part of digitizing the structures of buildings using computer technology. Due to the intricate textures and structures in building facade images, existing image-completion algorithms cannot accurately restore the rich texture and detailed information. In response, this paper proposes a novel network to simultaneously recover the texture and semantic structural features of building facades. By incorporating dynamic convolutions into each layer of the feature encoder, the shallow layers of the completion network can create a global receptive field, thus enhancing the model’s feature-extraction capability. Additionally, a spatial attention branch is integrated into the dynamic convolution module to boost the correlation between the completion area and its surrounding edge area, resulting in improved edge clarity and accuracy of the completed facade image. Experimental results on multiple public image datasets demonstrate that the proposed model in this paper achieves state-of-the-art results when applied to real-world datasets.

1. Introduction

Urban environments are characterized by their complexity and often contain obstructions such as vegetation, pedestrians and vehicles, which unavoidably occlude building facades [1]. These obstructions can impede subsequent applications such as semantic segmentation, facade analysis, building structure recovery and facade reconstruction. Recovering the occluded portions of facade images is a recurrent problem that is currently facing three main challenges: (1) The lack of large-scale datasets for building facade completion has made it difficult for many advanced image-completion networks to train high-precision completion models [2]. (2) Unlike images of faces or natural landscapes, building facade images are artificial scenes that contain various features, including texture, structure, dimension and regularity. However, existing image-completion algorithms struggle to maintain consistency in both local and global building facade features, resulting in completion results that do not adhere to building priors (as shown in the yellow box in Figure 1 NS algorithm). (3) The deep learning-based image-completion framework represented by Yu et al. [3] has introduced contextual attention mechanisms to capture long-range dependencies between the completion area and global features. However, it does not adequately handle the correlation between the completion area and its edge region, often resulting in distortions in the edge structures and blurred textures (as shown in the red box in Figure 1, Deepfill).

To address the aforementioned challenges, we developed a dynamic convolution module to extract features. Dynamic convolution continuously updates the weights of the convolution kernel according to the input data, which effectively increases the model’s capacity. As a result, the model can learn the features of building facades with fewer parameters, reducing the risk of overfitting and dependence on large-scale datasets. Additionally, we enhanced the global receptive field based on dynamic convolution, effectively capturing the long-range dependencies of the completion area and ensuring the consistency of the overall structure of the building facade. Furthermore, we incorporated a spatial attention branch to identify the edge features. It enhanced the correlation between the completion area and the surrounding area, making the edges of the completion area clearer. Finally, we conducted comprehensive comparative experiments based on multiple benchmarks. the results showed that the proposed model achieved state-of-the-art performance on real-world building facades.

In summary, the technical contributions of this paper are as follows:

1.

We propose a novel facade-completion network based on dynamic convolution that allows training with fewer model parameters, reducing the risk of overfitting. The network uses a global receptive field to process image features, ensuring the relevance between texture and semantic structural features in both the completion area and the global area of the building facade.

2.

We introduce a spatial attention branch to enhance the feature representation of the missing parts by strengthening the features of the edge region of the mask and weakening the background features, effectively improving the local structural correlation between the completion area and the surrounding area in building facades.

2. Related Work

2.1. Image Completion for Facade

Existing building facade-completion algorithms can be roughly divided into two categories: example-based methods and neural network-based methods.

Example-based methods generally aim to infer the layout of the facade occlusion based on the known structure arrangement. For example, Dai et al. [5] adopted irregular rectangular lattice (IRL) to partition the layout of semantic components on the building facade, such as doors and windows, and represent them as a semantic label map. Then, a genetic algorithm is used to guide the completion of missing areas based on the uncovered parts, ensuring overall structural consistency. Huang et al. [6] proposed an automatic guided patch-based building facade-restoration method utilizing mid-level structural clues. First, the feature points of the known area are detected, and then the arrangement rules are extracted. Finally, the completion area’s texture pixels are guided based on the regularity between the feature points, and the texture completion is performed using pixel offsets. Kottler et al. [7] adopted an improved patch-based completion method, where a rough structure completion is first performed based on context information and average pixel values. Then, an intuitively selected parameter is weighted with image pixels to further optimize the completion results. However, these methods lack the generalization capabilities of deep learning algorithms and are only suitable for a small number of building facade images with regular arrangements.

Guided by [5], Hensel et al. [8] also employed IRL to parse the semantic components of facades and represent them as a semantic label map. In contrast to [5], they used LSTM to learn the arrangement of doors and windows on building facades and utilized neural network learning to infer the semantic layout of missing areas of doors and windows. Then they combined the texture information to realize facade completion. However, this method had limitations in dealing with irregular facade structures since it relied on the regular arrangement structure of unobstructed parts to infer the consistent arrangement of obstructed parts. Generative Adversarial Networks (GAN) [9] has been used for a number of computer vision tasks such as face completion [10], image dehazing [11], image deraining [12,13], facade synthesis [14] and facade completion. Reference [15] proposed the use of semantic segmentation and GANs for building facade completion, which was implemented using the general image-completion algorithm, DeepFillV2 [16]. This method can achieve facade completion for a wider range of building types. However, it relies on large-scale training data. Obtaining a large number of building facade images is a challenging task, and existing open-source building facade datasets are typically small, which limits the application of this method. DeepFillV2 has a relatively complex network structure, which includes two stages: coarse network and fine network. The coarse network is mainly used for structural completion, while the fine network is used for texture repair. However, training a complex network structure on a small dataset is prone to overfitting, which often leads to blurred edges between the completion area and the surrounding areas. Kottler et al. [17] proposed a three-stage GAN for completing the facade contours, facade semantic structures and facade textures, respectively. However, obtaining a building facade semantic label dataset is a massive task, and the joint training of the three GAN networks increases the model’s complexity, reducing its stability.

2.2. Image-Completion Algorithm

Existing image-completion methods can be broadly classified into two categories: traditional image-completion methods and deep learning-based image-completion methods.

Traditional image-completion methods mainly include diffusion-based completion and exemplar-based completion. Diffusion-based methods [18,19,20] grow the edge pixels at the completion location inward along the same feature as the effective area of the image, diffusing to fill the entire area to be repaired. Example-based methods [21,22,23] search for similar image blocks in the original image and copy them to the completion area. However, due to the high computation cost of the search and optimization process, these methods are not suitable for handling building facade images with complex textures. Traditional image-completion algorithms only rely on shallow texture information, which is inadequate for completing building facades that contain objects such as windows and doors with rich geometric structure features.

Compared to traditional image-completion methods, deep learning-based completion methods can obtain high-level image features, strong independence and robustness, and achieve high accuracy and good results in image completion. Pathak et al. [24] combined the encoding–decoding structure with convolutional neural networks and utilized GANs as auxiliary structures to develop a context encoder. The context encoder uses multiple convolutional layers in the encoding–decoding structure to parameterize the semantic-sensitive content, with the ability to synthesize high-order features in a more extensive spatial range. However, this method can only handle very low-resolution inputs due to the difficulty in training. Global-Local Consistent Image Completion (GL) [25] employed a fully convolutional network for image completion. Two global and local discriminator networks are adopted to significantly improve the image-completion quality. The partial convolution algorithm [26] greatly enhanced computational efficiency by incorporating a mask into the convolution operation and distinguishing between completion and incompletion pixels, thus improving its sensitivity. Deepfill [3] proposes a coarse-to-fine network based on the GL method. This method also has two discriminators, a global discriminator and a local discriminator, and the generator part consists of two-stage feed-forward generation networks, including rough and refined completion. Additionally, a content-aware layer is introduced into the generator to extract relevant information for completion from effective features that are far away from the completion part, increasing the spatial consistency of image module perception. Cr-fill [27] used joint training with an auxiliary context-reconstruction branch (CR-loss) to achieve good completion performance without using any attention layers.

However, existing deep learning-based image-completion methods often have complex network structures and suffer from overfitting when trained on small-scale building facade datasets. Additionally, these methods struggle to maintain global and local consistency in building facade images which possess rich geometric structures and texture features.

Compared to the previous methods, the proposed facade-completion network has the advantage of a simple network structure, strong feature representation ability and less dependence on large datasets and facade semantic labels.

3. Method

As shown in Figure 2, our facade-completion network consists of a generator and a multi-scale discriminator. The generator employs dynamic convolution to adaptively extract complex structural features of the facade and produce more realistic completion results. The multi-scale discriminator, on the other hand, evaluates the fidelity of the completed image across various resolutions, thereby boosting the overall quality of the completed building facade. Our network aims to tackle the challenge of missing texture in building facade images and accurately distinguish between real and fake generated images to improve the accuracy and effectiveness of image restoration.

3.1. Dynamic Convolution with Dynamic Channel Fusion

In the task of image completion, a sufficiently large receptive field is required to predict the contextual information of the missing region. The larger the receptive field, the more contextual information the model can perceive, thereby improving the accuracy of image completion.

In recent image-completion works [3,16,25], dilated convolutions have been used to achieve this functionality. However, the convolution kernels of dilated convolutions are sparse and may skip some local details, affecting the completion results. Multi-scale fusion has been adopted in some work [28,29] to expand the receptive field and ensure dense convolution kernels. However, because of the complex texture and structure features of building facades, with multi-scale fusion it is difficult to adaptively extract complex structural details. The fixed convolution kernel cannot adapt to missing objects of different sizes and irregular shapes. The weights of the fixed convolution kernel also cannot be adjusted for the different characteristics of the input data. On the other hand, conventional dynamic convolutions, such as Condconv [30] and DRconv [31], also have drawbacks. They need to calculate the weighted sum of multiple static convolution kernels, which undoubtedly increases the computational cost of the model. Moreover, conventional dynamic convolutions calculate the weighting coefficients between static convolution kernels through attention mechanisms, which can cause smaller weighting coefficients to have a disproportionate impact on feature representation in network layers.

Inspired by [32], we propose a dynamic convolution module based on dynamic channel fusion. As shown in Figure 3, our dynamic convolution module utilizes dynamic channel fusion to generate convolution kernels. Firstly, image features are fed into a dynamic attention mechanism similar to the Squeeze-and-Excitation (SE) module [33]. The dynamic attention mechanism performs global average pooling on the feature maps of each channel to reduce the input image $I\in R^{(C\times H\times W)}$ to one dimension, obtaining a global description vector $I_{se}\in R^{(C\times 1\times 1)}$. This effectively overcomes the problem that the receptive field of the convolution operation can only obtain local contextual information.

The dynamism of our network is mainly reflected in the fact that the weights of our convolution kernels are obtained from the conv and conv1×1 layers shown in Figure 3. In other words, the weights for feature processing are dynamically determined by the input image information. We capture the dependencies between channels through two fully connected layers to improve the overall consistency of the feature map. The fully connected layers output two matrices, set the dimension of the fully connected layer to $C\times C$ to obtain the channel attention matrix $\Lambda \in R^{(C\times C)}$ and set the dimension of the fully connected layer to $L\times L$ to obtain the channel fusion matrix $\Phi \in R^{(L\times L)}$. $\Lambda$ is a diagonal matrix of size $C\times C$, where the values on the diagonal represent the attention weights of each channel. Compared with the traditional weights of size $C\times 1\times 1$, it offers more feature details, further improving the feature learning ability of the image-completion network. In contrast to the SE module, we adopt three convolutional branches parallel to the dynamic channel attention module to obtain the static feature weight matrix, $W_0\in R^{(C\times k^2)}$, where k is the dimension of the convolution kernel, as well as the matrices $P\in R^{(C\times L)}$ and $Q\in R^{(C\times L)}$, used for channel compression and expansion. The final dynamic convolution kernel weight can be expressed by the following equation:

$$W\left(x\right)=\Lambda \left(x\right)\ast W_0+P\ast \Phi \left(x\right)\ast Q^T$$

(1)

where x represents the input data, ∗ represents matrix multiplication. By performing matrix multiplication between the channel attention matrix $\Lambda$ and the static feature weight $W_0$, the convolution kernel’s ability to learn complex structural features in the image is enhanced. $P\ast \Phi \left(x\right)\ast Q^T$ performs dynamic channel fusion on the input feature weights, which not only improves the compactness of the model and avoids the suppression of key features by small dynamic coefficients, but also overcomes the problem of joint optimization between static kernel weights and dynamic attention mechanisms.

3.2. Spatial Attention Branch

In convolutional neural networks, the receptive field refers to the size of the input data region where each neuron in the input layer can extract features. Larger receptive fields can help the network learn more global features and improve the model’s generalization ability. Increasing the receptive field of a network is usually achieved by increasing the network depth or introducing a global receptive field module in the deeper layers. In image-completion tasks, a global receptive field should be introduced in the shallower layers of the network to enable the model to obtain a larger receptive field as early as possible [34]. Therefore, we use dynamic convolutions with larger receptive fields in the first layer of the generator to extract global features from the input image. The kernel size of the first dynamic convolution is $7\times 7$. We propose the Dconv-block module that incorporates dynamic convolution into residual blocks to improve the ability of the image-completion network to learn features of building facade images, thus further enhancing the accuracy of the network. Specifically, dynamic convolution can dynamically adjust the weights of the convolution kernels based on the different features of the input data, thereby better adapting to the complexity of the data. Meanwhile, residual modules can avoid problems such as gradient disappearance when using deep dynamic convolution. As shown in Figure 4, the generator in our network uses a residual network module composed of dynamic convolutions to construct a deep feature-extraction network. Each residual block contains three dynamic convolution layers of kernel size $3\times 3$.

Although the dynamic convolution layer based on dynamic channel fusion can effectively transmit valid global feature information to the deep layers of the network, ensuring the structural consistency of the complete features and global features. However, during the training process, the channel feature weight corresponding to local feature information with a less influential role in learning key features is small, which may cause the network model to ignore this detailed information. The local detail information is also crucial to obtain more realistic building facade-completion images. As shown in Figure 4a, we introduce the spatial attention mechanism (SA) into the branch to enable the dynamic residual module to learn this local feature information. The spatial attention mechanism first performs average pooling to reduce the feature map and expand the receptive field while reducing the computation cost. Then, two standard convolution modules with convolution kernel sizes of $3\times 3$ are used for local feature sampling to improve the model’s attention to the edge parts of the completion area. Next, an upsampling module is used to restore the features to their original size, obtaining locally enhanced features. Finally, pixel-wise multiplication is performed between the input feature map and locally enhanced features to increase the weight of local features in the completion area of the feature map. Therefore, the SA module can be expressed by the formula:

$$f_{SA}=f_{in}⨂f_l$$

(2)

where $f_{in}$ represents the input feature map, $f_l$ represents the locally enhanced feature information and ⨂ represents the pixel-wise feature multiplication operation.

Next, we use the feature addition method to achieve feature fusion between the globally features $f_{dcd}$ obtained after processing through a three-layer dynamic convolutional network, and the locally enhanced features $f_{SA}$. Finally, the fused features are input into the ReLU activation function to obtain the output result y of the Dconv-block-SA module.

$$y=\delta \left(add(f_{dcd},f_{SA})\right)$$

(3)

where $\delta$ represents the ReLU activation function, and $add$ represents the element-wise addition of feature values.

3.3. Network Structure

The overall generator adopts the structural framework of U-Net [35], with the key difference being the introduction of the dynamic convolution module, Dconv-block, in the skip connection part. Utilizing the dynamic convolution module for dynamic convolution operation on features within the skip connection part enables better extraction of image features and makes the generated results more accurate.

The feature-decoding part of the model, as shown in Figure 4b, is represented by ResNet-blocks, where the residual modules in the decoding part consist of two residual networks and an upsampling layer. These components effectively extracting information from low-resolution feature maps and gradually restore them to high-resolution images. Each residual network consists of two standard convolution operations and an SE module branch, which can further enhance the expression power of feature maps and the ability to extract key features, thus generating more realistic images.

We use a multi-scale discriminator for adversarial training to ensure that the completed images exhibit realistic texture details, effectively differentiating between the real high-resolution image $I_{gt}$ and the synthesized image $I_{in}$. Consequently, the loss function of the discriminator is given by ${\mathcal{L}}_D={\displaystyle \frac{1}{N}}{\sum }_N{\mathcal{L}}_{D_n}$, where N denotes the number of discriminator sub-modules in the multi-scale discriminator, which is set to $N=2$ in this study, and ${\mathcal{L}}_{D_n}=E\left[\log (1-D_n\left(G(I_{in},M)\right))\right]+E\left[\log D_n\left(I_{gt}\right)\right]$, where $D_n$ represents the nth discriminator, G represents the generator, M represents the mask of the completion area. The features at different scales are averaged using average pooling.

3.4. Loss Functions

The loss function of the image-completion network aims to measure the difference between the generated image and the real image, thereby guiding the optimization of the generator network. By optimizing the loss function, the generated images can be made closer to the real images. We use adversarial loss, pixel-wise loss and VGG loss to train the facade-completion network.

GAN loss: The role of the adversarial loss function is to train a discriminator network to make the generated images from the generator network as close as possible to real images, thereby improving the quality and realism of image completion. We first train the discriminator to increase the probability of accurately identifying real samples, inproving its ability to distinguish real samples. Subsequently, we train the generator to maximize the function $E_{\widehat{I}}[\log (1-D\left(\widehat{I}\right)]$, that is, to reduce $D\left(\widehat{I}\right)$ by generating better quality images $\widehat{I}=G(I_{in},M)$. Therefore, our adversarial generative loss function can be represented by Equation (4):

$${\mathcal{L}}_{adv}=E_{\widehat{I}}\left[\log (1-D\left(\widehat{I}\right))\right]+E_{I_{gt}}\left[\log \left(D\left(I_{gt}\right)\right)\right]$$

(4)

where $\widehat{I}$ represents the predicted image, $I_{gt}$ represents the real image, $D\left(\widehat{I}\right)$ represents the probability of the discriminator identifying the fake image and similarly, $D\left(I_{gt}\right)$ represents the probability of the discriminator identifying the real image.

Pixel-wise loss: Pixel-wise loss function, as knowns as L1 loss or mean absolute error (MAE) loss is used to measure the difference between the generator output image and the real image. The pixel-wise loss function calculates the difference between each pixel of the generated image and the real image and sums these differences to obtain a scalar value representing the degree of difference between the generated and real images. By minimizing the pixel-wise loss function, the generator can learn how to reconstruct missing areas in the image, making the generated image as similar as possible to the real image at the pixel level. Therefore, our pixel-wise loss function, which can be expressed as:

$${\mathcal{L}}_{pix}=\left|\right|I_{gt}-\widehat{I}{\left|\right|}_1$$

(5)

where $\widehat{I}$ represents the predicted image, $I_{gt}$ represents the real image and ${\parallel ·\parallel }_1$ represents L1 norm.

VGG loss: The VGG loss function aims to leverage a pre-trained VGG network to extract features. It calculates the difference between the generated image and the real image in the feature space, and guide the generator to learn how to generate completion results that are closer to the real image. We use a pre-trained VGG19 model to extract features, which consists of two parts: perceptual loss function and style loss function.

The formula for the perceptual loss function is as follows:

$${\mathcal{L}}_{per}^{\varphi ,j}={\displaystyle \frac{1}{C_j{H}_j{W}_j}}\left|\right|{\varphi }_j\left(\widehat{I}\right)-{\varphi }_j{\left(I\right)}_{gt}{\left|\right|}_1$$

(6)

where $\varphi$ represents VGG feature-extraction function, and $C_j{H}_j{W}_j$ represent the number of channels and size of the image in the layer j, respectively.

The style loss function takes as input the corresponding regions in the predicted completed image $\widehat{I}$ and the real image $I_{gt}$. And then feeds them into the pre-trained VGG19 network. The feature values are then extracted from each layer and a Gram matrix is used to measure the correlation between features in different parts of the image. Finally, the L1 loss is used to calculate the distance between them. As shown in Equation (7):

$$\begin{array}{c}\hfill {\mathcal{L}}_{style}={\displaystyle \frac{1}{C_j{H}_j{W}_j}}\left|\right|Gram\left({\varphi }_j(\widehat{I}\odot M)\right)-Gram\left({\varphi }_j(I_{gt}\odot M)\right){\left|\right|}_1\end{array}$$

(7)

where M represents the binary mask, and ⊙ denotes multiplying the pixel values at corresponding positions.

Therefore, the VGG loss function can be represented by the Formula (8).

$${\mathcal{L}}_{vgg}={\lambda }_{per}{\mathcal{L}}_{per}+{\lambda }_{style}{\mathcal{L}}_{style}$$

(8)

where ${\lambda }_{per}$ and ${\lambda }_{style}$ are, respectively, the weights for the perceptual loss function and the style loss function.

As a result, the loss function of the proposed completion network can be formulated as shown in Equation (9):

$${\mathcal{L}}_{GAN}={\lambda }_{adv}{\mathcal{L}}_{adv}+{\lambda }_{pix}{\mathcal{L}}_{pix}+{\lambda }_{vgg}{\mathcal{L}}_{vgg}$$

(9)

where ${\lambda }_{adv}$, ${\lambda }_{pix}$ and ${\lambda }_{vgg}$ are, respectively, the weights for the adversarial loss function ${\mathcal{L}}_{adv}$, pixel-wise loss function ${\mathcal{L}}_{pix}$ and VGG loss function ${\mathcal{L}}_{vgg}$. We set ${\mathcal{L}}_{adv}=1$, ${\mathcal{L}}_{pix}=2$, ${\mathcal{L}}_{vgg}=1$.

4. Experiment

4.1. Experimental Settings

Data and metrics: We used a total of 606 images from CMP facade dataset [36], 15,000 images from Paris Street View dataset [24], 180,000 images from CelebA dataset [37] and 4631 images from Places2 dataset [38]. Due to limited memory space of the device, we only extracted a portion of Places2 dataset to train the network model. In real life, the occluded parts of building facades are irregular. Therefore, we used the irregular mask dataset provided by [26] to generate irregular missing regions. To quantitatively evaluate the completion performance of the model on different datasets, we used the following different metrics:

• MAE: Mean Absolute Error (MAE) between predicted values and ground truth can roughly reflect the ability of the model to reconstruct the content of the original image. The smaller the MAE value, the better the generated result.
• PSNR: Peak Signal-to-Noise Ratio (PSNR) is a widely used objective image quality evaluation metric that calculates the error between corresponding pixels in two images. The higher the PSNR value, the less distortion and better performance.
• SSIM: Structural Similarity (SSIM) [39,40] is another reference image quality evaluation metric that measures image similarity from three aspects: luminance, contrast and structure. SSIM has a value range of [0,1], with a larger value indicating smaller image distortion.
• FID: Fréchet Inception Distance [41] uses the Inception v3 [42] image classification model to extract visual features from original images to separately measure the similarity between two sets of images. It is a metric that calculates the distance between feature vectors of real and generated images.
• LPIPS: LPIPS [43] uses a pre-trained VGG or Alex network to measure the perceptual image similarity between images, where a smaller value indicates a higher level of similarity.

We set the training images dimension to $256\times 256$, and perform random cropping and flipping operations on the data. Subsequently, we concatenate the data with the mask image at the channel level to form four-channel image data as input to the model. For our image-completion network, we use Adam optimization function, which dynamically adjusts the learning rate of each parameter using the first and second-order moment estimates of the gradient. We define the initial learning rate as $lr=0.001$. All the model training and evaluation were conducted on the same platform: an Ubuntu server with an Intel i7-9700F CPU running at 3 GHz, 32 GB of memory, 512 GB of swap space and two 2080Ti graphics cards with a total of 24 G of GPU memory.

4.2. Comparison with Existing Work

Quantitative Comparisons: In this study, we quantitatively compare our proposed model with several other image-completion algorithms, including three non-deep learning algorithms: FRS [44], NS [4], Telea [45] and four deep learning-based algorithms: PEN-Net [28], Cr-fill [27], DeepFillV2 [16] and Lama [34].

Firstly, we conducted experiments on CMP facade dataset and evaluated the algorithm performance using three quantitative metrics, namely PSNR, SSIM and LPIPS.

As shown in

Table 1. The quantitative evaluation results on CMP facade dataset [36] for various mask coverage rates in the ranges 10–20%, 20–40% and 40–60%. The symbol “↓” indicates that a lower value is better, while “↑” indicates the opposite. The best scores are in bold for emphasis. The following text follows the same notation and will not be annotated individually.

	PSNR (↑)			SSIM (↑)			LPIPS (↓)
Mask	10–20%	20–40%	40–60%	10–20%	20–40%	40–60%	10–20%	20–40%	40–60%
FRS [44]	32.7418	29.2348	10.0350	0.9708	0.9487	0.1583	0.5759	0.5806	0.6654
NS [4]	31.6565	27.7897	20.8385	0.9668	0.9393	0.7641	0.0357	0.0623	0.2565
Telea [45]	32.9517	27.7469	20.8646	0.9789	0.9378	0.7623	0.0205	0.0641	0.2632
PENNet [28]	26.6155	28.6404	21.3096	0.9395	0.8835	0.7598	0.0453	0.089	0.1880
DeepFillV2 [16]	32.7418	29.2348	21.4314	0.9708	0.9486	0.7916	0.0226	0.0367	0.1463
Cr-fill [27]	32.0074	28.8477	21.5734	0.9674	0.9458	0.7929	0.0253	0.0398	0.1464
Lama [34]	−	−	23.0938	−	−	0.8036	−	−	0.1326
Ours	33.5893	29.7769	22.5839	0.9733	0.9515	0.8098	0.0184	0.0306	0.1188

, in terms of SSIM, when the missing area of an image is extremely small, i.e., when the coverage rate of the mask is between 10% and 20%, the structure of the building facade image is relatively intact. Therefore, traditional image-completion algorithms such as Telea utilize the available information in the image for completion and achieve good results in terms of SSIM. However, in this scenario, our proposed completion algorithm achieves SSIM results second only to the Telea algorithm. Moreover, as the missing area increases, our proposed algorithm demonstrates much better completion performance in terms of SSIM than other image-completion algorithms. The Lama algorithm utilizes the global receptive field of Fourier convolution to effectively handle images with large areas missing. When the missing area of an image is extremely large, i.e., when the coverage rate of the mask is between 40% and 60%, our proposed image-completion algorithm performs only second to the Lama algorithm in terms of PSNR. However, considering the three metrics of SSIM, PSNR and LPIPS, as well as the completion results of building facades with different coverage rates, our algorithm demonstrates better overall completion quality, far exceeding existing image-completion algorithms.

Next, we will conduct a quantitative evaluation of the inpainting results of our proposed algorithm on the Paris Street View dataset and the CMP facade building semantic structure label dataset. This time, we will use three quantitative metrics, MAE, PSNR and SSIM, to evaluate the quality of the inpainting algorithm and compare it with several state-of-the-art image inpainting algorithms. As shown in

Table 2. The quantitative evaluation results of several deep learning-based completion algorithms and our proposed building facade-completion algorithm on Paris Street View dataset [24] and CMP facade semantic label dataset [36] in terms of MAE, PSNR and SSIM.

Algorithm	Paris StreetView			CMP Facade (Semantic Label)
	MAE (↓)	PSNR (↑)	SSIM (↑)	MAE (↓)	PSNR (↑)	SSIM (↑)
PENNet [28]	0.0402	24.3106	0.7858	−	−	−
Lama [34]	0.0417	23.7953	0.8387	0.1303	13.6811	0.5199
DeepFillV2 [16]	0.0327	24.8073	0.8237	0.1387	10.4645	0.7494
Cr-fill [27]	0.0331	24.9039	0.8183	0.0936	14.0850	0.6613
Ours	0.0292	26.2195	0.8387	0.0387	18.2277	0.8848

, our algorithm demonstrated a high peak signal-to-noise ratio (PSNR) on the Paris Street View dataset. It also achieved the highest structural similarity index (SSIM) compared to other inpainting algorithms, indicating that the structure of our output was most similar to the ground truth. Concerning mean absolute error (MAE), our algorithm had smaller content differences with the ground truth compared to other algorithms. On the CMP facade semantic label dataset, our proposed inpainting algorithm still outperformed other inpainting algorithms in terms of MAE, PSNR and SSIM, demonstrating better inpainting results.

Overall, our algorithm outperformed other image-completion methods on the building facade image datasets, exhibiting higher completion quality and more accurate detail preservation for both texture and structural information. This was demonstrated through better performance in terms of PSNR, SSIM and MAE on both the Paris Street View and CMP facade semantic label datasets.

Qualitative Comparisons: Similarly, we compared the quality of the proposed building facade-completion algorithm’s generated results with several commonly used non-deep learning image-completion algorithms: FRS, NS, Telea and state-of-the-art deep learning-based image-completion algorithms: PEN-Net, DeepFillV2, Cr-fill and Lama.

As shown in Figure 5, we trained our proposed dynamic convolution-based completion network and several commonly used image-completion models on the CMP facade dataset. We can see that compared to non-deep learning image-completion algorithms, our proposed image-completion algorithm can better restore the structural and texture information, resulting in more realistic completion results. Compared to deep learning-based image-completion algorithms, our proposed building facade-completion algorithm has a larger receptive field, which allows for obtaining more information from surrounding pixels and a better understanding of the contextual relationships in the image. Therefore, Our algorithm’s results exhibit improved texture feature integrity and diversity.

Lama algorithm uses Fourier convolution to construct the image-completion network, to give the model a global receptive field during feature learning. However, Fourier convolution can only process signals in the frequency domain, and the convolution kernel in the spatial domain is still a fixed small matrix, which cannot adapt to complex features to a certain extent. Moreover, the frequency domain convolution operation performed in the Fourier domain may lose spatial information. We use dynamic convolution to extract features by dynamically adjusting the weights of the convolution kernel, enabling it to adaptively extract complex feature information from the input data. This approach can better handle the complex texture and structural features of building facades. Additionally, we introduce a spatial attention branch to preserve the spatial information of pixel features. As shown in Figure 6, compared with the Lama algorithm, our completion algorithm has more realistic completion effects on texture details.

As shown in Figure 7, we conducted experiments on the Paris Street View dataset, and compared with the existing image-completion algorithms, our completion algorithm still has more realistic visual effects.

Semantic Fill Study: To demonstrate the effectiveness of our algorithm in recovering the semantic structure of building facades, we conducted experiments on the building facade semantic label image dataset provided by the CMP facade dataset and compared it with several existing image-inpainting algorithms. As shown in Figure 8, we can see that our proposed inpainting network has a larger receptive field, which can better capture the contextual structural information of the images. Additionally, we introduced a spatial attention mechanism to preserve the positional information between pixels. Therefore, our algorithm produces visually superior inpainting results on the building facade semantic structure dataset compared to other image-inpainting algorithms.

User Study: We conducted a user study on the results of building facade completion using the test set of the CMP facade dataset. We set up A/B/C/D options on a WeChat questionnaire, with one algorithm’s result image in each option, and the order of the result images for each algorithm was randomized. One of the images was the result of our completion algorithm, and the other three were the results of the existing algorithms for completion. We invited 12 participants to choose which image was closer to the real building facade image out of the four result images. We collected 36 sets of data for evaluation and counted the frequency of each algorithm selected. As shown in Figure 9, our algorithm outperformed the other compared algorithms in the user evaluation.

Few-sample Completion Study: We proposed an image-inpainting algorithm based on the dynamic convolution module, which can synthesize corresponding convolution kernels according to the complex structural features of building facades, thus achieving effective feature extraction of the input data. Therefore, dynamic convolution can be more generalizable than traditional convolution in some scenarios. To further explore the generalization ability of our proposed inpainting algorithm, we conducted experiments on the Paris Street View dataset in four stages. In the first stage, we used all the data in the dataset for experiments. In the second stage, we randomly sampled half of the data in the original dataset for experiments. In the third stage, we randomly sampled one-quarter of the data in the original dataset for experiments. In the fourth stage, we randomly sampled one-eighth of the data in the original dataset for experiments. We compared our proposed algorithm with two other image-inpainting algorithms, Cr-fill and DeepFillV2, and recorded the changes in PSNR values. As shown in Figure 10, with a decrease in the number of samples in the dataset, the gap between the PSNR values of the completion results of our proposed algorithm and the other two completion algorithms becomes larger. Moreover, as the number of samples in the dataset decreases, our algorithm only experiences a slight decrease in the PSNR value of the completion results, while the PSNR values of the completion results of the other two algorithms have a significant downward trend. Therefore, it can be concluded that our proposed building facade-completion algorithm has a strong generalization ability. This also indicates that our proposed building facade-completion algorithm can still exhibit good performance when dealing with datasets with a small number of samples, demonstrating its practicality.

4.3. Ablation Study

In this section, we will conduct ablation experiments to analyze the roles of our proposed algorithm components, the dynamic convolution module based on dynamic channel fusion and the spatial attention module, in image completion.

Effectiveness of Dconv-block: We set the structure-completion network of structureFlow [46] as the baseline and then added different dynamic convolution modules to help the model learn features better, namely baseline+condconv and baseline+DRconv. Condconv and DRconv are popular dynamic convolution modules proposed in recent years. Similarly, we added our proposed dynamic convolution module to the baseline and conducted experiments using the CMP facade dataset. We used four metrics, namely MAE, PSNR, LPIPS and FID, to quantitatively evaluate the performance. As shown in

Table 3. The quantitative evaluation results on the facade validation set for different dynamic convolution modules on the CMP facade dataset [36].

Method	MAE (↓)	PSNR (↑)	LPIPS (↓)	FID (↓)
Baseline [46]	0.0362	22.3843	0.1203	22.2626
Baseline+condconv [30]	0.0356	22.3831	0.1139	16.7377
Baseline+DRconv [31]	0.0357	22.3311	0.1167	14.4352
Baseline+Dconv-block (Ours)	0.0355	22.3891	0.1132	13.4204

, after adding the other two dynamic convolution modules, the completion performance of the baseline has been improved in terms of LPIPS and FID indicators, indicating that dynamic convolutions can provide better feature representation for image-completion networks. When the baseline is combined with the Dconv-block, it achieves the best performance in terms of the MAE, PSNR, LPIPS and FID metrics. Therefore, the effectiveness of the dynamic convolution module in the image-inpainting task was demonstrated through the ablation experiment, which indicates a positive impact of dynamic convolution on complex structure image inpainting. Moreover, compared with other dynamic convolution modules, Dconv-lock is more effective in improving the quality of image inpainting.

Effectiveness of SA Module: To validate the positive impact of the spatial attention mechanism on the performance of the entire completion network, we compared the completion network model without the SA module and the complete structure-completion network model on the CMP facade dataset, with the baseline as a reference for both. As shown in

Table 4. The quantitative evaluation of image-inpainting results with and without the spatial attention (SA) branch on the CMP facade dataset [36].

Method	MAE (↓)	PSNR (↑)	LPIPS (↓)	FID (↓)
Baseline [46]	0.0362	22.3843	0.1203	22.2626
Ours (without SA)	0.0363	22.3385	0.1138	14.9955
Ours	0.0355	22.3891	0.1132	13.4204

, we found that the removal of the SA Module resulted in a decrease in the generation performance of the completion network model in terms of MAE, PSNR, LPIPS and FID. Comparing to the baseline, adding the SA Module resulted in improvements in the MAE and PSNR metrics, while the LPIPS and FID metrics were consistently better than those of the baseline.

4.4. Experiments with Other Typed Data

The experiments were not only conducted on the building facade dataset but also on other style image datasets to evaluate the generality of our proposed inpainting algorithm. We conducted experiments on other style image datasets to evaluate the universality of our proposed inpainting algorithm. Specifically, we evaluated our method on the Places2 dataset and the CelebA face dataset. The Places2 dataset is a large-scale scene classification dataset, consisting of over 1.8 million 256 × 256 resolution images. To validate the inpainting performance of the model on a small-sample dataset, we selected only 4631 images from the dataset, dividing them into 3631 images for model training and 1000 images for model testing. The CelebA dataset is a large-scale facial dataset containing 180,000 facial images. We divided it into 162,079 images for training and 40,520 images for testing. In addition, we conducted visual evaluations of our proposed image-inpainting algorithm and compared it with commonly used image-inpainting algorithms, DeepFillV2 and Cr-fill, on these two datasets. The evaluation models of these two algorithms on the Places2 dataset were trained using their official pre-trained models. As shown in Figure 11, on the Places2 dataset, the training performance of our proposed completion algorithm on a small dataset is comparable to that of the other two completion algorithms on a large dataset. Comparing the test results on the CelebA dataset, we found that our proposed completion algorithm still has a more realistic completion effect for larger-scale facial missing regions.

We compared our proposed image-inpainting algorithm with DeepFillV2 and Cr-fill on CelebA and Places2 datasets and evaluated the quality of the completion results using three metrics: MAE, PSNR, SSIM, LPIPS and FID. As shown in

Table 5. Quantitative evaluation comparison on datasets CelebA [37] and Places2 [38].

Dataset	Method	MAE (↓)	PSNR (↑)	SSIM (↑)	LPIPS (↓)	FID (↓)
CelebA	DeepFillV2 [16]	0.0364	27.5051	0.8702	0.4048	16.6591
	Cr-fill [27]	0.0417	26.0711	0.8697	0.3179	22.3520
	Ours	0.0359	28.5568	0.8764	0.2184	11.3311
Places2	DeepFillV2 [16]	0.0412	24.7671	0.8114	0.9747	66.0904
	Cr-fill [27]	0.0458	24.1457	0.8008	0.9707	70.0322
	Ours	0.0376	24.3606	0.7941	0.5730	49.1062

, for the completion results on the CelebA face dataset, our proposed image-completion algorithm outperforms the other two image-completion algorithms in terms of MAE, PSNR, SSIM, LPIPS and FID. This indicates that our proposed algorithm has good performance and superior results in completing missing parts of face images, improving the visual quality and realism of the images. It also demonstrates the feasibility and effectiveness of the algorithm in practical applications. When comparing the completion results on the Places2 dataset, it can be observed that the DeepFillV2 algorithm performs better in terms of PSNR and SSIM. However, it should be noted that our proposed image completion model was trained on a small sample dataset. Compared with image-completion algorithms trained on large-scale datasets, our proposed algorithm performs similarly in terms of PSNR and SSIM, although it still has advantages, such as good performance on the MAE, LPIPS and FID metric.

The experimental results show that our proposed building facade-completion network not only successfully handles the task of building facade completion, but also has universality in other data sets. This means that the network can be extended to deal with other types of image-completion problems, and satisfactory results are obtained.

5. Discussion

In this study, we introduced an image-completion algorithm that leverages dynamic convolution and spatial attention modules to effectively restore building facade images. Our experimental results substantiate the efficacy of our proposed approach, specifically when compared to existing algorithms such as Cr-fill and DeepFillV2 under few sample sizes in the Paris Street View dataset.

This study has two key findings. (1) Effectiveness of Dynamic Convolution Blocks (Dconv-blocks): The dynamic convolution modules enable adaptive adjustment of convolutional kernel weights, facilitating the extraction of complex feature information. By integrating these dynamic convolution modules, we observed a significant improvement in the image-completion outcomes. (2) Introduction of the Spatial Attention Module: Incorporating a spatial attention branch demonstrated a notable ability to preserve the spatial information of pixel features. Comparative experiments highlighted that our algorithm delivers superior visual results.

Although our algorithm shows considerable advantages, there are limitations that warrant discussion. For example, our method may result in noticeable artifacts in the completed areas when synthesize high-definition pixel structures for high-resolution images. Additionally, when performing large-area completion on images with few repetitive features, the restoration effect of our method may be suboptimal. For future work, we plan to explore other state-of-the-art technologies, such as diffusion models [47,48], to further enhance image-completion performance.

6. Conclusions

This paper proposes an image-completion algorithm based on dynamic convolution modules and conducts experiments using a building facade dataset. We have demonstrated through experiments that the dynamic channel aggregation mechanism of dynamic convolutions can enlarge the receptive fields in the encoder of the image-completion network, improve the long-range dependencies of the completion network and effectively alleviate the joint optimization problem between static convolutional kernels and dynamic attention mechanisms. This ensures the structural integrity of the completion region and global features and significantly improves the generation results of the image-completion algorithm for building facade completion. In the dynamic convolution module, we introduced a spatial attention branch to optimize the problem of insensitivity of dynamic convolution to edge information in the completion area. This improvement effectively improves the performance of dynamic convolution and provides the ability to perceive edge information in the completion area. Compared with the state-of-the-art completion algorithms, our proposed completion algorithm can achieve better completion results on building facade images.