Virtual Restoration of Ancient Mold-Damaged Painting Based on 3D Convolutional Neural Network for Hyperspectral Image

Abstract

Painted cultural relics hold significant historical value and are crucial in transmitting human culture. However, mold is a common issue for paper or silk-based relics, which not only affects their preservation and longevity but also conceals the texture, patterns, and color information, hindering cultural value and heritage. Currently, the virtual restoration of painting relics primarily involves filling in the RGB based on neighborhood information, which might cause color distortion and other problems. Another approach considers mold as noise and employs maximum noise separation for its removal; however, eliminating the mold components and implementing the inverse transformation often leads to more loss of information. To effectively acquire virtual restoration for mold removal from ancient paintings, the spectral characteristics of mold were analyzed. Based on the spectral features of mold and the cultural relic restoration philosophy of maintaining originality, a 3D CNN artifact restoration network was proposed. This network is capable of learning features in the near-infrared spectrum (NIR) and spatial dimensions to reconstruct the reflectance of visible spectrum, achieving the virtual restoration for mold removal of calligraphic and art relics. Using an ancient painting from the Qing Dynasty as a test subject, the proposed method was compared with the Inpainting, Criminisi, and inverse MNF transformation methods across three regions. Visual analysis, quantitative evaluation (the root mean squared error (RMSE), mean absolute percentage error (MAPE), mean absolute error (MEA), and a classification application were used to assess the restoration accuracy. The visual results and quantitative analyses demonstrated that the proposed 3D CNN method effectively removes or mitigates mold while restoring the artwork to its authentic color in various backgrounds. Furthermore, the color classification results indicated that the images restored with 3D CNN had the highest classification accuracy, with overall accuracies of 89.51%, 92.24%, and 93.63%, and Kappa coefficients of 0.88, 0.91, and 0.93, respectively. This research provides technological support for the digitalization and restoration of cultural artifacts, thereby contributing to the preservation and transmission of cultural heritage.

1. Introduction

The cultural relics of calligraphy and painting, as the precious heritages of millennia-old Chinese tradition, not only encapsulate profound historical and cultural values but also contribute significantly to the cultural legacy of China and even all mankind [1,2,3]. Nevertheless, due to prolonged natural aging, improper preservation, and adverse environmental conditions, these venerable relics frequently confront various pathological threats, with mold being one of the most prevalent and destructive. (In scientific classification, mildew tends to describe a specific type of light-colored surface mold, particularly referring to the order Erysiphales. In contrast, mold is a broader term that encompasses molds of various colors and types, including fungi such as Penicillium, Aspergillus, and Rhizopus. In the professional field of art conservation and restoration, mold is more commonly used to describe fungi that have a destructive impact on artworks.). The growth of mold not only impairs the aesthetic quality of these artifacts, reducing their artistic merit but also leads to the degradation of fiber in paper- or silk-based materials, resulting in irreversible structural damage and substantially escalating the complexity and expense of cultural relics restoration [4,5,6]. Consequently, it is imperative to explore effective strategies for the restoration of calligraphic and pictorial artifacts, especially those methods aiming to restore the original appearance of the artifacts without compromising their original material. This holds important significance for the protection and transmission of cultural heritage.

Traditional methods for mold remediation predominantly encompass physical interventions and chemical treatments [7,8]. Physical interventions, such as the drying of paper or silk, dehumidification, cryogenic freezing, and the application of ultraviolet light, do not cause chemical reactions in the materials of cultural artifacts [8,9]. However, these measures are typically effective only at the initial stages of mold infestation and it is difficult to fundamentally eliminate mold. Conversely, chemical treatments, which include the utilization of disinfectants such as ethanol and sodium hypochlorite, are capable of effectively exterminating mold [1]. Nonetheless, these agents may inflict irreversible damage upon the coloration and fibrous structure of the artifacts thus increasing their fragility and susceptibility to the relics. Consequently, there is a pressing necessity to develop and implement innovative technological methodologies that will enable the restoration of artworks to their original condition effectively and safely, without introducing further detriment.

In recent years, with the emergence of digital technology, particularly the development of virtual restoration techniques, new opportunities have been brought for the digitization and preservation of artifacts [10,11]. The greatest advantage of virtual restoration is that it focuses on images rather than the physical artifacts themselves, substantially mitigating any interference with the original artworks and minimizing the risks associated with their conservation [12,13,14]. Furthermore, this approach underpins the conservation and restoration of physical artifacts with a robust scientific foundation and provides a controlled experimental setting. With high-resolution digital scans, researchers can meticulously analyze mold damage on the surfaces of artifacts, without needing physical contact, and perform virtual restorations through digital simulations. This not only facilitates the evaluation of potential restoration strategies but also enhances the refinement of actual restoration methods, significantly improving the accuracy and efficiency of restoration work.

According to the different methods of collecting data sources for the virtual restoration of painting and calligraphy artifacts, they can be mainly divided into two types. The first one is based on the traditional digital cameras to capture the surface color information of the painting artworks, followed by the application of computerized image processing techniques for restoration. The basic principle of this approach is based on the assumption that the regions within the image requiring restoration exhibit geometric characteristics similar to those present in the original image [15,16,17], that is, according to the existing image information, speculate and fill in the missing or damaged part, so that the repaired image can maintain an appearance and features similar to the original image. This class of approaches can be classified into four basic categories as follows: equation-solving-based methods, transformation-domain-based methods, sample-block-based methods, and deep learning [18]. Equation-solving typically utilizes mathematical equations to describe the features and structure of an image and achieve restoration, by solving these equations. For example, Poisson editing is a method of image restoration by solving the Poisson equation. In particular, this algorithm can effectively make the transition between the restored region and the surrounding environment more natural in color and brightness, avoiding conspicuous boundaries or unnatural transition effects, and is one of the most commonly used algorithms in digital image restoration [19,20]. Additionally, the image repair model based on higher-order partial differential equations proposed by Bertalmio et al. [21] can simulate complex structural changes in images and transform the image restoration problem into a variational problem of solving for the global functional extremum, effectively repairing damaged images. The Curvature-Driven Diffusion (CDD) model, devised by Tony et al. [22], also incorporates a curvature factor. This feature can adjust not only the diffusion intensity according to the gradient magnitude but also the diffusion intensity according to the curvature guidance thus significantly improving the precision and effect of image restoration. Despite their adaptability and capability to manage diverse complex scenarios, these algorithms may lead to a distortion of the texture detail when addressing extensive data loss, which, in turn, can accentuate the blurriness of the images. Transform domain-based image restoration methods perform the restoration process by transferring the image from the spatial domain to another mathematical domain (such as frequency or wavelet domain), where the restoration processes are executed. Subsequently, the restored image is inversely transformed back into the original spatial domain. Common transformations include Fourier transform [23,24], wavelet transforms [25,26,27], and sparse coding dictionary learning [28,29,30,31]. However, these methods may introduce unnatural structures of artifacts into the process, especially when dealing with highly damaged areas or using inappropriate parameters or thresholds. The image restoration algorithm based on sample block is designed to repair damaged regions by identifying and replicating similar sample blocks from undamaged regions in an image. The fundamental principle is to make use of the inherent redundancy within the image, wherein the visual content of certain areas bears resemblance or repeats that of others, that is, the visual content of some regions of the image is similar or repeated to other regions, such as the PatchMatch [13,32,33] and Criminisi [2,34,35] algorithms. However, it is difficult for such algorithms to accurately capture the overall structure of images with complex structures or a lot of details, resulting in unsatisfactory repair results. Recently, deep learning methods have also been used to predict the content of missing areas, for example, Generative Adversarial Networks (GANs) were used to automatically generate the missing image content for Indian temple murals [12].

Another approach involves the use of hyperspectral imaging, which capture the spectral data from images. This technique is a non-invasive and non-destructive form of imaging that records the spectral characteristics of objects at high resolution. Hyperspectral imaging excels in detecting subtle differences in the spectral information of target materials, effectively circumventing the phenomenon where identical materials exhibit different spectral profiles [36]. Previous studies have successfully combined hyperspectral technique with image-processing algorithms to restore cultural relics. These methods can be categorized into spatial information-based restoration methods and spectral information-based restoration methods. For example, based on hyperspectral images, Zhou et al. [37] proposed a classified linear regression method, which corrects the true color spectral bands that are more affected by the stains based on the spectral bands that are less affected by stains. However, it is easy to cause incoherence in texture structure with this method. Hou et al. [10] employed hyperspectral imaging to perform maximum noise fractionation (MNF), then removed the components with stains, and reverse transformed the remaining components for restoration. However, this method is prone to altering the colors of other pigments. Zhou et al. [13] proposed a patch-based color-constrained Poisson editing method by taking advantage of the spectral characteristics of stains, solving the problem of color distortion by gradient threshold fusion. Hou et al. [2] carried out principal component transformation (PCA) on hyperspectral data, then repaired the mold region of the first three principal components with the largest amount of information based on the Criminisi algorithm and finally carried out inverse transformation. However, this algorithm can easily lead to a distortion of the detailed structure. Yang et al. [38] developed a virtual color restoration approach for mural pigments based on bandpass energy integration. However, there are few studies on the restoration of cultural relic mold information, and the algorithms involved do not make full use of the spectrum features of cultural relic mold, resulting in the distortion of cultural relic restoration details and colors.

To effectively remove mold from ancient Chinese paintings and based on the conservation principle of ‘restoring the old as the old’, this paper proposes a virtual restoration method based on 3D Convolutional Neural Networks (3D CNN). Our research indicates that the mold information diminishes with increasing wavelength and disappears at approximately 720 nm. In consideration of this characteristic, we devised a method to extrapolate the visible spectrum reflectance from the near-infrared (NIR) spectral reflectance data to remove mold information. The main contributions of this paper include the following:

(1) Analysis of spectral characteristics in mold patch restoration: The spectral response of mold was studied to analyze the differences in spectral characteristics between molded and non-molded regions. It was observed that as the wavelength increased, the information related to mold gradually diminished. Based on this spectral characteristic, virtual restoration of the affected mold regions was accomplished.

(2) Proposed a restoration system combining mold spectrum characteristics with CNNs: Utilizing a 3D CNN model, we combined NIR spectrum and spatial features to reconstruct the visible spectrum reflectance accurately. This approach significantly improves the accuracy of spectral data recovery, providing technical support for the digitization and restoration of cultural relics.

(3) Evaluation of virtual restoration methods based on classification: From the perspective of cultural relic analysis, the primary focus is on color classification. An application-driven evaluation of the mold virtual restoration method was conducted by examining the impact of the Inpainting, Criminisi, and Inverse MNF transformation methods and the proposed 3D CNN on pixel-level color classification. The color classification of the data after virtual restoration was performed through the support vector machine method. The overall accuracy and Kappa coefficient of the classification results of different restoration methods were compared to evaluate the effect of virtual restoration.

Although hyperspectral data comprise hundreds of bands, this study focuses on recovering the spectral reflectance values of the RGB bands (R: 650.02 nm, G: 539.59 nm, B: 477.88 nm). Since the RGB bands correspond to the three primary color perception channels of the human eye (red, green, and blue), the restored images become more visually appealing and practical, which provides a basis for the display and study of artworks. Additionally, the RGB bands are the key to achieving accurate color reproduction, helping to ensure that the digitized images are consistent with the original artwork colors.

2. Materials and Methods

2.1. Materials

We selected three regions with more mold damage of the ancient painting titled ‘Shen Qinglan Hua Tie Luo’ from the Palace Museum as our experimental data (Figure 1). The artwork is a silk painting featuring the prominent Chinese traditional auspicious pattern ‘huá fēng sān zhù’. This painting was created during the first and tenth year of the reign of Emperor Jiaqing, dating back to the period from 1760 to 1770 AD. In Region 1, ‘Lady’s picture’, there was primarily a lady whose face and hands were affected by mold damage; in Region 2, ‘Clothes’ picture’, the mold damage was mainly on the right side of the clothing, and in Region 3, ‘Branch’s picture’, the mold damage was primarily on the background in the center.

In previous research [39], the primary pigments in the painting were analyzed. It was found that the painting employs a wide variety of pigments, primarily cinnabar, azurite, malachite, lead white, and carbon black, with most of the effects resulting from mixtures of various pigments. The bright red on the characters’ clothing and headgear, as well as the orange on their clothing, are mainly cinnabar, while the screen area predominantly features pure azurite; the white on the faces, primarily made of lead white; and the green on the figure’s clothing, mainly made of malachite [39]. These mold-affected regions significantly reduce the painting’s lifespan and artistic value.

The experimental data were captured by the HS-VN/SW2500CR Heritage Spectral Imaging System, which includes a visible and near-infrared-shortwave infrared (VNIR-SWIR) imaging camera provided by Headwall Corporation (Boston, MA, USA) (see Figure 2). The VNIR camera captures images with 370 wavelength bands ranging from 400 nm to 1000 nm with a spectral resolution of 1.6 nm, while the SWIR camera captures 169 bands from 950 nm to 2500 nm with a spectral resolution of 9.6 nm. The system’s parameters are shown in

Table 1. Instrument parameters of HS-VN/SW2500CR were used to record the images.

Parameters	VNIR	SWIR
Spectral range/nm	400–1000	950–2500
Spectral sampling/nm	1.6	9.6
Sensor type	sCMOS	Stirling cooled MCT
Digitalizing bit	16 bit	16 bit
Slit width/µm	20	25
Light source	halogen lamp	halogen lamp

.

2.2. Spectral Characteristics of Silk-Based Mold

The mold spots on ancient paintings and calligraphy involve fungi such as Aspergillus niger, Aspergillus flavus, Cladosporium, and so on [4,8,40]. Especially sandalwood- and grass-based Xuan paper are more susceptible to mold growth, due to their hygroscopic and dust-absorbing characteristics [7,41]. At the same time, in silk paintings, the silk fibers are mainly composed of proteins (fibroin and sericin). These organic substances provide a rich source of nutrients for mold, facilitating its growth and reproduction. Metabolites produced by mold, such as water, organic acids, and cellulase, can disrupt the fiber structure and have a significant impact on the preservation and display of the artifacts [42]. As shown in Figure 3, it was obvious that with the increase in wavelength, especially beyond 720 nm, the information of mold spots in the image significantly diminishes, almost becoming undetectable. Based on this observation, we have developed an innovative image restoration strategy that utilizes the reflectance data of NIR spectra to reconstruct the reflectance of the visible spectrum regions, thereby effectively eliminating mold-related information from the image.

To further validate these mold characteristics and acquire the spectral characteristics of the mold, 300 pixels containing mold and 300 non-mold pixels were selected for each study area, and the spectral average value was calculated as the spectral reflectance feature. Envelope removal was carried out to enhance the comparative analysis of spectral features, aiding in the identification and differentiation of spectral characteristics between mold and non-mold pixels.

Compared to the average spectral information between mold areas and healthy backgrounds (Figure 4), several key spectral differences were observed. Firstly, the distinct absorption valley at 942 nm is primarily caused by the vibrational groups of water molecules [43]. Secondly, in the wavelength range of 450–650 nm, the absorption spectra show significant differences between healthy backgrounds and mold areas. The spectral valleys in healthy regions typically occur between 468 and 485 nm, whereas in mold areas, the spectral valleys are found between 560 and 620 nm, accompanied by a wavelength shift from the blue to the red edge. This red shift indicates a shift in spectral features from shorter to longer wavelengths, which is an important indicator for identifying mold. Significantly, whether in mold patches or unaffected background areas, the spectrum tends to peak after 720 nm and remains relatively stable in the range of 720 nm to 942 nm. This phenomenon occurs because Aspergillus niger and Aspergillus flavus exhibit different absorption wavelengths due to their distinct spatial molecular structures. Previous research has shown that their absorption rates are nearly zero around 720 nm, indicating very low absorption spectra in this range [44,45]. Consequently, the underlying spectral information of mold coverage can be effectively revealed in the NIR band. These spectral characteristics provide crucial physical evidence for the detection and removal of mold.

2.3. Data Processing before Virtual Restoration

Before the virtual restoration of mold removal, the mold regions of paintings were extracted based on the Random Forests (RF) algorithm. The process of the RF algorithm includes data sampling, decision tree construction, and model ensemble. In classification problems, majority voting is usually used, that is, a sample is classified as the category with the most votes. This integrated approach can significantly improve the accuracy and stability of the model [46]. It can be seen that the decision tree is the fundamental unit of RF. The core advantage of a random forest is its ability to form a stronger classifier by combining multiple decision trees (regarded as weak classifiers). In addition, since each tree partially sees different aspects of the dataset when it is built, it has strong robustness. More detailed information and theoretical support on RF can be found in [46]. With this method, we can effectively extract the regions affected by mold in paintings.

To optimize the extraction results, the morphological operations were applied as follows: using a 3 × 3 structuring element for opening and dilation operations. Through these processes, the small noise points in the image were removed, expanding the mold areas, and making their edges smoother and more coherent.

2.4. Reconstruction of Mold Region Using 3D CNN

In this paper, we propose the virtual restoration of painting mold removal 3D CNN based on hyperspectral images. The framework is shown in Figure 5. The structure of this model is similar to U-Net, wherein the left contracting pathway is tasked with extracting high-dimensional spatial and spectral features, while the right symmetric expanding pathway is utilized for information recovery. Through the inclusion of batch normalization (BN) and activation functions after each convolutional layer, this model attains accelerated convergence speed and heightened stability. Finally, flattened and dense layers are appended at the model’s conclusion to facilitate prediction.

Different from the traditional U-Net model, the 3D CNN was used for feature extraction in this model. Using 3D CNNs to process HSI data involves several steps as follows: block extraction, convolution, pooling, batch normalization, feature vector flattening, and prediction [47]. Initially, small 3D data blocks are extracted from the complete HSI dataset, typically sized (h × w × b), where h × w denote the width and height of the spatial dimensions, respectively, and b represents the spectral depth. Ensuring the target pixel resides at the center of these blocks allows for the effective capture of the surrounding spatial and spectral features. These blocks undergo convolutional layer processing, utilizing kernels of various sizes to extract features, encompassing local characteristics such as spatial textures and spectral signatures. Subsequently, pooling operations are applied post-convolution to reduce feature spatial dimensions, thereby mitigating computational complexity and overfitting while retaining crucial feature information. Batch normalization follows pooling to standardize convolutional layer activations, facilitating accelerated network convergence and enhanced training stability. After multiple convolutions and pooling layers, feature vectors are flattened into one-dimensional vectors for processing by fully connected layers. Finally, a fully connected layer maps the extracted high-dimensional features to the final regression values, culminating in the network predicting the target pixel’s attributes, thereby completing the regression task.

Using BN layers to normalize the input data of each batch enables the model to utilize larger learning rates, reduces the dependence of gradients on parameters or their initial values, enhances the stability of input data for each hidden layer, and accelerates the training process of deep neural networks [48]. The calculation formula is as follows:

$$\begin{array}{l}{R}_{nor}^k={\displaystyle \frac{R^k-E[R^k]}{\sqrt{Var[R^k]}}}\\ y^k={\gamma }^k{R}_{nor}^k+{\beta }^k\end{array}$$

(1)

where $R_{nor}^k$ is the normalized value, $E[R^k]$ is the average value, $Var[R^k]$ is the variance value, ${\gamma }^k$ and ${\beta }^k$ represents the scaling and translation parameters, respectively.

ReLU activation function uses the zero-setting strategy to enhance the sparsity of neurons, accelerating the convergence of deep learning models and avoiding gradient vanishing issues [49]. The calculation formula is as follows:

$$\mathrm{Re}LU=\left\{\begin{array}{l}x, if x>0\\ 0, if x\le 0\end{array}\right.$$

(2)

To take full advantage of the known spectral reflectance in this study, we randomly selected 50% of the pixels in which mold pixels had been removed as training samples and 50% as validation samples to construct the model.

2.5. Quantitative Analysis

To evaluate the performance of the restored models, the average value of 60 pixels around the mold was considered to be the true value of the mold [2,13], and the following three parameters were calculated: the root mean squared error (RMSE), mean absolute percentage error (MAPE), and mean absolute error (MAE). The definitions were using the following formulas:

$$RMSE=\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{(R_i-R^{’})}^2}}$$

(3)

$$MAPE={\displaystyle \frac{100}{N}}{\displaystyle \sum \frac{|R-R^{’}|}{R^{’}}}$$

(4)

$$MAE={\displaystyle \frac{1}{N}}{\displaystyle \sum |R-R^{’}|}$$

(5)

where N is the total number of recovery samples, Ri represents the reflectance values of the recovered mold-affected pixels, and R’ represents the average reflectance of adjacent pixels in the non-mold-affected area, respectively.

Pigment classification is the most concerning issue in the analysis of painted artifacts and understand painting techniques. We also conducted an application-driven evaluation of the virtual restored methods, assessing the effectiveness of virtual restoration through classification. This evaluation method allows us to understand the accuracy and effectiveness of different restoration methods, providing a scientific basis for the conservation and restoration of artworks. Based on the knowledge of painting and calligraphy experts, different colors were outlined, from which 60% of random points were used as training samples and the remaining 40% were taken as test samples. Based on the training samples, SVM was used to classify the virtual restoration results, and the overall classification accuracy and Kappa coefficient of the test samples in different virtual restoration results were compared, to evaluate different virtual restoration methods.

3. Results

The results of mold regions are shown in Figure 6b, Figure 7b and Figure 8b. As the results depict, the mold regions were completely and accurately extracted. The virtual restoration results based on 3D CNN-Unet are shown in Figure 6f, Figure 7f and Figure 8f, and we also compared them with the inpainting method proposed by Telea [50], the Criminisi method proposed by Criminisi et al. [34], and the inverse MNF method proposed by Hou et al. [10], as shown in Figure 6, Figure 7 and Figure 8c–e, respectively.

It can be observed from the comparison between Figure 6c,d that both Inpainting and Criminisi’s methods damaged the internal details of the image when dealing with mold, noticeable detail distortions are seen in areas such as the lady’s earlobes, forehead, and hands, specifically showing structural losses in hands and earlobes, with inconsistencies in the forehead and surrounding information. This is due to their reliance on surrounding pixel information for restoring missing areas, with Criminisi focusing primarily on texture synthesis by extracting and synthesizing texture information from the surrounding region, while the Inpainting method prioritizes filling missing regions based on local information and inferring pixel values for repair. The result of inverse MNF was shown in Figure 6e and indicated that the inverse MNF visual mold removal effect is the weakest. Owing to the mold information not effectively focused on a single component, the inverse MNF method has resulted in an unsatisfactory performance.

As shown in Figure 7, the results of Region 2, the Clothes’ picture, showed that in cases where the structural texture was relatively simple, the Inpainting and Criminisi methods had a better virtual restoration result. This is because both the Inpainting and Criminisi methods are capable of utilizing the information surrounding the missing regions for restoration. In the case of a relatively simple texture structure, these methods can more accurately infer the content of the missing regions from the surrounding pixel information. Moreover, the simplicity of the texture structures implies a reduced presence of texture information in the vicinity of the missing regions. Consequently, the Inpainting and Criminisi methods are less susceptible to interference from complex textures during the restoration process, thereby enabling them to more effectively utilize the surrounding information for restoration purposes. Unfortunately, this method also loses detailed texture information, as shown in Figure 7, the contents of the yellow circle. Meanwhile, the inverse MNF alters the color of the face. The reason was that the components of mold information also contained a lot of color information about the face.

Similarly, in the results for Region 3, the Branch’s picture, due to the complexity of color and texture, the Inpainting and Criminisi methods modified the original texture structure, especially the region in the yellow circle, so the repair effect is visually poor. At the same time, compared with the Inpainting and Criminisi’s methods, the inverse MNF’s result significantly changed the original color of the painting, and red, cyan pigments. Overall, our method significantly outperforms the other methods in visual mold removal effectiveness during visual inspections.

4. Discussion

4.1. Quantitative Analysis

For the quantitative analysis,

Table 2. Quantitative analysis of Region 1 Lady’s picture.

		RMSE			MAPE			MAE
		R	G	B	R	G	B	R	G	B
White material on the face	Original	0.0648	0.0667	0.1238	8.4878	13.5048	29.5463	0.0476	0.0541	0.1183
	Inverse MNF transformation	0.0600	0.0616	0.1182	7.6019	12.2222	28.1356	0.0427	0.0489	0.1126
	Inpainting	0.0949	0.0811	0.1348	11.4468	14.4198	31.4747	0.0642	0.0577	0.1260
	Criminisi	0.0882	0.0731	0.1305	10.1655	11.9957	29.2393	0.0570	0.0480	0.1170
	3D CNN	0.0503	0.0510	0.0989	6.3177	9.8145	22.4118	0.0355	0.0393	0.0897
White material on hand	Original	0.0799	0.0868	0.0668	9.8863	14.7794	14.0087	0.0592	0.0664	0.0503
	Inverse MNF transformation	0.0722	0.0790	0.0594	8.8873	13.4263	12.3301	0.0532	0.0604	0.0442
	Inpainting	0.0756	0.0631	0.0559	7.6091	9.2662	10.4840	0.0456	0.0417	0.0376
	Criminisi	0.1130	0.0908	0.0792	9.9254	11.8401	13.8018	0.0594	0.0532	0.0495
	3D CNN	0.0576	0.0477	0.0425	6.9609	7.9137	8.9051	0.0417	0.0356	0.0319

,

Table 3. Quantitative analysis of Region 2 Clothes’ picture.

		RMSE			MAPE			MAE
		R	G	B	R	G	B	R	G	B
Light red material	Original	0.0626	0.0517	0.0316	8.6691	13.3147	11.8176	0.0493	0.0423	0.0260
	Inverse MNF transformation	0.0433	0.0328	0.0254	6.0930	8.5844	9.3905	0.0347	0.0272	0.0207
	Inpainting	0.0361	0.0286	0.0235	4.8210	6.9725	8.3940	0.0274	0.0221	0.0185
	Criminisi	0.0343	0.0288	0.0232	4.6273	7.3410	8.5941	0.0263	0.0233	0.0189
	3D CNN	0.0383	0.0337	0.0276	5.1616	8.2030	10.0719	0.0294	0.0260	0.0222
Crimson material	Original	0.0845	0.0565	0.0302	12.6938	17.7171	13.3757	0.0650	0.0459	0.0239
	Inverse MNF transformation	0.0548	0.0415	0.0342	8.3783	12.9686	15.1888	0.0429	0.0336	0.0271
	Inpainting	0.0590	0.0385	0.0239	8.2551	11.6485	10.7408	0.0423	0.0301	0.0192
	Criminisi	0.0585	0.0511	0.0359	9.0721	16.4108	16.3552	0.0465	0.0425	0.0292
	3D CNN	0.0547	0.0374	0.0242	8.1121	11.1807	10.0496	0.0416	0.0289	0.0179

and

Table 4. Quantitative analysis of Region 3 Branch’s picture.

		RMSE			MAPE			MAE
		R	G	B	R	G	B	R	G	B
Brown background	Original	0.0983	0.1111	0.0949	17.5595	26.6170	27.2211	0.0766	0.0884	0.0738
	Inverse MNF transformation	0.0839	0.0766	0.0766	14.9242	18.2449	22.2645	0.0651	0.0606	0.0603
	Inpainting	0.1195	0.1052	0.0927	19.3230	22.8916	25.3922	0.0843	0.0760	0.0688
	Criminisi	0.1178	0.1032	0.0957	19.9626	23.4313	27.1039	0.0871	0.0778	0.0735
	3D CNN	0.0776	0.0739	0.0832	13.1059	16.2916	23.2589	0.0572	0.0541	0.0630

showed the performance of image restoration methods in different regions. Larger values of RMSE, MAPE, and MAE indicate greater errors between the restored values and the surrounding values, implying poorer restoration performance.

In Region 1 (Lady’s picture), the proposed method performs best according to the metrics. For the white pigment on the face, the Inverse MNF ranks second, and Criminisi third; while for the pigment on the hand, the Inverse MNF ranks second, and Inpainting third. This is because the Inpainting and Criminisi methods, when inferring values in the mold-affected regions, rely on the most similar surrounding values, which may introduce other influences and result in a loss of structural details, consistent with visual observations. In Region 2 (Clothes’ picture), Table 3 shows that Criminisi performs best in the light red region, followed by the Inpainting method, with the proposed method ranking third. However, for the dark red region, our method performs the best. This may be because in the light red region, details are relatively simple, and Criminisi and Inpainting methods can rebuild based on surrounding values, while in the dark red area, our method performs better. In Region 3 (Branch’s picture), our proposed method performs best in the R (650.02 nm) and G (539.59 nm) bands, while the Inverse MNF performs best in the B band (477.88 nm), but it alters the overall color, contradicting the principle of restoring the image to its original state.

In conclusion, our proposed 3D CNN method achieves relatively better performances in image restoration, balancing the need to repair damaged regions while retaining the original colors and textures, making it highly effective in tasks that require the preservation of the image’s original integrity.

4.2. Classification of Pigments

Pigment classification was performed on the original RGB and restored RGB images based on SVM, as shown in Figure 9, Figure 10 and Figure 11. The accuracy of the classification results was evaluated through manually inspected samples. As shown in Figure 9, in Region 1 (Lady’s picture), the colors were mainly divided into red, blue, yellow, black, white, brown, light red, cyan, and other colors (mainly mold colors). Overall, the classification results based on 3D CNN for image restoration outperformed other methods, demonstrating lower misclassification and missed classification rates. In terms of the overall accuracy and Kappa coefficients (as shown in

Table 5. Classification accuracy of different Restoration methods.

Class	Original		Inverse MNF Transformation		Inpainting		Criminisi		3D CNN
	Prod. Accuracy (%)	User. Accuracy (%)	Prod. Accuracy (%)	User. Accuracy (%)	Prod. Accuracy (%)	User. Accuracy (%)	Prod. Accuracy (%)	User. Accuracy (%)	Prod. Accuracy (%)	User. Accuracy (%)
Region1 Lady’s picture
Red	98.33	100	98.33	100	98.33	100	98.33	100	96.67	100
Black	93.33	93.33	95	93.44	91.67	91.67	91.67	91.67	98.33	96.72
Light Red	59.26	96.97	51.67	96.88	55	100	55	100	63.33	100
Blue	98	92.45	98	98	98	94.23	98	94.23	100	90.91
White	84.13	70.67	85.29	67.44	86.76	70.24	82.35	69.14	88.24	75
Cyan	90	88.24	96	90.57	92	86.79	92	86.79	88	86.27
Yellow	92	85.19	92	83.64	92	85.19	92	85.19	92	88.46
Brown	94	90.38	92	88.46	94	88.68	94	83.93	92	86.79
Overall Accuracy		88.56%	87.95%		87.95%		87.28%		89.51%
Kappa Coefficient		0.87	0.86		0.86		0.85		0.88
Region2 Clothes’ picture
Red	100	90.48	100	82.19	100	77.92	100	77.92	100	85.71
Deep Red	100	100	96.67	100	96.67	100	96.67	100	96.67	100
White	92.5	97.37	97.5	92.86	92.5	100	92.5	100	92.5	100
Black	87.5	94.59	92.5	97.37	80	96.97	80	96.97	90	97.3
Yellow	90	100	90	100	87.5	100	87.5	100	90	100
Offwhite	95	95	88.33	98.15	95	96.61	95	96.61	93.33	94.92
Cyan	90	94.74	90	100	90	85.71	90	85.71	90	97.3
Brown	90	79.41	93.33	83.58	91.67	83.33	91.67	83.33	91.67	79.71
Black red	98.08	91.07	90	88.52	90	88.52	91.67	88.71	98.33	90.77
Dark teal	93.33	93.33	93.33	93.33	93.33	93.33	93.33	93.33	93.33	93.33
Gray	76.67	86.79	78.33	85.45	76.67	88.46	76.67	90.2	76.67	88.46
Overall Accuracy		92.25%	91.72%		90.69%		90.86%		92.24%
Kappa Coefficient		0.91	0.91		0.90		0.90		0.91
Region3 Branch’s picture
Red	90.16	98.21	95.08	89.23	96.72	83.1	98.36	81.08	91.8	98.25
Gray	98.33	93.65	98.33	96.72	98.33	89.39	98.33	95.16	98.33	93.65
White	95	100	93.33	100	95	100	95	100	93.33	100
Black	91.67	94.83	93.33	90.32	88.33	100	93.33	96.55	91.67	100
Brown	88.33	77.94	96.67	87.88	95	79.17	88.33	81.54	95	89.06
Background	94.12	87.27	100	92.31	93.33	91.8	95	91.94	96.67	86.57
Light gray	60	62.07	76.67	82.14	43.33	72.22	46.67	60.87	80	72.73
Cyan	96	100	88	100	96	100	96	100	96	100
Yellow	93.33	100	80	100	83.33	100	73.33	100	93.33	100
Overall Accuracy		91.13%	93.00%		90.66%		90.45%		93.63%
Kappa Coefficient		0.90	0.92		0.89		0.89		0.93

), the overall accuracy of the classification results of different restoration methods reached more than 85%. Among them, with the overall accuracy and Kappa coefficients of 89.51% and 0.88, the classification accuracy of 3D CNN method performed the highest, which was 0.95% and 0.01 higher than that of the original image, respectively. At the same time, the Criminisi method performed the lowest classification accuracy. The overall accuracy and Kappa coefficient of this method were 87.28% and 0.85. Compared with the 3D CNN method, the overall accuracy and Kappa coefficient were 2.23% and 0.03 lower, respectively. In addition, the classification accuracy based on the inverse MNF, Inpainting, and Criminisi methods were lower than the original data. This is due to the fact that most of the mold information was not well repaired in the classification of images by the inverse MNF method, while the Inpainting method and Criminisi method were wrong in repairing the texture structure of the earlobe, resulting in texture distortion that caused the white color of the earlobes portion to be incorrectly classified as yellow and brown, etc.

In Region 2 (Clothes’ picture) of Figure 10, the colors were mainly divided into red, dark red, white, black, yellow, off-white, cyan, brown, black-red, dark teal, gray, and other colors (primarily mold colors). Overall, the classification results of repaired images based on 3D CNN were superior to those of other methods. As assessed by overall accuracy and Kappa coefficients (as shown in Table 5), the overall accuracy of the classification results of different restoration methods reached more than 90%. Moreover, the classification accuracy of 3D CNN-restored images was almost the same as that of the original data, and the overall accuracy was 92.24% and 92.25%, respectively, while the Kappa coefficients were both 0.91. The classification accuracies of the Inpainting-restored images were the lowest, with overall accuracy and Kappa coefficients of 90.69% and 0.9, respectively. In general, the accuracies of these methods were relatively close, as the texture of this region was relatively simple, resulting in good restoration effects.

As shown in Figure 11, in region 3 (Branch’s picture), the colors primarily included red, black, light gray, gray, brown, white, background color, cyan, yellow, and other colors (mainly mold colors). The 3D CNN-restored images still exhibited the best performance with lower misclassification and missed classification rates in the classification results. According to overall accuracy and Kappa coefficients (as shown in Table 5), the overall accuracy of the classification results of different restoration methods reached more than 90%. The classification accuracy of 3D CNN-restored images was the highest in particular, which was 2.5% and 0.03 higher than the original images. The inverse MNF-restored image achieved the second highest classification, with an overall accuracy and Kappa coefficient of 93% and 0.92, which were 1.87% and 0.02 higher than the original images, respectively. Conversely, the Criminisi-restored image displayed the lowest classification accuracy, with overall accuracy and Kappa coefficient of 90.45% and 0.89, indicating a reduction of 3.18% and 0.04 in comparison to the 3D CNN restored image. Additionally, it should be noted that the accuracy of the images restored by the Inpainting and Criminisi methods was also lower than the original data. This is because the texture structure of this region is relatively complex, and the Inpainting and Criminisi methods made errors in the restoration of texture structures, misclassifying the colors of the tree branch background as black and brown, among other colors.

Through the above analysis, it can be seen that the method of 3D CNN combined with the mold spectral characteristics proposed in this study can effectively leverage both spectral and spatial information by predicting the visible reflectance from the NIR spectral reflectance data, and it had performed exceptionally well in the virtual restoration of paintings for mildew removal in three different research regions. However, only RGB mold information was restored in this study. The method of structural spectral unmixing will be considered in the following study, and the whole hyperspectral information of mold regions can then be recovered to provide strong technical support for the digitization and the protection of cultural relics.

5. Conclusions

Aimed at solving the problem of mold in ancient painting and calligraphy, the spectral characteristics of mold were analyzed in this study. It was found that the reflectance of the region affected by mold spots stabilized after 720 nm and remained relatively stable within the wavelength of 720 nm to 942 nm. Based on these, a 3D CNN restoration framework was proposed. This method effectively estimates the reflectance of the visible spectrum using the NIR reflectance to acquire the virtual restoration of paintings for mildew removal. Experiments were conducted in three regions with different colored backgrounds and compared with the inverse MNF, Inpainting, and Criminisi restoration methods, respectively. Both visual inspection and quantitative analysis have demonstrated that the 3D CNN restoration achieved better performance than other methods. Specifically, this method completely removed mold information while preserving the original texture and colors of paintings and calligraphy, whereas the Inpainting and Criminisi methods altered the texture structure, and the inverse MNF method caused changes in other colors. Furthermore, color classification of the repaired images was performed based on the same samples and the SVM method. The results showed that the restored images with 3D CNN obtained the highest classification accuracies among the three regions, with overall accuracies of 89.51%, 92.24%, and 93.63%, and Kappa coefficients of 0.88, 0.91, and 0.93, respectively, proving the effectiveness of this method. However, only the RGB mold information was recovered in this study, and future studies will explore the method of combining mixed pixels, unmixing to achieve a more comprehensive restoration of mold spectral information and provide more effective technical support for the protection and restoration of ancient paintings and calligraphy.