Research on an Ultraviolet Spectral Denoising Algorithm Based on the Improved SVD Method

Abstract

The utilization of ultraviolet (UV) absorption spectroscopy for monitoring the concentration of specific decomposition gas components in gas-insulated switchgear (GIS) can provide a means to assess its insulation status. Nevertheless, UV optical modules currently deployed in the field are susceptible to external interferences like ambient noise and equipment vibrations. Real-time spectral data acquisition often suffers from significant noise contamination, directly impinging on subsequent feature extraction and detection accuracy. This paper presents an optimized singular value decomposition (SVD) noise reduction method for mitigating noisy spectral signals. First, each singular value within the noisy signal is transformed into a component signal. Next, the highest frequency value in the signal serves as an indicator to characterize the signal. Finally, the primary frequency values are arranged based on the decreasing singular values of the original noisy signal. The singular value corresponding to the first primary frequency value surpassing a preset threshold is selected as the effective order for denoising. Random noise with varying intensities was intentionally introduced to the UV spectral signal of sulfur dioxide (SO₂), followed by noise reduction procedures. It is shown that the improved SVD noise reduction algorithm proposed in this paper enhances the signal-to-noise ratio (SNR) by 18.02% and 16.86%, and reduces the root-mean-square error (RMSE) by 15.13% and 14.92%, respectively, compared with the singular value difference spectrum (SVDS) denoising method and wavelet transform denoising method under the condition of low SNR. Furthermore, there exists a linear relationship between the concentration of SO₂ samples and the eigenvalues of the UV spectra, demonstrating a higher linear goodness with a coefficient of 0.99735. The denoising method proposed in this paper does not require the manual setting of various types of parameters, and has a better ability to deal with the noise of UV spectral signals in engineering sites with complex environments.

1. Introduction

GIS has garnered significant prominence within the electric power industry due to its exceptional sealing capabilities and robust operational stability [1,2,3]. The sulfur hexafluoride (SF₆) gas employed in GIS demonstrates robust electrical insulation properties and excels in arc extinguishing performance. However, insulation defects that may arise during GIS manufacturing and operational processes can lead to partial discharges, resulting in SF₆ decomposition and the formation of various derivative compounds. Among the most prevalent characteristic decomposition components of SF₆ gas, the concentration of SO₂ serves as a pivotal indicator for assessing GIS insulation conditions. The detection of these characteristic SF₆ decomposition components in GIS is typically achieved through a variety of methods, including gas chromatography [4,5], infrared absorption spectrometry [6,7,8], and photoacoustic spectrometry [9].Gas chromatography offers high accuracy in gas detection but is plagued by extended detection periods. Photoacoustic spectrometry boasts high detection sensitivity, albeit susceptible to interference from environmental noise. Infrared absorption spectrometry permits simultaneous detection of multiple gases, but encounters challenges associated with the mutual interference of absorption peaks from each gas and high equipment costs [10].Within the wavelength range of 290 nm to 310 nm, SO₂ does not exhibit any characteristic absorption overlap with other SF₆ derivatives [11,12]. Consequently, UV absorption spectroscopy presents notable advantages in SO₂ detection. These advantages encompass shorter detection periods, reduced equipment costs, and minimized gas consumption.

In the on-site GIS insulation monitoring process, the utilization of portable ultraviolet spectrometers can be hindered by environmental noise and equipment vibrations, potentially submerging the genuine characteristics of the ultraviolet spectral signal within the noise. To facilitate a more accurate quantitative analysis of SO₂ within the GIS, it is imperative to undergo a denoising process of the ultraviolet spectral signal. Conventional methods for spectral data noise reduction encompass Fourier transform, wavelet transform, Savitzky–Golay filtering, and empirical mode decomposition (EMD). Fourier transform excels in analyzing signals in the frequency domain, particularly periodic ones, but falls short in extracting features from localized signals. Conversely, wavelet transform and the Savitzky–Golay filter necessitates the manual selection of various filtering parameters, rendering them less suitable for on-site inspections. Furthermore, EMD is an iterative algorithm that demands repeated signal decomposition, potentially leading to modal overlap issues and yielding unstable decomposition results.

SVD, as a data-driven signal processing technique, distinguishes itself by eliminating the need for manual parameter adjustments and demonstrates superior denoising capabilities for both linear and nonlinear signals [13]. Notably, the difference between the authentic spectral signal and the noise becomes more discernible in the singular values. The crux lies in accurately identifying the boundary between them to achieve effective denoising. As a result, this study introduces a novel approach for determining the optimal order of SVD. In comparison to previous methods, this approach maximizes denoising effectiveness and optimizes computational resources to enhance both computational efficiency and denoising accuracy. The following is a summary of this paper’s contribution:

• A passive, built-in optical sensor has been engineered, featuring a high-reflectivity concave mirror seamlessly integrated into the flange. This sensor is designed for direct mounting onto the GIS, enabling real-time online monitoring.
• Our innovative approach involves the reconstruction of singular values from noise-inclusive signals into distinct component signals. These component signals are then integrated with the fast Fourier transform (FFT) algorithm, introducing FFT peaks as metrics to characterize the signals. These metrics are subsequently ranked through a decremental process. The singular value corresponding to the first FFT peak surpassing a predefined threshold is chosen as the effective order for denoising.
• Detection experiments involving various concentration gradients were carried out using the SO₂ ultraviolet spectroscopy detection platform. The denoising performance of the proposed method was then compared to that of the conventional approach. The results clearly indicate that the method presented in this study outperforms the conventional approach in terms of denoising accuracy, making it a promising option for practical applications.

2. Related Work

The selection of the order for SVD directly impacts the effectiveness of signal denoising. An excessively high-chosen order of the singular spectrum may inadvertently retain noise information in the filtered signal, thereby impeding the desired denoising outcome. Conversely, an excessively low-chosen order of the singular spectrum might mistakenly treat valid signal components as noise, leading to their unintentional removal during the filtering process. Traditional SVD-based noise reduction is typically carried out by directly examining changes in singular values, and the optimal denoising order corresponds to the first singular value that undergoes a substantial alteration. Subsequently, the denoising order is determined by selecting the singular value order associated with the initial significant change in difference values. Singular values beyond this chosen order are then set to zero, effectively achieving the denoising objective [14].

Researchers have introduced various effective order selection methods. Aiang et al. proposed a method based on the singular value curvature spectrum [15]. Cheng et al. employed the SVD algorithm in conjunction with the principle of minimum mutual entropy to distinguish noise from the genuine signal with the aim of denoising [16]. Mao et al. introduced an innovative denoising algorithm that combines segmented SVD with the lifting wavelet transform (LWT) based on the ensemble empirical modal decomposition (EEMD) [17]. Lei et al. combined SVD with variational mode decomposition (VMD) to develop a novel denoising method [18]. In the work of Ren et al., a novel denoising methodology was introduced, amalgamating intrinsic time scale decomposition (ITD) and permutation entropy (PE)-based dual noise reduction techniques with SVD [19].

3. Theoretical Method

In this paper, we present an innovative method for the optimal order selection in SVD denoising. The process begins by reconstructing each singular value associated with the noise-containing signal into a one-dimensional signal. This signal is then transformed into the frequency domain. Next, we employ a FFT on each individual signal to identify the peaks as distinctive signal characteristics. Subsequently, we differentiate the FFT peaks by delaying the first-order values and arrange them in descending order according to the singular values. The singular value corresponding to the first FFT peak within the differential spectrum, surpassing a predetermined threshold, is identified as the optimal choice for denoising.

3.1. Principle of SVD Denoising

Given an original signal X = (x₁, x₂, x₃, …, x_N), a noise signal S = (s₁, s₂, s₃, …, s_N), and a noisy signal Y = (y₁, y₂, y₃, …, y_N), where N represents the length of the signal, the relationship among the three signals can be expressed as follows:

$$Y=X+S$$

(1)

To reconstruct the noisy signal Y in phase space, it is transformed into a Hankel matrix H_m×n (m ≤ n):

$$H_{m\times n}=\left[\begin{array}{cccc}{y}_1& y_2& \cdots & y_n\\ y_2& y_3& \cdots & y_{n+1}\\ ⋮& ⋮& \cdots & ⋮\\ y_m& y_{m+1}& \cdots & y_N\end{array}\right]$$

(2)

where $1<n<N$, m is the embedding dimension and satisfies $m+n-1=N$.

SVD of the Hankel matrix H:

$$H=U\sum V^T$$

(3)

where the orthogonal matrix $U\in R^{m\times m}$ and the orthogonal matrix $V\in R^{n\times n}$ are the left and right singular matrices of the H matrix, respectively, and $\sum$ is the singular value matrix of the following form

$${\sum }_{m\times n}=\left[\begin{array}{cc}\Delta & 0\\ 0& 0\end{array}\right]$$

(4)

where $\Delta =diag({\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_r)$, and ${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_r\ge 0$, ${\lambda }_r$ are the singular values of the Hankel matrix, r is the rank of this matrix, and 0 is the zero matrix.

Normally, the vital attributes of the genuine signal are encapsulated by the first k (k < r) significant singular values, with the noise signal being contained in the remaining (r − k) singular values. Therefore, through the proper selection of the SVD order, the subsequent setting of the remaining singular values to zero, and the subsequent reconstruction of the Hankel matrix using the inverse of the SVD, the denoising goal can be accomplished. The resultant Hankel matrix can then be simplified into a one-dimensional signal, effectively eliminating the noise.

3.2. Determining the Optimal SVD Decomposition Order

An excessively high choice of the SVD order can result in the inclusion of noise information in the filtered signal, thus hindering the desired denoising effect. Conversely, when the selected singular order is excessively low, it may inadvertently filter out valuable components as noise. In this study, we individually reconstruct each singular value of the spectral signal into a one-dimensional signal. We then employ the frequency domain to compare and analyze each singular component, facilitating the determination of the optimal order for SVD-based denoising while preserving the essential components.

The singular values of the signal Y are obtained through Equations (2)–(4) mentioned earlier. During each iteration Δ, only one of the λ_i is preserved, while the remaining singular values are set to zero. This process generates a collection of singular value vectors ${\Delta }_i^{\prime }=\mathrm{diag}(0,\cdots ,{\lambda }_i,\cdots ,0)$, along with the singular value matrix and Hankel matrix, which can be represented in the following form:

$${\sum }_i^{\prime }=\left[\begin{array}{cc}{\Delta }_i^{\prime }& 0\\ 0& 0\end{array}\right]$$

(5)

$$H_i^{\prime }=U{\sum }_i^{\prime }{V}^T$$

(6)

Each singular value λ_i in ${\Delta }_i^{\prime }$ is transformed into a matrix of singular values. The matrix is then inverted using singular value decomposition, resulting in the transformation matrix $H_i^{\prime }$, as depicted in Equations (5) and (6) provided earlier. Subsequently, the Hankel matrix is reduced to yield the component signals Y_i, which correspond to the original signal Y. These component signals are utilized to reconstruct the spectral signal.

We apply the FFT to each of the r component signals. Within each signal, we identify the frequency value corresponding to the maximum amplitude. These frequency values are then arranged in descending order, based on the singular values of the original signal. The primary frequency difference spectrum is derived through the application of first-order lagged difference processing. To accommodate specific situations, we establish a predetermined threshold range for differences. Singular values that precede the first difference value surpassing this threshold are preserved in the primary frequency difference spectrum, while the remaining singular values are nullified.

4. Experimentation and Evaluation

4.1. Denoising of SO₂ UV Spectral Signal Simulation

We selected the original noiseless differential absorption spectra of 5 µL/L SO₂ gas within the wavelength range of 290 nm to 310 nm. The data had a sampling interval of 0.11 nm and a signal length of 203. We also used simulated signals with added noise at a SNR of 5 dB, as show in Figure 1 and Figure 2.

The noisy signal, with a SNR of 5 dB, is transformed into a 101 × 103 Hankel matrix. This matrix is then subjected to SVD, resulting in the reconstruction of component signals for each singular value, following the previously described methodology. The resultant component signals are displayed in Figure 3.

As depicted in Figure 3, the characteristics of the original signal are clearly evident in the singular value component signals. The sum of the first six component signals can be considered the genuine and effective representation of the original signal. A notable change occurs in the seventh component signal. According to the principle of SVD denoising, the order corresponding to the alteration in the singular value, which coincides with the optimal order for singular value noise reduction.

In Figure 4, the first-order difference exceeding the predefined threshold of 0.3 serves as the effective order for SVD. It is evident that the order of the first singular value exceeding the preset threshold is 6, consistent with the observation of the singular value component signal transition from the seventh in Figure 3. The first six singular values are retained, and the remaining singular values are set to zero. The noise-reduced Hankel matrix is then reconstructed through the inverse transformation of SVD and condensed into a one-dimensional signal. The signal after denoising is smoothed, and its feature distribution closely aligns with the original signal. The denoising results for the noisy signal with a SNR of 5 dB are presented in Figure 5. The noise reduction algorithm proposed in this study is entirely data-driven and effectively distinguishes between the signal and noise based on the differences in singular value information. This capability enhances the accuracy of selecting the effective order for SVD denoising.

To evaluate the denoising capability of the optimization algorithm for the UV spectrum of SO₂, we assess the denoising performance of the UV-difference spectrum of SO₂ using the proposed denoising algorithm, the SVDS method, and the wavelet denoising method. For the wavelet denoising method, we choose the “sym14” wavelet basis function and set the number of decomposition layers to five [20]. The simulation dataset comprises a total of 30 samples. To gauge the denoising effectiveness, we employ the SNR and RMSE metrics. The denoising results are presented in Figure 6 and Figure 7.

The method presented in this paper demonstrates superior denoising effectiveness at lower SNR. It is shown that the improved SVD noise reduction algorithm proposed in this paper enhances the signal-to-noise ratio (SNR) by 18.02% and 16.86%, and reduces the root-mean-square error (RMSE) by 15.13% and 14.92%, respectively, compared with the SVDS denoising method and wavelet transform denoising method under the condition of low SNR. However, for higher SNR, approximately exceeding 20 dB, the denoising effect becomes comparable to that of the wavelet noise reduction method. The rationale for this approach is rooted in the observation that, once the SNR surpasses 20 dB, the noise content in the signal becomes minimal and closely resembles the original signal. Consequently, it becomes challenging to evaluate the denoising performance of these two methods under such circumstances. The SVDS method exhibits an overall average denoising effect and has limited capacity to process spectral data with specific SNR.

4.2. Denoising Experiments for Each Concentration Gradient of SO₂

To further validate the denoising effectiveness of the algorithm on measured spectra, a UV absorption spectroscopy-based SO₂ detection platform was established. This platform was equipped with a setup designed for SO₂ analysis. We meticulously prepared five sets of SO₂ test samples, each with concentrations of 1, 2, 5, 10, and 20 µL/L. A schematic of the experimental testing platform is depicted in Figure 8.

In the course of the experiments, a Bwtek Quest X-ray spectrometer and a HAMAMATSU L9455-11 scintillation xenon lamp were utilized. Single-mode optical fibers, selected for their optical transmission efficiency of at least 80% within the 200 nm to 400 nm wavelength range, were employed for optical transmission. The gas absorption cell, featuring an optical path length of 0.85 m, incorporated several concave mirrors coated with a high-reflectivity aluminum film. The inner wall of the cell was lined with a Teflon film to effectively mitigate gas adsorption effects. The UV light emitted by the xenon lamp was guided through the single-mode optical fiber and directed into the gas absorber cell using the collimating mirror. Within the gas absorption cell, the UV light thoroughly interacted with the SO₂ gas before being directed to the spectrometer via the focusing mirror. The spectrometer was connected to a computer, which further analyzed and processed the acquired spectral data. The high-reflectivity concave mirrors used in the experiments were integrated into the back of the flange, and the microspectrometer, xenon lamp, single-mode fiber optic, and PC processing terminal were integrated into the portable toolbox, as illustrated in Figure 9.

Spectral data for SO₂ at five different concentrations were subjected to noise reduction using three distinct methods: the algorithm proposed in this paper, wavelet transform, and SVDS. Following the Beer–Lambert law, absorbance is directly proportional to gas concentration, taking into account the absorption coefficient and optical path length. Similarly, in the frequency domain, attributes such as the maximum magnitude following FFT are also proportionate to gas concentration. Figure 10 illustrates that the UV absorption spectra of SO₂ at varying concentrations, after processing with the algorithm presented in this paper, display a well-defined distribution of absorption peaks and consistent frequency domain characteristics.

To compare the performance difference between the improved SVD denoising algorithm proposed in this paper and other denoising methods, line curves depicting the relationship between the FFT values of the UV differential spectra and the SO₂ gas concentration after various denoising methods were fitted, as illustrated in Figure 11.

The figure above illustrates that, following the improved SVD processing, the line fit of concentration to the FFT eigenvalues surpasses the performance of both the wavelet transform denoising algorithm and the singular value difference spectral denoising algorithm processing. Moreover, the determination coefficient is enhanced from 0.93627 to 0.99735 for the original data.

5. Conclusions

(1)

This study introduces the novel application of the SVD method for denoising SO₂ ultraviolet spectral signals. It also proposes an optimized method for selecting the effective order of singular value denoising. This method involves reconstructing each singular value of the original spectral signal into a one-dimensional signal, analyzing it in the frequency domain, and using the frequency value with the highest amplitude as an index to characterize each singular value component.

(2)

The denoising algorithm’s effectiveness is maximized by selecting the singular value order corresponding to the first significantly changed frequency value as the algorithm’s effective order.

(3)

In low SNR conditions, our study reveals that the improved SVD noise reduction algorithm presented in this paper leads to substantial improvements. It boosts the SNR by 18.02% and 16.86%, as compared to the SVDS method and the wavelet transform denoising algorithm, respectively. Additionally, the RMSE diminishes by 15.13% and 14.92%, respectively. Furthermore, the linear relationship between the concentration of SO₂ samples and the characteristic value of the UV spectra achieves a remarkable coefficient of 0.99735. The denoising method proposed in this paper does not require manual setting of various types of parameters, and has a better ability to deal with the noise of UV spectral signals in engineering sites with complex environments.

These findings highlight the effectiveness of our algorithm, particularly in situations with low SNR. In comparison to wavelet denoising and the SVDS method, our approach offers superior denoising capabilities. Although wavelet denoising and the singular value difference spectral method exhibit good denoising performance, our improved SVD method excels in scenarios with low SNR.

The focus of this paper is solely on the application of the improved singular value decomposition algorithm to the denoising of SO₂ UV spectra. In future research endeavors, it would be worthwhile to explore whether this method can be extended to the spectra of other characteristic decomposition component gases of SF₆.