Research on a Data Preprocessing Method for a Vehicle-Mounted Solar Occultation Flux–Fourier Transform Infrared Spectrometer

Abstract

A vehicle-mounted solar occultation flux–Fourier transform infrared spectrometer uses the sun as an infrared light source to quantify molecular absorption in the atmosphere. It can be used for the rapid three-dimensional monitoring of pollutant emissions and the column concentration monitoring of greenhouse gases. The system has the advantages of high mobility and a capacity for noncontact measurement and measurement over long distances. However, in vehicle-mounted applications, vehicle bumps and obstacles introduce aberrations in the measured spectra, affecting the accuracy of gas concentration inversion results and flux calculations. In this paper, we propose a spectral data preprocessing method that combines a self-organizing mapping neural network and correlation analysis to reject anomalous spectral data measured by the solar occultation flux–Fourier transform infrared spectrometer during mobile observations. Compared to the traditional method, this method does not need to adjust the comparison threshold and obtain the training spectra in advance and has the advantage of automatically updating the weights without the need to set fixed correlation comparison coefficients. The accurate identification of all anomalous simulated spectra in the simulation experiments proved the effectiveness of the method. In the vehicle-mounted application experiment, 342 anomalous spectra were successfully screened from 1739 spectral data points. The experimental results show that the method can improve the accuracy of gas concentration measurement results and can be applied to a vehicle-mounted solar occultation flux–Fourier transform infrared spectrometer system to meet the preprocessing needs of a high number of spectral data in mobile monitoring.

1. Introduction

Solar spectrum analysis is a commonly used method in atmospheric research; the sun is used as a light source to quantify molecular absorption in the atmosphere and subsequently determine the concentration of trace gases [1]. The SOF–FTIR is an optical remote sensing technique that uses a Fourier transform infrared spectrometer (FTIR) in conjunction with a mobile solar tracking system, which is suitable for monitoring the spatial and temporal distributions of emissions on a large scale in a mapped city [2]. The SOF–FTIR is mounted on a vehicle that moves in a circle around the target pollution source, and the solar spectrum is extracted from the measured interferograms via a Fourier integral transform to determine the gas emissions from the source [3].

However, when cloud and building shadows are present in mobile ground monitoring, the solar intensity changes greatly and the amplitude of the interferogram decreases, leading to a reduction in the spectral signal-to-noise ratio [4]. The second vehicle bumps, the car turns quickly, and other factors affecting the FTIR within the solar beam jitter, causing misalignment in the optical path. On the one hand, this leads to the infrared detection of the size, location, and intensity of the changes, reducing the signal-to-noise ratio; on the other hand, it results in the expansion of the angle of incidence, resulting in spectral spread and drift, which directly affect the instrument line function. This, in turn, affects the spectral quality and accuracy of the inversion of the gas concentration [5]. Therefore, rejecting the measured error of the spectral data can reduce the error in the spectral inversion results.

To date, most efforts have been devoted to improving the spectrometer and the solar tracking system [6,7]; however, rectifying low-quality spectra remains a key challenge. Continuous mobile monitoring produces approximately 1 spectral data point per 4 s, while continuous monitoring produces more than 900 spectral data points per hour. The percentage of invalid interferometric data can be more than 50% when traveling in areas with poor conditions, such as cities with tall buildings [8]. The traditional spectral rejection method mainly uses the Spearman correlation between the measured spectrum and the standard spectrum to achieve the rejection of the spectral data [9]; however, it is very difficult to achieve the spectral rejection of the attenuated intensity with an insignificant absorption structure. The selection of correlation coefficients is also a key challenge. The infrared spectrum has many absorption peaks, and the Levenberg–Marquardt optimization algorithm [10] and the classic climbing algorithm [11] tend to fall into local optimal solutions when there are multiple peaks; thus, they may not be able to find the global optimal solution. The classic least squares algorithm [12] is very sensitive to outliers. Due to the harsh conditions of in-vehicle applications, complex, abnormal data may appear, which may lead to model instability.

Neural networks offer another method for spectrum suppression. Neural networks simulate the intelligent activities of the human brain by developing appropriate learning algorithms. The features include massively parallel processing, distributed storage, elastic topology, high redundancy, and nonlinear operation. They offer high computational speed, strong association abilities, strong adaptability, strong fault tolerance, and self-organization abilities [13]. To reject abnormal target values, researchers in other fields have proposed improved methods from an algorithmic point of view. Yuan et al. utilized the spectral difference between the abnormal target and the background, as well as the spatial sparsity of the abnormal target, to achieve anomaly detection in spectral data based on a fast robust anomaly detection algorithm [14]. Ma K et al. used deep neural networks to learn spectral and temporal features and a deep learning method to detect changes in the system caused by system disturbances in order to achieve anomaly detection in hyperspectral images [15]. Therefore, the application of neural networks to anomalous data processing is an effective method. When directly applying the deep learning method to anomaly detection in spectral data, it is impossible to provide a perfect training criterion because the SOF–FTIR spectral intensity changes with the intensity of solar irradiance.

However, neural network clustering methods do not require a standard learning spectrum in advance. We only need to specify the number of categories for classification, and the algorithm will classify all samples according to the similarity principle. The SOM network is an important example of a neural network based on unsupervised learning methods. It was first proposed in 1981 by Kohonen, a neural network expert at the Helsinki College of Technology, Finland. The results are relatively easy to visualize and interpret [16]. This technique has been shown to be effective in solar cell design, using structured feature agglomerative clustering as an unsupervised dimensionality reduction step to identify the main features of the spectrum, which can reduce datasets from thousands of solar spectra to several characteristic surrogate spectra. The network successfully used these surrogate spectra to predict annual average efficiency as a function of solar cell design [17].

In this paper, we propose a method for preprocessing the data from a vehicle-mounted Fourier transform infrared spectrometer to solve the above problems; we successfully applied this method to eliminate abnormal spectral data from a vehicle-mounted SOF–FTIR. In this article, first, the causes of abnormal spectra are analyzed, the principle of the SOM method is introduced, and a spectral data preprocessing method combining the SOM neural network and correlation analysis is proposed. The working principle of spectral rejection based on this method and the practical effect of spectral rejection, which solve the problem of selecting correlation comparison coefficients, are also proposed. Second, the effectiveness of the proposed spectral data preprocessing method is verified using simulated spectra. Third, the validity of the spectral data rejection method for the spectral data preprocessing method in mobile SOF–FTIR monitoring is verified. Finally, some conclusions are drawn in the Conclusion section.

2. Spectral Data Preprocessing Methods

2.1. Spectral Anomaly Cause Analysis

The simplest type of spectral anomaly in vehicle-mounted SOF–FTIR applications is low intensity in the center of the interferogram (at an optical range difference of zero), which is a deviation due to the sun being obscured by obstructions such as clouds or tree shadows. In our study, the intensity of the interferogram decreased by a factor of approximately 100 in clear weather when the incident optical path was completely obscured (Figure 1a,b). A lower intensity in the center of the interferogram leads to a lower calculated spectral amplitude in the single beam, which does not necessarily lead to errors in the measurement results when concentration calculations are performed on the spectral fit (Figure 1c,d). Therefore, it is not desirable to use the intensity at the center of the interferogram to reject anomalous data.

Second, the low-frequency components caused by the vibration of one or more optical components during the use of a SOF–FTIR in vehicle-mounted systems due to vehicle shock, engine vibration, and wind, as well as electrical interference from the vehicle power supply, are included in the interferogram, which can result in an anomalous peak in the interferogram or oscillations that can lead to upward-emitting peaks or sinusoidal oscillations in the absorption spectrum. The further the peak is from the mean intensity of the interferogram, the higher the frequency of the oscillation. Figure 2 shows that single-beam spectra are affected due to incorrect spectral fitting results from superimposed emission peaks and oscillations in these spectra. However, we used a detector with an effective bandwidth of 650 cm⁻¹~5000 cm⁻¹, and any frequency perturbation outside the absorption band had no negative influence on the calculation of the gas absorption. Therefore, it was more effective to directly discard the spectral data within the effective band.

2.2. Data Preprocessing Methods

The SOM accepts an n-dimensional vector as input, which corresponds to an input layer with m nodes. The nodes of the input layer are connected to the competing layer via weight vectors, and each input sample corresponds to a node in the competing layer. When the training is completed, the input samples corresponding to the same node of the competing layer are classified into the same category [18]. The SOM includes an input layer and an output layer. The winning neuron of the SOM influences its neighboring neurons from near to far and gradually switches from excitation to inhibition. Therefore, not only does the winning neuron itself need to adjust the weight vector in its learning algorithm but the neurons around it also need to adjust the weight vector under its influence.

We obtained spectral data samples and normalized the individual spectral data to form an mxn multidimensional data matrix. Here, m represents the number of neurons for each spectral dataset, and n represents the number of spectral data samples.

$$\mathrm{x}=\left\{\begin{array}{c}{\mathrm{x}}_{\mathrm{1,1}},{\mathrm{x}}_{\mathrm{1,2}},\cdots ,{\mathrm{x}}_{1,\mathrm{m}-1},{\mathrm{x}}_{1,\mathrm{m}}\\ {\mathrm{x}}_{\mathrm{2,1}},{\mathrm{x}}_{\mathrm{1,2}},\cdots ,{\mathrm{x}}_{2,\mathrm{m}-1},{\mathrm{x}}_{2,\mathrm{m}}\\ \cdots \\ {\mathrm{x}}_{\mathrm{n}-\mathrm{1,1}},{\mathrm{x}}_{\mathrm{n}-\mathrm{1,2}},\cdots ,{\mathrm{x}}_{\mathrm{n}-1,\mathrm{m}-1},{\mathrm{x}}_{\mathrm{n}-1,\mathrm{m}}\\ {\mathrm{x}}_{\mathrm{n},1},{\mathrm{x}}_{\mathrm{n},2},\cdots ,{\mathrm{x}}_{\mathrm{n},\mathrm{m}-1},{\mathrm{x}}_{\mathrm{n},\mathrm{m}}\end{array}\right\}$$

(1)

We created a self-organizing mapping network model and set the weight vector between the input and output neurons to $w_{ij}$ to form the neural network model.

$$x^{\prime }=\frac{x}{‖x‖}$$

(2)

$$w_i^{\prime }(s)=\frac{w_i^{\prime }(s)}{‖w_i^{\prime }(s)‖}$$

(3)

We also created a self-organizing mapping network model and defined the weight vector between the input and output neurons as the composition of the neural network model.

Network training was also performed using the input spectral data and the nearest neuron was found via iterative training. The method used to calculate the distance is the Euclidean distance, and the expression for the Euclidean distance is shown in (2). The closer the weights are to the neuron, the smaller the distance.

$$\mathrm{dis}=‖x-w‖$$

(4)

In the SOM algorithm, not only does the winning neuron itself have to adjust the weight vector but so do the neurons in its vicinity. After the nearest neuron was found based on the Euclidean distance, the learning rate and neighborhood size parameters of the current iteration number were calculated to determine the neighborhood area. We updated the weight vectors of all neurons within the domain.

$$w(s+1)=w(s)+dis·\alpha (s)·(x-w(s))$$

(5)

α is the learning rate; 0 < α ≤ 1. α generally decreases with the learning progress; that is, the degree of adjustment becomes smaller and smaller, tending toward the cluster center.

We determined whether the maximum number of iterations had been reached. If not, we continued the iterative training. After the training of the network was completed, each spectrum was inputted into the neural network, and each spectrum corresponded to an output neuron to obtain the results of spectral classification (Figure 3).

We performed a linear correlation analysis between the average spectrum of the classification result and the standard spectrum, and the spectrum with the highest correlation was the effective spectrum. The correlation coefficient is a measure of the close relationship between variables. Equation (6) is the expression for the correlation coefficient.

$$r=\frac{{\displaystyle \sum (S_i-\overline{S})(M_i-\overline{M})}}{\sqrt{{\displaystyle \sum {(S_i-\overline{S})}^2{\displaystyle \sum {(M_i-\overline{M})}^2}}}}$$

(6)

where S represents the standard spectral intensity and M represents the average spectral intensity of the classification results.

3. Preprocessing Experiments for Simulated Spectra

3.1. Simulation of Abnormal Spectra

The sunlight falls on the spectrometer through the sun-tracking system, and the complex color light interference signal formed by the Michelson interferometer is received by the infrared detector. The relationship between the interference signal and the complex color light can be expressed as Equation (7):

$$I(x)=2I_0+2I_0{\displaystyle {\int }_0^{\infty }}{e}^{-i2\pi vx}dv$$

(7)

where ν is the wavenumber, x is the optical distance difference, and 2I₀ is the DC component of the interferometric signal. The AC signal in the spectral measurement is the effective interferogram (Figure 4a).

$$I(x)=2I_0{\displaystyle {\int }_0^{\infty }{e}^{-i2\pi vx}}dv$$

(8)

To simulate measurement conditions such as vehicle-mounted random vibrations, a random variable η(N) with a noise intensity S, a mean value of 0, a variance of 1, and a Gaussian distribution was added to the interferogram to obtain a simulated interferogram with Gaussian noise (Figure 4b).

$$I_{\mathrm{i}gm}(x)=I(x)+S·\eta (N)$$

(9)

To simulate measurement conditions, such as low-frequency components caused by the vibration of one or more optical components due to vehicle unevenness, engine vibration, wind, and electrical interference from the vehicle’s power supply, random sinusoidal (Figure 4c) noises with a fixed frequency were added to the interferograms, and simulated interferograms with the addition of sinusoidal noises were obtained.

$$I_{\mathrm{igm}}(x)=I(x)+A\times \sin (2\pi ft+\varphi )$$

(10)

where A, f, and φ are the amplitude, frequency, and phase of the sinusoidal noise, respectively.

To simulate fluctuations in incident light caused by vehicle bumps and wind, we added sinusoidal impact interference noise to the interference data (Figure 4d).

$$I_{\mathrm{igm}}(x)=I(x)+A\times \sin (2\pi ft+\varphi )+B\times (u(t-t_0)-u(t-t_0-T))$$

(11)

where u(t) is the unit step function (Heaviside function), t0 is the starting time, T is the duration, and B is the amplitude.

Equation (12) represents the computational relationship between the spectrogram and the interferogram. In Fourier transform spectroscopy, the spectral information in the frequency domain is obtained from the output signal of the measurement in the time domain, and the recovered spectrum in the frequency domain is denoted by $B(v)$. Figure 5 shows the interferograms with various interfering noises and the corresponding single-beam spectra.

$$B(v)={\displaystyle {\int }_0^{\infty }{I}_{igm}(x)e^{-i2\pi vx}}dx$$

(12)

3.2. Experimental Results and Analysis

MATLAB software (R2016b) was used to simulate 51 abnormal spectra and 49 effective solar spectra with different amplitudes and frequencies, forming a 10,160 × 100 multidimensional spectral data matrix to test the performance of the spectral data preprocessing method. The dataset consisting of 100 spectra was classified using an SOM. The neuron positions and 100 spectral datasets in the topology of the SOM are shown in Figure 6a, assigned to two neurons with maximum hits of 51 and 49. The correlation coefficients between the average and standard spectra of the spectral datasets in each neuron were calculated. The class with a high correlation is the valid spectra, and the other class is the abnormal spectra. From the SOM calculation results, we can see that the neuron input spectra with a correlation coefficient of 0.999 are effective and the neuron input spectra with a correlation coefficient of 0.878 are abnormal. Based on a spectral data preprocessing method combining the SOM neural network and the correlation analysis proposed in this paper, all the emulated abnormal spectra were successfully identified in the classification results. According to the results of the correlation calculation, all three abnormal spectra are strongly correlated with the standard spectra. The spectral data preprocessing method proposed in this paper, which combines the SOM neural network and correlation analysis, avoids the difficulty of automatically distinguishing abnormal spectra with higher correlation due to the inappropriate selection of threshold parameters for correlation judgment, thereby affecting the accuracy of gas concentration inversion.

4. Data Acquisition and Processing

4.1. Experimental Instruments

The SOF–FTIR system (Model AG-FTIR-SOF2000) included a solar tracking system (Figure 7a), a Global Positioning System (GPS) module, an FTIR spectrometer, and a computer (Figure 7b). We calculated the altitude and azimuth angle based on the GPS longitude and latitude information, roughly calculated the sun’s position, and used a position-sensitive detector to accurately record the changes in the sun’s position [19]. The solar spectra were obtained via the Fourier integral transformation of the measured interferograms and were used for the qualitative and quantitative analysis of the morphology of various substances.

4.2. Data Acquisition

Vehicle-mounted SOF–FTIR observations of NH₃ were conducted at an industrial park in Hefei City to verify the accuracy of the SOM-based data preprocessing method. The park mainly includes companies in the fields of video, photoelectrics, photovoltaics, liquid crystal glass, and new materials. For the mobile SOF–FTIR observation, sunlight selectively absorbed by the plume entered the FTIR spectrometer via the solar tracker (Figure 8).

The SOF–FTIR instrument lifted the solar track head out of the top of the vehicle via the lifting platform (Figure 8). The tracking vehicle traveled around the factory at an average speed of 40 km/h in a straight line and at a maximum speed of 20 km/h in curves.

4.3. Analysis of the Experimental Results

The 1739 spectral data points measured during mobile monitoring were summarized in a 10,160 × 1739 multidimensional spectral data matrix. The SOM was used to classify the dataset comprising the 1739 spectra. The neuron positions in the topology of the SOM and the 1739 spectral datasets are shown in Figure 9a and assigned to two neurons, where the maximum number of hits for the two neurons is 1399 and 340. We calculated the correlation coefficients between the average and standard spectra from the spectral datasets for each neuron. The class with high correlation is the effective spectrum, and the other class is the abnormal spectrum (Figure 9b). From the results of the SOM calculation, we can see that the neuron input spectra with a correlation coefficient of 0.938 are effective and those with a correlation coefficient of 0.812 are abnormal.

We selected the absorption cross-section for NH₃ from the HITRAN and OAsoft databases and performed nonlinear least squares fit, along with the instrument parameters, to calculate the concentration of the pollutant. The 915 cm⁻¹~980 cm⁻¹ band with less interference from water vapor and carbon dioxide was selected for analysis. The transmission absorption spectrum in the corresponding area of the infrared transmission spectrum showed a consistent absorption pattern (Figure 10a). In the stationary analysis, the concentration of the transmission spectrum was inverted and fitted using the least squares method. The average RMS of the rejected spectrum fitting residuals was 4.28%, and the average RMS of the effective spectrum fitting residuals was 0.95% (Figure 10b).

The NH₃ column concentrations before and after rejection are visualized using a time series and a three-dimensional distribution diagram (Figure 11). The monitoring results include the latitude, longitude, and NH₃ column concentration, and the gas concentration is represented by the small colored columns. From the mobile monitoring results, we can see that the average column concentration of NH₃ before the rejection of the abnormal spectra was 30.221 ppmm, and, after the rejection of the anomalous spectra via the spectral data preprocessing method proposed in this paper, the average column concentration of NH₃ was 26.648 ppmm.

5. Conclusions

We proposed a spectral rejection method for preprocessing data from vehicle-mounted Fourier transform spectrometers for occultation fluxes. This method introduces a neural network system, does not require the preparation of training spectra or the setting of fixed correlation comparison coefficients, and has automatic advantages in updating weights. Taking the column concentration measurement of NH₃ as an example, we analyzed 1739 experimentally measured spectra in the range of 915 cm⁻¹~980 cm⁻¹ and used the least squares method to fit the transmission spectra. The average RMS of the rejected spectral fit residuals was 4.28% and that of the effective spectral fit residuals was 0.95%. The experimental results show that the improved spectral data preprocessing method proposed in this paper can be used to process the spectral data from a vehicle-mounted occultation flux–Fourier transform spectrometer, which greatly improves the detection rate of abnormal spectra and reduces the impact of abnormal spectra on gas concentration calculations. To store more effective data, the next step will be to improve the tracking accuracy and speed of the solar tracking system and the stability of the spectrometer under vehicular motion.