The Identification and Assessment of Rail Corrugation Based on Computer Vision

Abstract

The identification and assessment of rail corrugation are two of the essential tasks of daily railway inspection to guarantee the safety of train operation and promote the development of an efficient maintenance strategy. In view of the requirements for automatic identification and smart decision-making, computer vision-based rail corrugation identification and assessment methods are presented in this paper. Firstly, an improved Spatial Pyramid Matching (SPM) model, integrating multi-features and locality-constrained linear coding (IMFLLC), is proposed for rail corrugation identification. After that, an innovative period estimation method for rail corrugation is proposed based on the frequency domain analysis of each column in the corrugation region. Finally, the severity of the rail corrugation is assessed with the help of the wear saliency calculation and fuzzy theory. The experiment results demonstrate that the proposed corrugation identification method achieves a higher precision rate and recall rate than those of traditional methods, reaching 99.67% and 98.34%, respectively. Besides, the validity and feasibility of the proposed methods for the rail corrugation period estimation and severity assessment are also investigated.

1. Introduction

Rail corrugation refers to the periodic non-uniform plastic deformation and wave-shaped irregularities along the longitudinal surface of rail heads [1]. In recent years, the development of urban rail transit has been severely affected by the problem of corrugation on the metro operation tracks. Due to the occurrence and evolution of rail corrugation, not only will the sudden changes in wheel-rail forces shorten the service life of the critical components of the rail line, which increase the maintenance costs but also the violent vibration between wheels and rails will generate great noise, reduce the passenger comfort and the safety of vehicle operation [2]. Corrugation detection has attracted more and more attention in recent years [1,3,4]. Therefore, it is of great significance to inspect corrugation in time and alleviate the development of corrugation effectively, which can reduce the cost of railway maintenance and ensure the safety of railway traffic.

For a long time, the detection methods for rail corrugation mainly based on manual inspection with a particular calliper or a corrugation inspection car. However, the fixed detection cycle cannot inspect the track state in time and this results in low detection efficiency [3]. Therefore, a growing number of research studies have been focused on improving the detection rate and accuracy. Nowadays, researchers have proposed a variety of new non-contact corrugation detection methods, including the chord surveying method, inertial reference method, computer vision method and so forth [5,6,7]. Among them, the first two methods are mostly based on mechanical and physical mechanisms, which are easy to calculate and handle but there are various factors to be considered, the anti-interference ability is insufficient and they are not efficient enough for a routine inspection. Conversely, with the rise of computer vision technology, the image processing technique has made significant achievements in the track inspection device, which has the advantages of being non-contact, low cost and fast. In addition, it can achieve multiple target detections simultaneously and analyze the images for a track field to figure out rail defects with advanced pattern recognition algorithms. Hence, it is remarkably suitable to performing long-term, high precision and repetitive tasks under complex environmental conditions [8,9,10,11].

At present, the vehicle track inspection system based on computer vision, mainly for rail fasteners and surface defects, has been used in some countries and the corresponding identification methods have been investigated and applied. Nevertheless, there are only a few reports on rail corrugation identification based on computer vision. Mandriota et al. proposed a rail corrugation identification method based on spatial filter characteristics and a machine learning algorithm [12]. The Gabor texture features are first extracted from the entire rail image and then the K-nearest neighbor method is used for corrugation recognition, in which a high recognition precision is obtained. It has become a widely used corrugation identification method based on computer vision [13] but it is sensitive to disturbances such as rust on the rail surface. Li et al. proposed an identification method for rail corrugation based on rail image features in the frequency domain, including the image acquisition subsystem and the corrugation identification subsystem installed under the vehicle body [14,15]. Instead of global texture features, local frequency features used in this method can reduce the time-consuming problem of traditional image methods and further improve the identification accuracy. However, it is still only able to detect the existence of corrugation and cannot analyze the periodicity and severity of the corrugation. Therefore, it is necessary to make some further efforts on this issue. These motivate the following works.

In this paper, the rail corrugation identification is converted into an image classification problem. Specifically, a learned classifier is used to determine whether the captured rail image is a corrugation rail image or not. At present, image classification methods based on local image descriptors are widely applied, which mainly includes Bag of Features (BOF) based method [16] and a Spatial Pyramid Matching (SPM) based method [17]. What is crucial to this kind of classification method is visual feature extraction and feature encoding process [18]. However, the traditional BOF or SPM only extracts a single feature of the image and the obtained feature information is not sufficient to adequately represent the rail image in many cases. In addition, by adopting the vector quantization (VQ) encoding method, the reconstruction error is large. These result in low image classification accuracy [19]. Therefore, taking the characteristics of corrugation rail image into account, the integrated idea is adopted to improve rail image classification performance. At first, to extract the image features more effectively, the Uniform LBP [20] and dense_SIFT [21] are used simultaneously to construct the feature set of the images. After that, to express the image information more specifically, codebooks for the two feature sets are created separately by using k-means clustering [22] to improve the robustness of the results. Finally, to reduce the loss of image information while improving the accuracy of the calculation, the LLC (locality-constrained linear coding) [23] is used to encode each image feature and further merged as the final image feature vector. In summary, an improved SPM model based on integrated multi-features and locality-constrained linear coding (IMFLLC) for rail corrugation identification is proposed. This method is expected to achieve a better identification result.

In addition, as an important visual property of rail corrugation, period plays a vital role in further corrugation analyzing. Inspired by References [24,25], a corrugation period estimation method in combination with image Fourier analysis and normalized distance matching function [25] (FFT-NDMF) is presented to achieve the automatic estimation of the corrugation period along the rail direction. It is useful to analyze these factors which result in the corrugation of the rail. To our best knowledge, this is the first paper handling the rail corrugation period estimation problem based on computer vision.

Furthermore, valid and reliable corrugation severity assessment is helpful to make maintenance strategies for a lower maintenance cost and to extend the service life of the rails. Therefore, given the successful application of visual saliency in target detection and content analysis [26], a novel method for rail corrugation severity assessment based on synthesizing the visual saliency and fuzzy theory is proposed in this paper.

The remainder of this paper is organized as follows. In Section 2, the problem solved in this paper is stated. In Section 3, the proposed rail corrugation identification method is presented in detail. In Section 4, the steps of the rail corrugation period estimation using the FFT-NDMF algorithm is investigated. After that, the method of the rail corrugation severity assessment is provided in Section 5. Finally, some conclusions of this paper are given in Section 6.

2. Problem Statement

For quite some time, the occurrence and evolution of rail corrugation have seriously affected the development of urban rail transit. The rail corrugation not only causes strong vibration of the vehicle-track structure but also strengthens the wheel-rail rolling noise, which reduces the passenger comfort and increases the track maintenance cost. Therefore, it is critically important to identify and assess rail corrugation automatically and intelligently. The corrugation is a periodic irregular wear phenomenon on the rail surface and a rail image contains corrugation is shown as in Figure 1, which is collected from the track line between the Nanluoguxiang Station and the Dongsi Station of the Beijing Metro Line 6.

In this paper, for real application, three issues will be investigated. First, the identification method of rail corrugation. Second, the method of estimating the rail corrugation period. Finally, the severity assessment method of rail corrugation. The ideas how to solve these problems are introduced in the following.

Rail corrugation identification based on improved SPM model. First of all, to reduce the impact of noise and uneven illumination, a two-stage pre-processing method is considered to improve the original image quality. After that, rail head surface positioning is achieved by enhancing the rail its edge followed by weighted gradient projection analysis. Finally, according to the local texture and gradient features of the corrugation, an improved SPM model is proposed to identify rail corrugation.

Estimation of rail corrugation period. To estimate the rail corrugation period more efficiently, the horizontal gray gradient statistical analysis and mathematic morphology algorithm are first used to localize the corrugation interval. After that, the proposed method by integrating the fast Fourier transform and normalized distance matching function is applied to estimate the rail corrugation period.

The severity assessment of the rail corrugation. Based on the visual significance analysis for the rail corrugation image, the concept of wear saliency is introduced to assess the corrugation severity. In detail, the normalized frequency-tuned model is used to calculate the corrugation saliency map. Subsequently, each corrugation area is localized by employing the adaptive threshold and mathematic morphology algorithm. In the end, after calculating the wear saliency, each corrugation area severity is determined with the help of fuzzy theory.

3. Rail Corrugation Identification Based on Improved SPM Model

In this section, the whole frame and relevant details of the rail corrugation identification method proposed in this paper are presented.

3.1. Image Pre-Processing

Owing to the complex operating environment of the metro railway system, the rail images collected from the field are susceptible to noise and uneven illumination, which would degrade the image quality, affect the ultimate rail corrugation identification precision and impact the period estimation results of the rail surface images. To alleviate this problem and improve the image quality, a two-stage pre-processing method by Bilateral Filter (BF) [27] and Rolling Guidance Filter [28] with Parameter Adaptive Gamma Function [29] (RGF-PAGF) is considered, which are reviewed in the following.

3.1.1. Image De-Noising

The first stage of pre-processing is noise reduction, which is implemented by using image filters. The bilateral filter [27] is a non-linear filter that can achieve noise reduction and edge-preserving simultaneously. In addition, in comparison with other conventional filter-based methods, it can commendably preserve the rail surface texture characteristics. Hence, it is suitable for image de-noising. The bilateral filter is developed from the traditional Gaussian filter, which can be described as follows

$$I_{out}(x)=\frac{1}{W_p}\sum _{x_i\in \Omega }I(x_i)f_r(\parallel I(x_i)-I(x)\parallel )g_s(\parallel x_i-x\parallel )$$

(1)

where I and $I_{out}$ represent the original and filtered image, respectively. x is the coordinates of the pixels to be processed while $\Omega$ is the neighborhood window. $f_r$ and $g_s$ denote the distance and intensity filter, respectively. $W_p$ stands for the normalization function. More details about BF can be found in Reference [27].

3.1.2. Uneven Illumination Correction

According to Retinex theory [30], a visible image consists of two components: illumination and reflectance. That is, the visible image $I(x,y)$ can be described as the product of illumination component and reflectance component as follows [30]

$$I(x,y)=L(x,y)•R(x,y)$$

(2)

where $(x,y)$ denote the pixel coordinates. $L(x,y)$ and $R(x,y)$ represent the illumination and reflectance, respectively. In general, the illumination components exist in the low-frequency part of the image and varies smoothly. Therefore, to correct the uneven illumination phenomenon, it is crucial to extract the illumination component efficiently from the observed image.

There are several filter based methods for extracting illumination component [31,32,33]. In this paper, given the outstanding edge-preserving capabilities and the low computation complexity, the rolling guidance filter (RGF) [28] is used to extract the illumination. It is a new type of edge-preserving filter based on a scale structure, which can be formulated as follows

$$J^{t+1}(p)=\frac{1}{K_p}\sum _{q\in N(p)}exp(-\frac{(\parallel p-{q\parallel )}^2}{2{\sigma }_s^2}-\frac{(\parallel J^t(p)-J^t(q){\parallel )}^2}{2{\sigma }_r^2})I(q)$$

(3)

where $J^{t+1}$ denotes the $t+1$ iteration result which is the extracted illumination component of the input image I (also the $J^1$). $K_p$ stands for the normalization function. Other parameters are the same as the one mentioned above.

After that, an uneven illumination correction function is constructed. In the light of the parameter adaptive gamma function (PAGF) mentioned in References [29,34], the constructed correction function is given by

$$O(x,y)=I{(x,y)}^{\gamma },\gamma =2^{\frac{m-L(x,y)}{m}}$$

(4)

where $O(x,y)$ stands for the enhanced intensity value. $\gamma$ denotes the exponential value including the dynamic local characteristic of the intensity enhancement. m represents the mean value of the intensity for illumination component. The whole procedure of the two-stage pre-processing method is shown as in Figure 2.

3.2. Rail Head Surface Localization

The rail images collected from the field usually contain irrelevant components such as fasteners, sleepers and subgrade on both sides of the track, which would cause unnecessary interference. Therefore, the exact rail head surface area should be first extracted to reduce this interference. It is observed that the gray level of the rail area is mainly continuous along the longitudinal direction of the rail, while in the other non-rail regions, the gray scale of the pixels has apparent discontinuity both on the horizontal and vertical direction. And, along with the transverse direction, there are distinct gray scale discontinuities between the rail area and the other regions. Furthermore, the rail is usually located in the middle of the obtained rail image.

Based on these observations, the vertical edge of the rail image is enhanced by using horizontal gradient operator to highlight gray level change along the horizontal direction and sharpen the boundaries of the rail head surface. From Reference [35], the operator templates used in this paper are stated as follows

$$p=\left[\begin{array}{ccc}-1& 2& -1\\ -2& 4& -2\\ -1& 2& -1\end{array}\right],q=\left[\begin{array}{ccc}-1& 0& 1\\ -2& 0& 2\\ -1& 0& 1\end{array}\right]$$

(5)

hence, the enhanced rail image $E(x,y)$ is computed as

$$E(x,y)=|p\ast O(x,y)|+|q\ast O(x,y)|$$

(6)

where $(x,y)$ denote the pixel coordinates. $O(x,y)$ is the corrected rail image. * represents the convolution operation.

The enhanced rail boundaries indicate significant gradient information along the transversal direction, as shown in Figure 3b. Therefore, the horizontal gradient for each pixel of the enhanced rail image is first computed and then the sum of gradient value (SG) for each column of the rail image is calculated, the SG is defined as

$$SG(y)=\sum _{x=1}^{h}grad(x,y),y=1,2,\dots ,w.$$

(7)

where w and h stand for the width and height of the rail image, respectively. $grad(x,y)$ represent the gradient value at the coordinates $(x,y)$.

Subsequently, in terms of the horizontal position of each column in the rail image, the SG is further multiplied with an adaptive weight. The adaptive weight $AW(y)$ of position y is defined as

$$AW(y)=\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{{(y-\mu )}^2}{2{\sigma }^2}},y=1,2,\dots ,w.$$

(8)

where $\mu$ is set with $w/2$ in this study. Therefore, the weighted gradient SUM value is computed as

$$WG(y)=SG(y)\times AW(y),y=1,2,\dots ,w.$$

(9)

Finally, a weighted gradient projection profile for rail locating is formed by using all of these values. In the light of practical experience, when the setup of image acquisition equipment is fixed, the rail width $w_r$ in a rail image will be a constant. Therefore, the starting position of the rail is obtained based on the following rules [15]

$$p_r=arg\underset{x}{max}SWG(y),y=1,2,\dots ,w-w_r.$$

(10)

with the sum of weighted gradient value

$$SWG(y)=\sum _{i=y}^{i+w_r-1}WG(i)$$

(11)

where y stands for the starting position with the length of $w_r$ and then the exact rail location would be determined by them. The procedure of rail surface localization is shown as in Figure 3. Brief pseudocode about the method for rail surface localization is described in Appendix A.

3.3. Improved Spatial Pyramid Matching (SPM) Model

The traditional SPM (Spatial Pyramid Matching) model [17] has developed from the BOF (Bag of Features) method [16]. To overcome the inherent demerits of BOF, SPM introduces the spatial location information between local feature points by calculating the distribution of image feature points at a different resolution [17], as shown in Figure 4. That is, the image is divided into progressively refiner spatial sub-regions and then local feature histograms are computed from each region. The diagram of SPM is shown in Figure 5.

Although SPM exhibits excellent performance in many image recognition and classification tasks, it only extracts a single feature of an image leading to the obtained feature information is insufficient to represent the rail corrugation image adequately. Moreover, the vector quantization (VQ) encoding method used in the SPM has a large reconstruction error and will result in low image classification accuracy. Therefore, taking the characteristics of rail corrugation image into account, an improved SPM model based on integrated multi-features and locality-constrained linear coding (IMFLLC) for rail corrugation identification is proposed, which is reviewed briefly as follows.

3.3.1. Multi-Feature Extraction

In comparison with normal rails without corrugation, typical corrugation rails exhibit distinctive textural features. A widely-used method for texture feature extraction is the local binary pattern (LBP). To extract the textural characteristics of rail image more effectually, the rotation invariant Uniform LBP [20] that the number of patterns decreased from 256 to 59 is used in this paper. It significantly compresses the dimension of the eigenvector and reduces the effect of high-frequency noise, allowing for better representation of image information under a variety of complex conditions.

Another local feature descriptor with excellent performance is SIFT, which has been widely applied since was first reported [36]. In this paper, the dense_SIFT proposed by Liu et al. [21] is applied. It takes a fixed-size mask and extracts the dense_SIFT patches from left to right and top to bottom on the image with a certain step size and then extracts 128-dimensional SIFT descriptors from each patch as the local feature descriptors. Compared to the traditional SIFT, dense_SIFT reduces the computation complexity and extracts dense_SIFT features that can be better used for linear classification.

In view of the well-characterized ability and complementarity of the two local feature descriptors mentioned above, the image features are extracted by using Uniform LBP and dense_SIFT, respectively, which can reflect the different properties of the image data effectively. After that, these two features are used in combination to train the classifier. This can take full advantages of the two methods, therefore, a better classifier performance can be obtained.

3.3.2. Vector Quantization of Image Features

The low-level features extracted above may contain some redundancy and noise. In this paper, to improve the robustness and distinctiveness of feature representation, an efficient feature vector quantization method is needed to encode the low-level features. In general, the vector quantization method of image features can be divided into two main steps [18]. First of all, the visual codebook is obtained by clustering the descriptors of the image training set and then the features extracted from the image are encoded according to the obtained visual codebook. Different visual codebook or different feature encoding methods will lead to different performance during the quantization procedure.

In general, there are several clustering algorithms for obtaining sparse or dense visual codebook from the image training set, of which the most widely used is the K-means clustering algorithm. It is fast, straightforward and still can maintain high efficiency and scalability when the data volume is large [22]. Therefore, the most classical partition-based K-means clustering method is first used to generate different visual codebooks for the two types of features extracted from the image training set, which can be used to comprehensively express different types of image information, therefore, improving the quality and robustness of the results.

Furthermore, to effectively organize the feature vectors and reduce the reconstruction error of image features, the LLC (locality-constrained linear coding) is used to encode local image feature descriptor. In comparison with VQ used in traditional SPM, LLC encodes and reconstructs low-level features on a local manifold by adding locality constraints [23], as shown in Figure 6. In addition, the output image feature representation based on LLC is nonlinear and satisfactory results can be obtained by using a linear classifier.

3.4. Rail Corrugation Identification Based on IMFLLC

In this paper, to achieve better identification accuracy, an improved SPM model for rail corrugation identification is proposed, which integrates multi-features and locality-constrained linear coding, named IMFLLC. The diagram of IMFLLC is shown as in Figure 7.

More specifically, the Uniform LBP and dense_SIFT of all training rail images are first extracted to construct two corresponding feature sets. Secondly, the K-means clustering algorithm is used to cluster the two feature sets of the training image, respectively. As a result, two visual codebooks are obtained which can represent different aspects of the image information. After that, based on the two obtained visual codebooks, LLC is used to encode each image, followed by SPM pooling to receive two different image representations of each image. Then, different image representations are integrated to form the final image representation of each image. Finally, the final image representations with artificial labels are used to train a linear SVM classifier [8].

For new or test rail images, the Uniform LBP and dense_SIFT feature sets are first extracted. Subsequently, based on the corresponding two different visual codebooks, each image feature is encoded by using LLC and the final feature representation of each image is obtained after SPM pooling and feature integration. Finally, the linear SVM classifier obtained from the training procedure is used to test the image and get the final class label to judge whether it is a corrugation rail image or not.

3.5. Identification Experiments and Results

3.5.1. Introduction to Experiment Dataset

The dataset used in this paper consists of images taken from Beijing Metro Line 6. These images are captured by the handheld DSLR camera. Additionally, the collected images are characterized in that the track is located in the middle of the image and each image is an RGB image with a resolution of 3456 × 4608 pixels. The dataset contains 200 typical corrugation rail images as positive samples and 200 non-corrugation images as negative samples. In the experiment, 70% of the images of each class are used as the training images and the rest is used as the test images.

3.5.2. Performance Evaluation for Corrugation Identification

Corrugation identification is one of the critical parts of this work and this subsection will evaluate the performance of the proposed method for corrugation identification. The proposed identification method is compared with three baselines including Gabor+SVM [13], Accumulate Energy Thresholding (AET) [14] and Maximum Energy (ME)+SVM [15]. More precisely, the global 8-dimensional Gabor filtering features and SVM classifier are adopted in Gabor+SVM. AET uses the local dominant frequency features and fixed thresholding. And the local dominant frequency features and SVM classifier are applied in ME+SVM. In addition, all parameters are consistent with the original literature and 10-fold cross validation method for SVM training and prediction is used.

In this subsection, to assess the proposed method quantitatively, indicators such as precision (P), recall (R) and accuracy (ACC) are used, which are widely adopted in many research fields [37]. The formula for calculating these indicators as follows:

$$P=\frac{TP}{TP+FP}$$

(12)

$$R=\frac{TP}{TP+FN}$$

(13)

$$ACC=\frac{TP+TN}{TP+FP+TN+FN}$$

(14)

where $TP$ represents the number of corrugation images that are correctly detected. $FN$ indicates the number of corrugation images for missed inspection. $FP$ is the number of images that are falsely reported as corrugation. $TN$ stands for the number of non-corrugation images that are correctly excluded.

Table 1. The comparison of detection performance between the proposed method and the baselines.

Method	Precision (%)	Recall (%)	Accuracy (%)
Gabor + SVM	91.80	93.33	92.50
AET	97.00	96.50	96.80
ME + SVM	98.97	97.00	98.00
The proposed	99.67	98.34	99.00

shows the comparison result between the traditional methods and the proposed method. Note that the size of the two visual codebooks is adaptively set to $patchNum\text{\_}sift\ast 0.7$ and $patchNum\text{\_}lbp\ast 0.7$, respectively ( $patchNum\text{\_}sift$ is the number of patches extracted according to dense_SIFT and $patchNum\text{\_}lbp$ refers to the number of patches extracted according to Uniform LBP). It can be seen that the algorithm proposed in this paper achieves a sound performance, especially the recall rate is significantly improved. The recall rate is the most critical indicator among the three indicators and the high recall rate means that the number of undetected corrugation images is small. In summary, in the light of the most distinguishable texture features and spatial structure relationship of the corrugation, the multi-features extracted in this paper can represent the corrugation image more comprehensively than those mentioned in the baselines. Moreover, the results demonstrate that the LLC method can facilitate precise and efficient encoding of the corrugation image features, which benefits from the degradation of image feature reconstruction error and the effective organization of image feature vectors.

3.5.3. Analysis of Computational Complexity

Computational complexity is considered to be another significant indicator of the corrugation identification algorithms. Therefore, in this subsection, the computational time complexity of the proposed IMFLLC algorithm is theoretically analyzed at first and then the practical identification speed is given.

In general, computational complexity consists of space complexity and time complexity. The reason why only time complexity is considered here is that the time source is more important than memory space for the corrugation identification algorithms. The time complexity of an algorithm is a function that qualitatively describes the running time of the algorithm. In addition, the complexity function is usually represented by $O(.)$ notation, which excludes the low-order terms and the first coefficient of this function [38].

At first, the computational time complexity function of IMFLLC is analyzed. As discussed in Section 3.4, IMFLLC-based corrugation identification involves three key steps: (1) rail image multi-feature extraction; (2) vector quantization of extracted features; and (3) rail corrugation identification. In the first step, the Uniform LBP and dense_SIFT feature of the input rail image are extracted, respectively. In terms of the feature extraction principle of Uniform LBP [20] and dense_SIFT [21], the time complexity of the first step is approximately equal to $O(N)$, where N is the pixel number of the input rail image. Secondly, by using LLC method, the computational complexity of vector quantization procedure is $O(M+K^2)$ [23], in which M denotes the codebook size and K stands for the nearest neighbors number. In the last step, based on the linear SVM classifier obtained from the training procedure, the input rail image is judged whether it is a corrugation rail image or not. This procedure using simple linear SVM obtains a constant complexity, so the computational time complexity of rail corrugation identification is $O(1)$. As a whole, the time complexity of IMFLLC is $(O(N)+O(M+K^2)+O(1))$.

After that, the actual time of a corrugation rail image identification is tested. This experiment was carried on a desktop with Intel(R) Core(TM)2 Duo CPU (2.93 GHz) and 4-GB memory. The proposed IMFLLC algorithm is implemented with MATLAB. Furthermore, the program is performed on the test images and the mean of the run time is recorded. The final average identification time of the corrugation rail image is 0.27 s, which indicates that an acceptable corrugation identification speed based on the IMFLLC can be achieved.

4. Estimation of Rail Corrugation Period

The corrugation is a type of wave-like rail surface defect that appears as a periodic texture pattern in the rail surface image. In general, the wavelength of corrugation is used to further analysis corrugation and mathematically, the wavelength is proportional to the period. Hence, it is meaningful to estimate the period of the corrugation. However, the corrugation rail image obtained from the field includes not only the corrugation area but also other regions (such as edge rust and normal surface area), as shown in Figure 8a. Therefore, the corrugation interval is first localized and then the corrugation period is estimated based on the combination of one-dimensional fast Fourier transform [39] and normalized distance matching function (FFT-NDMF) in the corrugation interval.

4.1. Corrugation Interval Localization

Due to the complex operation environment of the vehicle and the wheel-rail interaction, there is rust on the edges of the rail surface. Besides, it is observed that there is a significant gray scale difference between the rusted edge and the remaining rail surface area along the horizontal direction. Therefore, the rusted edge is first cropped from rail surface image based on the horizontal gray gradient statistical (briefly as HGGS) analysis. The starting and ending coordinates of the remaining rail surface area are computed as follows

$$mgrad(x_i,y)=\frac{1}{h}\sum _{y=1}^h[I(x_{i+1},y)-I(x_{i-1},y)],i=2,3,\dots ,w-1.$$

(15)

$$R_l=\underset{i}{min}\left[peaks(mgra{d}_{2\le i\le w/2}(x_i,y))\right]$$

(16)

$$R_r=\underset{i}{max}\left[peaks(mgra{d}_{w/2\le i\le w-1}(x_i,y))\right]$$

(17)

where $mgrad(x_i,y)$ denotes the average horizontal gradient value of the i-th column of the image. $I(x_i,y)$ is the gray value of the i-th column of the image. w and h stand for the width and height of the rail image, respectively. $peaks$ represents the calculation of the extreme values of $mgrad$. $R_l$ and $R_r$ are the starting and ending coordinates of the remaining rail surface area, respectively. The result of the horizontal gray gradient statistical analysis are shown as in Figure 8b. Brief pseudocode for HGGS is shown in Appendix B.

After eliminating the interference of rust edge, the Otsu algorithm [40] is used to achieve the segmentation of corrugation area and the background area, resulting in a binary image B. After that, the mathematic morphological reconstruction and opening operators are used to remove small, bright pixels. Finally, the corrugation interval is localized based on the foreground pixel statistical analysis of each column of the image. The starting and ending coordinates of the corrugation interval are computed as follows

$$C_l={arg}_{i}min[S{B}_i|S{B}_i>\delta ],i=1,2,\dots ,m.$$

(18)

$$C_r={arg}_{i}max[S{B}_i|S{B}_i>\delta ],i=1,2,\dots ,m.$$

(19)

where $S{B}_i$ denotes the sum binary value of the i-th column of the obtained binary image. m represents the width of the remaining rail surface area. $C_l$ and $C_r$ are the starting and ending coordinates, respectively. $\delta$ stands for the threshold of the sum binary value. The result of the corrugation interval localization is shown as in Figure 8a.

4.2. Corrugation Period Estimation Based on FFT-NDMF

After the corrugation interval is localized, Fourier transformation analysis is performed on each column of the corrugation interval image. Subsequently, the simplified NDMF is used to obtain the period of each column. Finally, the results of the entire interval are statistically analyzed to determine the period of the corrugation. The diagram of FFT-NDMF is shown as in Figure 9.

Specifically, let L be a set of H-dimensional image columns obtained from the corrugation interval, that is, $L=[l_1,l_2,\dots ,l_N]\in {\Re }^{H\times N}$ where $l_i$ represents a column of the corrugation interval image at position i, H denotes the height of the corrugation interval image and N is the width of the corrugation interval. Firstly, one-dimensional fast Fourier transformation is conducted to each column of interval image, that is

$$f_m=\sum _{k=0}^{H-1}{l}_{ik}{e}^{\frac{-j2\pi mk}{H}},1\le i\le N$$

(20)

where $l_{ik}$ denotes the gray value of the image column $l_i$ at position k. $f_m$ for $0\le m\le H-1$ represents the Fourier coefficients with frequency m. As a result, an amplitude-frequency curve for each column is obtained. In this section, to better introduce subsequent steps, the amplitude-frequency curve of some interval image columns randomly selected in corrugation interval is shown as in Figure 10.

Each curve has an amplitude peak at the same frequency, as observed in Figure 10. In most cases, the peak is the maximum but in a few cases, the peak is the second maximum due to the interference of the sub-period or the multiple of the main period. Therefore, when locating the peak value of each amplitude-frequency curve, the frequency corresponding to the first two maximum peaks are extracted simultaneously and then the corresponding period is calculated as the possible period of each image column. On this basis, NDMF is calculated to determine the final period of each image column.

The traditional distance matching function (DMF) proposed by Oh et al. [24] is the most used method for period estimation, which is based on the gray level co-occurrence matrix (GLCM) inertial characteristics [24]. In practice, to improve its robustness to the data differences, enhance the stability of the algorithm and reduce the amount of computation, the normalized distance matching function (NDMF) is often used [25]. To meet the period estimation demand of the corrugation interval image, a simplified NDMF of a one-dimensional image function $l_j$ with h pixels in corrugation interval is defined as

$$NDMF(d)=\frac{1}{H-d}\sum _{i=1}^{H-d}|l_j(i)-l_j(i+d)|$$

(21)

where H represents the height of the corrugation interval image. $l_j$ stands for a column of the corrugation interval image at position j.

First of all, let $d_{i1}$ and $d_{i2}$ be the potential periods corresponding to the maximum and the second maximum peak value of the amplitude-frequency curve of image column $l_i$, respectively. Then, the NDMF values of $d_{i1}$ and $d_{i2}$ are calculated to determine the exact period, which is given by

$$NDM{F}_{i1}(d_{i1})=\frac{1}{H-d_{i1}}\sum _{j=1}^{H-d_{i1}}|l_{ij}-l_{i(j+d_{i1})}|$$

(22)

$$NDM{F}_{i2}(d_{i2})=\frac{1}{H-d_{i2}}\sum _{j=1}^{H-d_{i2}}|l_{ij}-l_{i(j+d_{i2})}|$$

(23)

where H denotes the height of the corrugation interval image. Given the definition of NDMF, the smaller the NDMF value calculated by Equation (21) is, the greater the probability that it is the real period is. Hence, the final period of the image column $l_i$ is $p_i=min(NDM{F}_{i1},NDM{F}_{i2})$.

Finally, the period value of each image column is statistically analyzed in the corrugation interval and the mode of all period values is calculated as the final period of the corrugation rail image, that is

$$P_C=mod(P)$$

(24)

where $P=[P_1,P_2,\dots ,P_N]$ denotes the period value set of the corrugation interval. N is the width of the corrugation interval.

4.3. Performance Evaluation for Corrugation Period Estimation

In this subsection, to verify the validity and accuracy of the proposed method in estimating rail corrugation period, experiments are performed on images selected from the corrugation image dataset mentioned above. Figure 11 shows the visual period estimation results of some samples, with the red boxes indicating the period estimated by the method in this paper. The results show that the proposed method of period estimation can be applied to rail corrugation images, which is consistent with the artificial visual result.

In addition,

Table 2. The period estimation results of corrugation displayed in Figure 11.

Sample Image	(a)	(b)	(c)	(d)	(e)	(f)
Frequency (num)	5	4	4	3	5	6
Period (pixel)	102.4	128	128	170.7	102.4	85.3
Manual result (pixel)	102	128	128	172	102	85

presents the corresponding concrete results of the corrugation periodicity in Figure 11, note that the height of the images used in this experiment is 512 pixels. The comparison between the estimated period and the manual measurement shown in Table 2 demonstrate that the estimation method proposed in this paper can estimate the rail corrugation period effectively.

5. The Severity Assessment of the Corrugation

The evolution of the rail corrugation not only affects the security state of the rail line but also increases the maintenance costs. Therefore, it is critically important to assess the severity of corrugation to provide auxiliary maintenance strategy. Generally, the severity of the corrugation is related to the wave depth of corrugation, while the depth variation of the wear is reflected in the change of the gray level of the image [41], that is, the difference in grayscale of the corrugation image can be used to assess the severity of the corrugation. In this paper, it is considered as the issue of visual saliency calculation. For a specific wear state, the concept of wear saliency (WS) is introduced to assess the severity of corrugation, in which $0<WS<1$. When the wear saliency is close to 0, it means that there is no corrugation. Correspondingly, when the wear saliency is close to 1, it means that the corrugation is severer. Concretely, the rail corrugation is classified into three levels of wear severity including level 1 (slight wear), level 2 (medium wear) and level 3 (severe wear).

5.1. Visual Significance Analysis for Corrugation Image

At present, visual saliency has been most applied in many fields such as target detection and tracking, image retrieval and recognition, scene analysis and so forth [26]. Saliency identifies a particular area of a scene that has unique properties and can attract more visual attention and the area is considered to be visually significant. For humans, the most eye-catching is often the goal with different colors, because the human visual system (HSV) is most sensitive to color. In other words, the targets which are different from the surrounding colors will most likely attract attention. Moreover, as can be seen in Figure 1, the color of the defect area and the background area are significantly different in the corrugation image. There are also significant differences between the corrugation defect areas of varying severity. Hence, compared with other low-level visual features, the color characteristics of the defect area of the corrugation image are more significant.

5.2. Visual Saliency Feature Extraction and Saliency Map Generation

Nowadays, there are several ways to extract significant visual features and further generate saliency maps, mainly including spatial-based and frequency-based methods. Among them, the frequency domain-based saliency detection method is increasingly investigated owing to its simple implementation and high real-time performance. In this paper, in terms of the characteristic of corrugation images, the significant visual features are extracted by the frequency-tuned (FT) model [42] and the full resolution saliency map is further obtained. In the first step, the input RGB image is converted into the CIELab color space closer to human vision. After that, the arithmetic mean of the image features is calculated and Gaussian smoothing is performed on the image. Finally, the difference of each pixel is calculated and the saliency map is obtained after the value is normalized. The saliency map S for a corrugation image I can be written as

$$S(x,y)=N(\parallel L_G(x,y)-L_m\parallel )$$

(25)

where $(x,y)$ denote the pixel coordinates. $L_m$ stands for the average image feature vector in CIELab color space. $L_G(x,y)$ is the relevant image pixel vector value in the intermediate image blurred by a 5 × 5 Gaussian kernel. $\parallel ·\parallel$ is the $L_2$ norm. $N(·)$ represents the min-max normalization operation.

5.3. Severity Assessment Based on Fuzzy Theory

After the saliency map is obtained, the rail corrugation severity is assessed based on fuzzy theory [43]. Fuzzy theory is an effective tool to deal with uncertain information. Hence, it is suitable for solving the problem of corrugation severity assessment with ambiguity. The whole procedure of corrugation severity assessment is shown as in Figure 12.

5.3.1. Image Binarization and Corrugation Area Localization

In general, image binarization can reduce the amount of computation and can simplify the localizing process of each corrugation area. Therefore, the adaptive threshold algorithm, mentioned in Reference [42], is used to get the binarized image of the corrugation saliency map, in which the adaptive threshold $T_a$ is computed as follows

$$T_a=\frac{2}{w\times h}\sum _{x=0}^{w-1}\sum _{y=0}^{h-1}S(x,y)$$

(26)

where w and h stand for the width and height of the corrugation saliency map, respectively. $S(x,y)$ denote the saliency value at the coordinates $(x,y)$.

After that, with the intention of localizing each corrugation area more accurately, the mathematical morphology algorithm is used. Firstly, the morphological closing operation is used to merge smaller regions without changing the overall region, that is

$$S_m=(S_b\oplus b)\Theta b$$

(27)

where $S_b$ denotes the binarized saliency map. b is the structure element used in this paper. ⊕ and $\Theta$ represent dilation operation and erosion operation, respectively. Subsequently, the label matrix method is used to decide the position coordinates of each corrugation area, which is given by

$$S_{c_i}=S_m(L_s==i),i=1,2,\dots ,k$$

(28)

where $S_{c_i}$ represents the i-th corrugation area. $L_S$ stands for the label matrix of $S_m$. k is the number of corrugation area. The visual results of the used mathematical morphology algorithm are shown in Figure 13, where the scope of each corrugation area is accurately located without interference.

5.3.2. Corrugation Severity Assessment

First of all, taking the relationship between visual saliency and wear state into account comprehensively, the concept of wear saliency (WS) is introduced to assess the severity of corrugation, which is defined as

$$W{S}_{c_i}=mean(S_{c_i}),i=1,2,\dots ,k$$

(29)

where $W{S}_{c_i}$ is the wear saliency of the i-th corrugation area.

After that, three level membership functions are designed for the slight wear, medium wear and severe wear, respectively, which are shown as in Figure 14.

Finally, the maximum subordination principle is used to determine the severity level of each corrugation area, which is defined as

$$max({\mu }_{slight},{\mu }_{medium},{\mu }_{severe})=\left\{\begin{array}{ccc}\hfill {\mu }_{slight}& & level1\hfill \\ \hfill {\mu }_{medium}& & level2\hfill \\ \hfill {\mu }_{severe}& & level3\hfill \end{array}\right.$$

(30)

5.4. Case Study for Rail Corrugation Severity Assessment

In this subsection, the visualization process and feasibility of the proposed corrugation severity assessment method will be demonstrated by a case study. The image used in this case, with a slight corrugation, is collected from the small radius curve between the Nanluoguxiang Station and the Dongsi Station of the Beijing Metro Line 6.

In the first step, the horizontal gray gradient statistical (HGGS) analysis method is applied to crop the rusted edge of the corrugation image (see (b) in Figure 15). Then, the rustless corrugation image is converted to the CIELab color space and the saliency map of it is obtained by using the normalized frequency-tuned (NFT) model (see (c) in Figure 15). Next, the adaptive threshold method is used to get the binarized corrugation saliency map (see (d) in Figure 15) and then, the morphological closing operation and label matrix method are adopted to localize each corrugation area in it. After that, the wear saliency (WS) of each corrugation area is calculated according to Equation (29) (see (e) in Figure 15). Finally, the severity level of each corrugation is determined in terms of the level membership function and maximum subordination principle defined in Equation (30) (see (f) in Figure 15). The visualization process and results are shown as in Figure 15.

In summary, according to the site actual conditions, the visualization process and severity assessment results of this case demonstrate that the proposed method is a feasible approach for assessing the rail corrugation severity.

6. Conclusions

In this paper, the rail corrugation identification and assessment issues are concerned. Firstly, the improved SPM model for rail corrugation identification is introduced, in which the multi-features (including Uniform LBP and dense_SIFT) and locality-constrained linear coding are used. The quantitative experimental results indicate that the proposed identification method achieves markedly better performance. Specifically, the precision and recall of the proposed corrugation identification method are 99.67% and 98.34%. After that, to further analysis the corrugation, an innovative estimation method for rail corrugation period based on one-dimensional fast Fourier transform and normalized distance matching function is proposed. The experimental results demonstrate that the proposed method can be applied to estimate the rail corrugation period efficiently. Finally, the wear saliency (WS) based on the visual significance analysis of the corrugation image is calculated to assess the rail corrugation severity. To the best knowledge of the authors, this method is the first one that introduces the concept of wear saliency (WS) to assess the severity of the rail corrugation in terms of the human visual system.

In the future, our research work will mainly focus on the following aspects. On the one hand, the investigation and construction of an intelligent visual inspection system for rail corrugation, including an image acquisition subsystem installed at the bottom of the operating vehicle and a background corrugation identification and analysis system. The methods proposed in this paper will be further validated and the relevant results will be published. On the other hand, other different types of signals (e.g., vibration acceleration and sound) will be carefully investigated and combined in the future researches to assess the severity of the rail corrugation more accurately.