Automatic Detection of Ballast Unevenness Using Deep Neural Network

Abstract

The amount of freight transported by rail and the number of passengers are increasing year by year. Any disruption to the passenger or freight transport stream can generate both financial and human losses. Such a disruption can be caused by the rail infrastructure being in poor condition. For this reason, the state of the infrastructure should be monitored periodically. One of the important elements of railroad infrastructure is the ballast. Its condition has a significant impact on the safety of rail traffic. The unevenness of the ballast surface is one of the indicators of its condition. For this reason, a regulation was introduced by Polish railway lines specifying the maximum threshold of ballast unevenness. This article presents an algorithm that allows for the detection of irregularities in the ballast. These irregularities are determined relative to the surface of the sleepers. The images used by the algorithm were captured by a laser triangulation system placed on a rail inspection vehicle managed by the Polish railway lines. The proposed solution has the following elements of novelty: (a) it presents a simple criterion for evaluating the condition of the ballast based on the measurement of its unevenness in relation to the level of the sleeper; (b) it treats ballast irregularity detection as an instance segmentation process and it compares two segmentation algorithms, Mask R-CNN and YOLACT, in terms of their application to ballast irregularity detection; and (c) it uses segmentation-related metrics—mAP (Mean Average Precision), IoU (Intersection over Union) and Pixel Accuracy—to evaluate the quality of the detection of ballast irregularity.

1. Introduction

Railway ballast condition has a significant impact on the safety of rail traffic, the lifecycle of railway track elements and the state of the rolling stock. According to [1], the ballast transmits the load from the sleeper/ballast interface to the sub-ballast and the subgrade, provides adequate permeability for drainage and keeps the sleepers dry, absorbs noise, vibration and energy, and provides stability to the track by withstanding vertical, longitudinal and lateral forces. Therefore, the good condition of track ballast is key for the safe and smooth running of a train [2]. For this reason, issues relating to ballast diagnostics have become of interest to researchers.

One of the devices used to determine the condition of the ballast is the Ground Penetrating Radar (GPR) [3,4]. It uses polarized high-frequency radio waves, usually in the range of several hundred MHz to several GHz. An extensive reference sampling and laboratory analysis were performed in [5] to aid in developing the GPR-based classification method for qualifying ballast fouling. The classification is made using a fouling index, which is calculated from the frequency contents of the GPR signal. Another article [6] also presents a system based on GPR. Algorithms such as Fast Fourier Transform (FFT) and Artificial Intelligence (AI), i.e., Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN), were applied to monitor ballast fouling. Other articles [7,8] used the Dielectric Constant Method and 2 GHz GPR to evaluate ballast. In [9], the ballast state is assessed based on the GPR signal waveform and wavelet transform. Since the conversion of electromagnetic signal characteristics of GPR to ballast fouling is performed indirectly, the main drawback of methods based on GPR is the susceptibility to electromagnetic interference. It causes problems if the railway runs through areas that contain special structures such as turnouts, bridges with rail guards, etc., as the signal cannot be correctly detected because of iron interference.

Other methods that were used in the assessment of ballast are based on vibration measurements. Paper [10] demonstrates the detection of the damage status of ballast under a sleeper through the monitoring of the vibrations of the corresponding sleeper. The vibration is generated through a simple impact hammer test. Another paper [11] explored the feasibility of using particle acceleration responses to diagnose mud-pumping ballast. It used innovative wireless sensors with 3D-printed shells resembling the real shape of ballast particles. These sensors are placed in the ballast and measure acceleration. According to [11], it seems promising to use particle-scale acceleration underneath tie plates as readily implementable indicators for smart in-service track health monitoring. The main drawback of these methods is that the measurement is made at fixed points in the track.

Another paper [12] attempted to use infrared thermography to detect fouling ballast. It investigated the impact of the intensity of solar radiation and rainfall on the surface temperature of the ballast. The experimental results showed that noticeable temperature differences between clean and fouled ballast occur on sunny days and after rainfall. In contrast, the differences are imperceptible at nighttime.

Another type of device that can be used in ballast diagnostics is LIDAR (Light Detection and Ranging). It consists of a laser illuminating the track and a camera recording the image of the reflection from the ballast (surface map) [13,14]. Article [15] compared the performance of a low-cost solid-state LiDAR system with a traditional static LiDAR system. Both systems were mounted on a mobile inspection trolley and used to measure ballast geometry. The test was carried out on a section of rail track comprising a steel rail, concrete sleepers and ballast as structural support. The experiment showed that a low-cost LiDAR is able to determine ballast geometry but with slightly less accuracy than a traditional static LiDAR. Ref. [16] used a laser scanner together with classification methods to detect elements of railway tunnels based on the point cloud. Paper [17] used close-range photogrammetry to evaluate ballast particle degradation. The authors took into consideration the morphology of the ballast (e.g., shape, angularity and surface texture). Another article [18] used an algorithm for the extraction of vegetation and local muddy areas from ballast images.

The rapid development of both hardware and image-processing algorithms has led to the development of visual methods for railroad track diagnostics. In particular, the appearance of the paper [19] on the application of a deep convolutional network for object detection was a breakthrough in the field of machine learning. Since then, many papers on deep neural networks have been published [20,21,22,23,24,25,26]. The authors used deep learning networks in an algorithm that allows for the detection of irregularities in the ballast. These irregularities are determined relative to the surface of the sleepers. The detection of sleepers is performed using deep learning networks such as Mask R-CNN and YOLACT, which are used for image segmentation. The images used by the algorithm were captured by a laser triangulation system placed on a rail inspection vehicle managed by Polish railway lines. It allows for the continuous inspection of the track both during the day and the night and it provides immunity to electromagnetic interference, which eliminates the aforementioned limitations.

Our contributions are:

• We propose a simple criterion for the evaluation of the condition of the ballast based on the measurement of its unevenness in relation to the level of the sleeper;
• We treat ballast irregularity detection as an instance segmentation process. We examine two instance segmentation algorithms, Mask R-CNN and YOLACT, in terms of their application to ballast irregularity detection;
• The use of instance segmentation makes it possible to determine the unevenness of the ballast individually relative to each sleeper;
• We use segmentation-related metrics—mAP (Mean Average Precision), IoU (Intersection over Union) and Pixel Accuracy—to evaluate the quality of the detection of ballast irregularities.

2. Algorithm for Automatic Detection of Unevenness in Ballast

The poor condition of ballast has a significant impact on both traffic safety and travel comfort. The unevenness of the ballast surface is one measure of the condition of the ballast. The algorithm presented in this paper allows us to determine the unevenness of the ballast. According to Polish railway line standards, when ballast unevenness is within +/− threshold = 35 mm, it does not pose a threat to traffic safety. These levels are measured relative to the top surface of the sleepers. Figure 1 presents a cross-section of the track with marked thresholds.

The state of the railway track is monitored using a track inspection vehicle managed by the Polish railway lines—Figure 2. All available data used in our research were collected using this vehicle. One of the several measurement systems installed on the board of the vehicle is a laser triangulation system. It comprises pulsed high-power laser line projectors and synchronized cameras. This system captures a high-resolution intensity image and 3D range profile of the railway track. Laser light is used to illuminate railway surfaces and high-speed cameras are used to capture images of the projected light including its intensity. Figure 3 shows a block diagram of the laser triangulation system. Such a system captures both Intensity and Range images of the railway track simultaneously. Intensity images are produced by mapping the intensity of the reflected laser light, and Range images (depth map), marked in this figure as Dr, are produced by mapping the elevation of each measurement point. The pixels of these two maps correspond one-to-one, that is, the corresponding image regions in the intensity image appear at the same position in the depth map. If a certain target is detected in the intensity map, the corresponding target can be found at the same position in the depth map (Range image).

Figure 4 shows a block diagram of the ballast unevenness detection algorithm. It is composed of two components. The first component presented in this figure as the “detection of sleepers” box is responsible for the detection of sleepers in the track and is described in Section 2.1. It uses the intensity image coming from the laser triangulation system as an input. Deep neural networks for instance segmentation are used for sleeper detection (both wooden and concrete). The second component presented in this figure as the “detection of ballast out of range +/− Threshold” box specifies the areas of ballast for which its height is outside the defined range of the +/− threshold = 35 mm and is described in Section 2.2. It uses the range image (map) coming from the laser triangulation system as an input and sleeper masks generated by the first component. The width and the height of both the intensity and the range image are equal to 288 and 1024, respectively.

2.1. Detection of Sleepers

We used the intensity image generated by the laser triangulation system as an input. The unevenness of the ballast is measured relative to the surface of each sleeper. On this basis, the algorithm should detect each sleeper as well as its mask. Therefore, it provides the classes of the image objects and the location of the image objects in the form of bounding boxes. Additionally, it should provide labels for every pixel belonging to the objects. Each pixel is labelled according to the object class within which it is enclosed. This process is called instance segmentation and simultaneously solves the problem of object detection as well as semantic segmentation (pixel labelling). According to [27,28], instance segmentation algorithms can be divided into four groups:

-

Classification of mask proposals (modest segmentation accuracy, slow and difficult to optimize training, slow testing, not suitable for real-time applications);

-

Detection followed by segmentation (relatively simple to train, better generalization, relatively faster, good segmentation accuracy);

-

Labelling pixels followed by clustering (relatively simpler techniques, lesser segmentation accuracy, intense computation necessitating high computational power, not suited for real-time applications);

-

Dense sliding window methods (modest segmentation accuracy, use complex algorithms, difficult to train and optimize, not suitable for real-time applications).

Due to its advantages, detection followed by segmentation was selected. Two networks—Mask R-CNN and YOLACT—belonging to this group were tested in the article. Mask R-CNN and YOLACT are a two-stage and a one-stage instance segmentation algorithm, respectively. Mask R-CNN as a two-stage algorithm is more accurate and slower than YOLACT.

2.1.1. Mask R-CNN Network

The Mask R-CNN is the network that performs instance segmentation. Figure 5 shows the structure of the Mask R-CNN network. This network uses a convolutional neural network (CNN) to extract feature maps that are used to describe the objects being detected (sleepers). The CNN is often called the “backbone”, which is usually a pre-trained neural network. At present, three types of convolutional neural networks—AlexNet, VGG-Net and ResNet—can be used as a backbone. AlexNet, as the oldest network type, which produces the largest classification error on the ImageNet database (16.5%) [19], was discarded. VGG-Net reduced the classification error to 7.6% while increasing the number of network layers to 16–19 [29]. Thanks to the adoption of the core idea of Identity Shortcut Connection [30], ResNet further reduces this error to 3.6%. Using classification error on the ImageNet test database as a criterion, the authors decided to test Resnet51 and Resnet101 as a backbone. In order to narrow down the search for an object in an image, the network can be divided into two parts. In the first part, the region of interest (ROI) is determined using the Region Proposal Network (RPN)—marked by red rectangles on the feature map. To generate the ROI, the input of an n × n spatial window is slid over the convolutional feature map output, and the features covered by the spatial window are fed into two sibling fully connected layers—a box classification layer and a box regression layer. A box classification layer determines the probability of occurrence of the objects in the ROI area. On the other hand, a box regression layer generates the coordinates of the ROI center and its width and height.

The size of the ROI area generated by the RPN is variable. For this reason, in the next step (part) of the algorithm, the variable ROI size is converted into a fixed size. Here, fully connected layers also occur—box classification layers and box regression layers—which generate the class of the detected object and the coordinates of the bounding box, respectively. Additionally, the ROI is also fed to a fully convolutional network (FCN) [31] to generate the mask of an object being detected. According to [32], the algorithm uses the following objective function during training:

$$L=L_{CLS1}+L_{CLS2}+L_{box1 }+L_{box2 }+L_{mask}$$

(1)

where L_CLS1 is a binary classifier loss corresponding to the classes block in RPN; L_box1 is a regression loss corresponding to the boxes block in RPN; L_CLS2 is a loss for the classifier corresponding to the classes block in the box head; L_box2 is a regression loss corresponding to the boxes block in the box head; and L_mask is a mask binary cross-entropy loss corresponding to the mask block in the mask head.

2.1.2. YOLACT Network

The YOLACT network is a one-stage algorithm used for instance segmentation. The algorithm can be divided into two parallel subtasks, namely a prototype network block and a target detection block. Figure 6 shows a block diagram for the YOLACT algorithm. Both blocks use a convolutional neural network along with a Feature Pyramid Network (FPN) [33]. The FPN adds lateral connections between the P5 and P3 layers of the pyramid. The idea is to take top-down strong features from the C5 layer of the CNN and propagate them to the high-resolution feature maps in the C3 layer of the CNN, thus creating strong features across all levels. The prototype network block generates a set of image-sized “prototype masks”. These masks are independent of the instance. It predicts a set of k prototype masks for the entire image using the FCN. The P3 layer from the Feature Pyramid Network (FPN) is fed to an FCN.

The target detection block consists of three parallel branches. The first branch predicts c class confidences and the second one predicts the four coordinates of the bounding box. The third branch predicts k mask coefficients, one corresponding to each prototype. After generating k mask coefficients and passing them through the Non-Maximum Suppression algorithm (NMS), they are combined with prototype masks using the following formula [34]:

$$M=\sigma \left(P{C}^T\right)$$

(2)

where P is an h × w × k matrix of prototype masks and C is an n × k matrix of mask coefficients for n instances surviving NMS. The final mask M is cropped with the predicted bounding box generated by the target detection block.

YOLACT uses the following loss function while training [34]:

$$L=L_{CLS}+L_{box}+L_{mask}$$

(3)

The classification loss L_CLS and the box regression loss L_box are defined in the same way as in [35] and the mask loss L_mask is the pixel-wise binary cross entropy between the assembled masks and the ground truth masks.

2.2. Determination of the Area of Unevenness of the Ballast

This part of the algorithm is designed to determine areas of ballast unevenness in reference to sleeper levels. The laser triangulation system can simultaneously generate an intensity image (map) and a range image (depth). The pixels of these two images correspond one-to-one. Therefore, the coordinates for the sleepers generated by the deep neural network based on the intensity image can be directly mapped to the range image. The algorithm is presented in the form of pseudocode in Algorithm 1.

Algorithm 1. Algorithm to determine areas of ballast unevenness in reference to sleeper levels

1: for each sleeper detected by instance segmentation network in the intensity image do 2: Using range image calculate the mean level of the sleeper = ML 3: calculate Upper_Threshold := ML + Threshold 4: calculate Bottom_Threshold := ML − Threshold 5: Mask := ø 6: for each m from 1 to image_height do 7:    for each n from 1 to image_width do 8:   if (D(m, n) < Bottom_Threshold) or (D(m, n) > Upper_Threshold) then 9:    Mask := Mask + (m, n) 10:   end if 11:  end for 12: end for 13: Mark points in the range image based on Mask 14: end for

Where D(m, n) denotes the depth corresponding to the coordinates (m, n) in the range image; threshold = 35 is the threshold value determined towards the sleeper level; and Image_height and image_width are the height and width of the range image, respectively.

According to this pseudocode, the unevenness of the area is calculated separately for each sleeper detected by the instance segmentation algorithm. These areas are defined relative to the mean level of the sleeper surface.

3. Experimental Results

To train both the Mask R-CNN and the YOLACT networks, we collected 1600 intensity images of the track. All images came from a track inspection vehicle managed by Polish railway lines. The intensity images are captured by a fast camera installed under the floor of the inspection vehicle. This camera is equipped with a Gigabit Ethernet (1 GigE) interface. The captured images are sent via 1 GigE to a server installed on the board of this vehicle. The group of selected images with wooden and concrete sleepers included images recorded in good weather (900) as well as during rainfall (700). All images were pre-processed. In this process, each image pixel was normalized so that the average pixel intensity value in the image was zero. This pre-processing significantly improves the convergence of the gradient optimization used to train both the Mask R-CNN and the YOLACT networks. All the available images were randomly divided into three groups in the proportions 60%, 20% and 20%: training data hereinafter referred to as train_data, validation data hereinafter referred to as valid_data and testing data hereinafter referred to as test_data.

We did not train both networks from scratch. To improve the performance of both networks and speed up their training we decided to use transfer learning [36,37]. This means that a model developed for one task is reused as the starting point for a model on a second task.

3.1. Training and Testing Mask R-CNN Network

The Mask R-CNN network was implemented using the Keras package in Python. Using the idea of transfer learning, two pre-trained networks were tested. The first uses Resnet51 as the backbone, while the second uses Resnet101. Each of the networks was trained for 400 epochs on a computer equipped with an Intel I5 CPU, NVIDIA GeForce GTX 1080 Ti GPU and 48 GB RAM. We used AdamOptimizer with a learning rate Lr = 0.0001, weight decay w_dec = 0.0005 and momentum = 0.9 to minimize the loss function (loss error) defined by Formula (1). While training, two groups of data were used: train_data for training and valid_data for validation. Figure 7 shows the loss error for the Mask R-CNN with the ResNet101 backbone during the learning process for the train_data set (Figure 7a) and the valid_data set (Figure 7b).

After training, the performance of each network was verified on a test_data set. The network performs both the detection of individual sleepers and their segmentation. For this reason, the Average Precision (AP) metric was used to assess the quality of sleeper detection, while Intersection over Union (IoU), Dice coefficient and Pixel Accuracy metrics were used to assess the quality of the segmentation.

Pixel Accuracy calculates the percentage of pixels in the image that were correctly classified. It is commonly reported for each class separately. Pixel Accuracy can be defined as:

$$Pixel accuracy=\frac{TP+TN}{TP+TN+FP+FN}$$

(4)

where TP true positive represents pixels that are correctly classified to belong to the sleeper class (according to the target (truth) mask); TN true negative represents pixels that are correctly classified as not belonging to the sleeper class; FP false positive represents pixels that are classified as belonging to the sleeper class though they do not belong to sleeper class; and FN false negative denotes pixels that are classified as not belonging to the sleeper class though they belong to the sleeper class.

Intersection over Union (IoU) quantifies the percentage overlap between the target (truth) mask and our prediction output (predicted mask). The IoU is defined as:

$$IoU=\frac{T\_A\cap P\_B}{T\_A\cup P\_B}=\frac{TP}{TP+FP+FN}$$

(5)

where T_A denotes pixels belonging to the target mask and P_B represents pixels that are classified as belonging to the target mask.

The dice coefficient can by defined as:

$$Dice=\frac{2\ast (T\_A\cap P\_B)}{T\_A+P\_B}=\frac{2\ast TP}{TP+TN+FP+FN}$$

(6)

AP is calculated based on the PASCAL VOC 2017 challenge. AP is used to evaluate sleeper detection. For this reason, in the IoU metric in (5), a target bounding box is used instead of a target mask. Additionally, true positives (TP1), false positives (FP1) and false negatives (FN1) were redefined. TP1 is calculated when the IoU for the detected object is higher than the assumed threshold, whereas FP1 is calculated when the IoU is lower than the assumed threshold. FN1 is determined when the model did not predict a bounding box at a certain position although the target bounding box exists at that position.

The precision–recall curve is used while defining the AP metric. Precision is the ratio between the correct detections and the total number of detections:

$$\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\frac{\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t} \mathrm{d}\mathrm{e}\mathrm{t}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s}}{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l} \mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r} \mathrm{o}\mathrm{f} \mathrm{d}\mathrm{e}\mathrm{t}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s}}=\frac{TP1}{TP1+FN1}$$

(7)

Recall is the ratio between the correct detections and all available objects (target bounding boxes):

$$\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\frac{\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t} \mathrm{d}\mathrm{e}\mathrm{t}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s}}{\mathrm{a}\mathrm{l}\mathrm{l} \mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t} \mathrm{b}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g} \mathrm{b}\mathrm{o}\mathrm{x}\mathrm{e}\mathrm{s}}$$

(8)

A precision–recall curve plots the value of precision against the recall for different confidence threshold values. The AP metric is defined as:

$$AP={\int }_{r=0}^{r=1}p\left(r\right)dr$$

(9)

where p and r denote precision and recall, respectively. The precision–recall curve is usually approximated with rectangles and the integral in (9) is calculated as a sum of rectangles [38].

Table 1. Detection and segmentation metrics for Mask R-CNN with two different backbones.

	Mean Average Precision	IoU	Dice Coefficient	Pixel ACCURACY	Mean Processing Time [ms]
Resnet51	0.8571	0.7602	0.8046	0.7845	125
Resnet101	0.8901	0.78004	0.8304	0.8063	200

shows the AP, IoU, Dice coefficient and Pixel Accuracy calculated for the networks with Resnet51 and Resnet101 backbones. Additionally, the mean processing time is also shown. An IoU threshold of 0.5 was used for all AP metric calculations.

As can be seen from Table 1, the highest values of the metrics were obtained for the Mask R-CNN network with Resnet101, while the shortest processing time was obtained for the network with Resnet51. This is due to the difference in complexity of the Resnet101 and the Resnet51 network structures. The complexity of the Mask R-CNN network with Resnet101 is higher, which results in its higher accuracy (higher values of metrics) at the expense of increased processing time.

Figure 8 shows examples of detection and segmentation of the sleepers from the intensity images of the track obtained for Mask R-CNN with the Resnet101 backbone. The first column shows the image of the track, the second the image with the predicted bounding box and mask overlaid on it, the third the predicted bounding box and mask and the last the target (truth) bounding box and mask. If we compare the predicted bounding boxes (dashed lines) and predicted masks (green areas) in column three with the corresponding truth bounding boxes (dashed lines) and truth masks (green areas) in column four, we can see negligible differences between them. The exception is the last lower sleeper found in the image from row three. Due to its very small size, it was not detected by the Mask R-CNN network.

3.2. Training and Testing the YOLACT Network

The YOLACT network was implemented using the Pytorch package in Python. Using the idea of transfer learning, two pre-trained networks were tested. The first uses Resnet51 as the backbone, while the second uses Resnet101. Each of the networks was trained for 400 epochs on a computer equipped with an Intel I5 CPU, NVIDIA GeForce GTX 1080 Ti GPU and 48 GB RAM. We used AdamOptimizer with a learning rate Lr = 0.0001, weight decay w_dec = 0.0005 and momentum = 0.9 to minimize the loss function (loss error) defined by Formula (3). While training, two groups of data were used: train_data for training and valid_data for validation.

After training, the performance of each network was verified on a test_data set. The performance of the network was checked based on the same metrics as the Mask R-CNN network.

Table 2. Detection and segmentation metrics for YOLACT with two different backbones.

	Mean Average Precision	IoU	Dice Coefficient	Pixel ACCURACY	Mean Processing Time [ms]
Resnet51	0.8214	0.7226	0.7668	0.7737	44
Resnet101	0.8571	0.7380	0.7912	0.7956	61

shows the obtained results. As for Mask R-CNN, the highest accuracy was obtained with the Resnet101 backbone, whereas the lowest processing time was obtained using Resnet51.

Figure 9 shows examples of the detection and segmentation of sleepers from the intensity images of the track obtained using YOLACT with a Resnet101 backbone. The first column shows the image of the track, the second the image with the predicted bounding box and mask overlaid on it, the third the predicted bounding box and mask and the last, the target (truth) bounding box and mask. In order to compare the results of the Mask R-CNN network with those of the YOLACT network, Figure 9 contains the same test intensity images (column one) as Figure 8. The differences between the predicted bounding boxes (solid lines) and the predicted masks (colored areas) in column three and the corresponding truth bounding boxes (solid lines) and truth masks (colored areas) in column four are slightly larger than for the Mask R-CNN network. In addition, the YOLACT network did not detect the last lower sleeper found in the image from row two. It should be noted that this sleeper was detected by the Mask R-CNN network (Figure 8, row two). The YOLACT network is a one-stage network. It is less accurate than the two-stage Mask R-CNN network. This is particularly evident when detecting objects of a small size.

In order to confirm the difference in the performance of the two models (Mask R-CNN with Resnet101 versus YOLACT with Resnet101), a statistical hypothesis test was performed. Because instance segmentation (Mask R-CNN and YOLACT) is composed of an object classification part (binary classification—sleepers and no sleepers) and a pixel classification part (binary classification—pixel belongs to the sleeper or pixel belongs to background), the test was conducted on both parts. The statistical test was selected based on the following criteria:

-

Models are deep neural networks that are very large and their training is time-consuming. We prefer a statistical hypothesis test that enables us to compare the models on a single test data set instead of performing multiple training (k-fold cross-validation with a modified paired Student’s t-test);

-

Models are binary classifiers;

-

There are no assumptions about the type of data distribution;

-

The test data for models are the same.

According to [39], McNemar’s test satisfies all the aforementioned criteria. This test uses the contingency table to calculate the McNemar’s test statistic [39]:

$$statistic=\frac{{\left(a-b\right)}^2}{a+b}$$

(10)

Let model_1 denote the first model and model_2 denote the second model. Then, a denotes the number of samples correctly classified by model_1 and simultaneously misclassified by model_2; b denotes the number of samples misclassified by model_1 and simultaneously correctly classified by model_2; a and b are obtained from the contingency table.

If a and b are similar, both models make errors in much the same proportion, just on different instances of the test set. In this case, the result of the test would not be significant and the null hypothesis would not be rejected. On the other hand, if a and b are not similar, not only do both models make different errors, but in fact have a different relative proportion of errors on the test set. In this case, the result of the test would be significant and we would reject the null hypothesis. Therefore, the rejection of the null hypothesis means that the two models have different performances when trained on the particular training. The null hypothesis is rejected when p value ≤ alpha and is not rejected when p value > alpha. Alpha denotes a significance level and p value is probability and measures how likely it is that any observed difference between the models is due to chance. A p value close to 0 indicates that the observed difference is unlikely to be due to chance.

McNemar’s test was performed separately for the object classification part and the pixel classification part of the Mask R-CNN and YOLACT models. The test was conducted on 320 images (test_data set) that contained 1287 sleepers and 10,194,844, pixels belonging to sleepers. During testing, the significance level alpha was set to 0.05. In the test of the null hypothesis, “the object classification part of Mask R-CNN with Resnet101 and the object classification part of YOLACT with Resnet101 are the same model”, the McNemar’s test generated p value = 0.00464. Therefore, this hypothesis can be rejected and it means that these two models are different for the object classification parts. In the testing of the pixel classification part, the null hypothesis was formulated as “the pixel classification part of Mask R-CNN with Resnet101 and the pixel classification part of YOLACT with Resnet101 are the same model”. The McNemar’s test generated a p value = 0.00138. Therefore, this hypothesis can be rejected as well and it means that these two models are different for the pixel classification parts.

Based on Table 1 and Table 2 and McNemar’s tests, the authors decided to choose the Mask R-CNN network with Resnet101 for the detection and segmentation of the sleepers. We emphasized the accuracy of the detection and segmentation of the sleepers at the expense of an increased processing time.

3.3. Detection of the Unevenness of the Ballast

After determining the position of the sleepers, the algorithm in Algorithm 1 detects areas of ballast irregularity whose levels exceed the assumed threshold = 35 mm relative to the sleeper level. Because the assessment of unevenness of the ballast boils down to an instance segmentation process, we checked the quality of the proposed algorithm using AP, IoU and Pixel Accuracy metrics for images from the test_data set. These three metrics were obtained for the areas where the height is out of the range of +/− 35 mm.

Table 3. Detection and segmentation metrics for ballast irregularity detection algorithm.

Average Precision AP	IoU	Pixel Accuracy
0.91	0.93	0.91

shows these metrics.

Figure 10 shows the results of the proposed algorithm for the automatic detection of ballast irregularities. Figure 10a,c,e,g show the range images (depth map) of the track section and Figure 10b,d,f,h show the image with the detected sleepers and area of the ballast with the height out of range +/− threshold overlaid on it (the area marked in light green).

4. Conclusions

The algorithm presented in this article, based on deep neural networks, allows for the detection of ballast unevenness in compliance with a regulation introduced by the Polish railway lines. The unevenness of the ballast surface is one of the indicators of its condition. For this reason, a regulation was introduced by the Polish Railways specifying the maximum threshold of ballast unevenness. The algorithm presented uses track images which are scanned by a LASER triangulation system installed on a track inspection vehicle managed by the Polish railway line authorities. One of the key elements of the algorithm is the deep neural network used to detect and segment the sleepers from the intensity image. Two networks were tested: Mask R-CNN and YOLACT with two types of backbones: Resnet101 and Resnet51. Due to its higher accuracy (AP = 0.8901, IoU = 0.78004 and Pixel Accuracy = 0.8063), the Mask R-CNN network with Resnet101 was selected. Because the determination of unevenness of the ballast based on the track image boils down to an instance segmentation process (areas of unevenness of the ballast undergo detection and segmentation), the quality of the detection of ballast irregularity was verified with the help of the AP = 0.901 measure, while the quality of the segmentation of the ballast was checked using the IoU = 0.93 and Pixel Accuracy = 0.91 measures. The algorithm presented in the article, using Mask R-CNN, makes it possible to effectively assess the level of unevenness of the ballast, which is correlated with the state of the ballast. The proposed system allowed for the effective implementation of the regulations introduced by the Polish Railways regarding ballast evenness. It can be an alternative to other existing systems.