Design of an Infrared Image Processing Pipeline for Robotic Inspection of Conveyor Systems in Opencast Mining Sites

Abstract

Conveying systems play an essential role in the continuous horizontal transportation of raw materials in mining sites. Regular inspections of conveyor system structures and their components, especially idlers, are essential for proper maintenance. Traditional inspection methods are labor-intensive and hazardous; therefore, robot-based thermography can be considered a quality assessment tool for the precise detection and localization of overheated idlers in opencast mining sites. This paper proposes an infrared image processing pipeline for the automatic detection and analysis of overheated idlers. The proposed image processing pipeline can be used for the identification of significant temperature anomalies such as hotspots and hot areas in infrared images. For the identification of such defects in idlers, firstly, the histogram of captured infrared images was analyzed and improved through the pre-processing stages. Afterward, the location of thermal anomalies in infrared images was extracted. Finally, for the validation of segmentation results, the shapes and locations of segmented hot spots were compared with RGB images that were synchronized by captured infrared images. A quantitative evaluation of the proposed method for the condition monitoring of belt conveyor idlers in an open-cast mining site shows the applicability of our approach.

1. Introduction

Conveyors have been developed and used as the most common system for conveying all forms of material in the mining industry. For decades, conveyors have been used for transporting raw materials due to their efficiency and relatively straightforward design. Despite the conveyor advantages, there are still significant challenges for conducting regular inspections to guarantee their operation under harsh environmental conditions in mines [1,2,3,4,5,6,7].

Idlers are important parts of the conveyors that support the belt to carry the material along its full length [8,9]. Idlers can be damaged by friction, tear, wear, jamming, or seizure. Faulty idlers can become overheated and cause belt damage; thus, the temperature, noise emissions, and vibrations of idlers should be constantly monitored through regular inspections. Idlers are located along the conveyor, and the typical length of conveyors in mining tunnels could reach a kilometer [10]. Human inspections of idlers by walking along the belt is time-consuming, costly, and hazardous, as even a small conveyor of 150 m consists of nearly 450 carrying rollers and 50 return rollers that should be inspected individually [11].

Monitoring the surface temperature of idlers is a key to finding faulty idlers because the abnormal temperature rise is an important characterization of idler failures on conveyor systems. The detection of overheated idlers focuses on identifying areas in IR images with a significantly higher temperature than other elements in a image. However, the automatic identification of overheated idlers in IR images is difficult due to the presence of sunlight reflection or non-informative objects (from the hot idler detection perspective) [12,13].

To summarize, according to the current status of condition monitoring (CM) methods for conveyor systems, there is a challenging need to minimize the presence of humans by the automatization of the inspection processes. Direct monitoring methods, such as IR thermography, are capable of detecting and diagnosing defects in idler modules. In this paper, IR image processing techniques alongside shape detection algorithms are experimentally assessed for their applicability for CM of idlers. The proposed techniques are applied to IR images of idler modules captured by a mobile robot during several field IR thermographic measurements. The direct relation between idlers surface temperature and their health status in terms of efficiency was the main topic for investigation.

The paper is organized as follows. First, the problem becomes increasingly explored (as predictive maintenance and inspection robotics are discussed by many authors); thus, a comprehensive literature review is provided. It has been divided into several paragraphs, as a few perspectives need to be mentioned. Then, an original procedure for overheated idler detection is proposed. Next, we describe the experimental trials and data acquired by the inspection robot in the real environment, and finally, the results of the proposed methodology applied to real data are presented and discussed.

2. Literature Review

2.1. Application of IR Thermography for Diagnosing Industrial Infrastructures

Critical infrastructures are almost always equipped with many sensors and supervisory systems. However, in some situations, as is considered in this paper, CM systems can be applied on a limited scale only. Drive units (engine, gearbox, etc.) are usually monitored by supervisory control and data acquisition (SCADA) [14] but the rest of the conveyors (for example, a typical conveyor is 1 km length), namely the moving belt, hundreds of rotating idlers, etc. are difficult to monitor by stationary installations and need to be inspected by maintenance staff [15]. Unfortunately, the mining environment is very harsh for that reason; therefore, there is a general tendency to minimize the presence of humans and atomization of inspection processes by intelligent robots.

IR thermography is categorized as a non-destructive CM technique that can be used for analyzing the temperature patterns in objects based on utilizing IR radiations that are emitted from an object surface [16,17]. The simple analysis of IR images can give us information about the surface temperature of machines, while by further analysis, we can find possible thermal emission abnormalities. Together with extracted features from IR images, the degree of deterioration can be evaluated by analyzing the thermodynamics and physical characteristics of inspected machines [18].

Several types of faults and conditions in rotating machinery such as coupling looseness, rotor imbalance, misalignment, rolling element bearing damage, and lubricant inadequacy can be detected in IR images [2,19,20,21].

It is worth mentioning that while IR imaging applications in the identification of overheated modules in industrial infrastructures have already been discussed in controlled laboratory environments, robot-based IR imaging methods for the identification of abnormal temperature in real case experiments so far have been rarely discussed and their results rarely presented.

In [22], researchers developed a method for the CM of rotating machinery using IR image processing techniques. The authors discussed the advantages of IR imaging-based machine health monitoring over vibration-based methods. However, in their proposed method, they only used IR images as a reference for the identification of faulty modules; furthermore, they used a stationary IR camera system for conducting their research in a controlled environment. Similarly, in [23], the authors investigated a fault detection method using thermography techniques for identifying air leakages in the pipeline in a laboratory environment.

In [4,12,15], different methods for CM of conveyor systems in mines are discussed. In [4], researchers propose a method for analyzing the thermal state of a belt conveyor in an underground mine. However, the thermal images were captured by a human inspector on a limited scale. Inspired by the same problem, ref. [15] proposed a fault analysis method for the identification of faulty idlers in conveyor systems. However, they conducted their experiments in a controlled environment, and thermal images present a few idlers in high resolution; furthermore, the authors did not propose a solutions for images with complex backgrounds that should be considered in real case scenarios.

Particularly, Dabek et al. [12] suggest an automatic robot-based IR imaging method for the identification of overheated idlers in an open-cast mining site. The authors propose an efficient diagnostic procedure for the detection of overheated idlers, such as defining regions of interest (ROIs) on captured images to reduce the redundant data as well as the color-based segmentation method. For improving the proposed method by Dabek et al., firstly, we discuss a new ROI estimation technique for removing the non-ROIs areas. Furthermore, we propose a histogram-based technique for improving the overall quality of captured extracted IR frames. Therefore, we could accurately track the location of idlers in capturing IR and RGB frames and improve the overall accuracy of segmentation results.

Automated Diagnostic Methods

During the measurements, the inspection robot was able to capture the sequence of IR images without information about the true temperature of the conveyor elements. Due to the automatic scaling of colors to the temperature range, “hard” thresholding based on the predefined value of temperature was not possible.

The hotspot areas in grayscaled IR images can be extracted using an automatic thresholding method where the maximum gray pixel value determines the maximum temperature in the defined region of interest. The problem with general IR image processing pipelines is they do not provide accurate results in identifying the objects in a complex background, as IR images tend to be over-segmented. Some examples of IR image segmentation results using automatic thresholding methods are shown in Figure 1. One can notice that most of the segmented images tend to be over-segmented in comparison to the ground truth image, which leads to some parts of the equipment or components to merged with the background image.

In this paper, our detection strategy was focused on firstly proposing techniques for improving the general characteristics of extracted IR frames histograms and in the next step proposing an outlier-based automatic segmentation method together with shape detection algorithms for the identification of overheated idlers.

The overall quality of IR images can be improved based on histogram enhancement techniques. In [30], the authors propose techniques for the enhancement of the subtle thermal signatures. They suggest noise smoothing by means of, e.g., median or Gaussian filtering, as the most common preprocessing procedures. The application of both median and Gaussian filters to IR images with poor or insufficient information about the health of PV modules is discussed by [31]. The combination of Contrast Limited Adaptive Histogram Equalization (CLAHE) with Gamma correction can be considered an effective way for improving the overall quality of digital images [32,33]. In our work, an automatic gamma correction method together with CLAHE and median filter method has been used for improving the contrast and reducing the noises in captured IR images.

Due to the nature of IR images which are quite different in comparison to visual light images, extracting the hot regions within an IR image is a very challenging task [34]. The distribution of pixel intensities in IR images is based on the heat distribution of an object. Low-intensity contrast and over-centralized intensity distributions in IR images are important factors that bring some difficulties to automatic segmentation methods.

For addressing the mentioned issues in this paper, we exploit the concept of outliers. If in a given IR image, any hot element will appear in the distribution of pixels, the right tail of image histograms related to “hot” colors will be heavier than for “normal temperature” elements. Our target is to detect really hot elements in the conveyor, i.e., significantly higher temperatures than other elements in the picture. In this context, refs. [35,36,37,38,39] proposed IR image histogram analysis techniques for the identification of thermal anomalies in industrial infrastructures.

3. Automatic Procedure for Detection of Overheated Idlers in IR Images

3.1. General Concept

The aim of this section is to describe the key elements of the methodology. A summarized flowchart of the proposed procedure is presented in Figure 2.

The proposed method started by loading the captured data by the inspection robot. The captured data during the experiment were download in a local computer and were further processed offline. Afterward, the total number of captured frames was extracted from loaded IR and RGB videos. The extracted IR frames were converted into 8-bit grayscaled images for further analysis. The camera system captured wide-angle videos from the mining site; therefore, the extracted videos contained many non-informative objects that were not related to the conveyor system. For reducing the number of non-informative objects in the captured frames, ROIs were defined on IR and RGB image data sets.

During the measurements, several data acquisition sessions were performed. It is worthwhile to notice that the environmental conditions are time-varying (even if it is in a kind of indoor condition). A critical issue is that during the experiment, the true temperature of the conveyor element automatically adjusts the colors to a given temperature range producing many complicated images. By additionally assuming the linear relation between the intensity of the brightest pixel (hottest area) to the darkest pixel (coldest area), we could apply statistical methods to segment overheated idlers from the background in grayscaled IR images.

To analyze and assess the quality of idlers in conveyor systems with respect to thermal defects, we first detect the overheated idler modules that have significantly higher temperatures in comparison to other modules. The overheated idlers appeared in the ROI as areas with an average temperature higher than their surroundings. We follow a statistical, data-driven approach that consists of the following steps: (1) normalization (2) correction and refinement, (3) thresholding, (4) canny edge detection implementation, and finally (5) blob detection algorithm for detecting the overheated idlers. The proposed pipeline was developed in the Python language using the OpenCV library for computer vision algorithms.

3.2. IR Image Histogram Analysis

A histogram of IR images acts as a graphical representation of the color or intensity distribution of pixels. For a gray-level image, the intensity value of pixels refers to discrete temperature values. The statistical analysis of an IR image histogram is a practical way of indicating anomalies in temperature patterns in captured scenes. The mean value, variance, and standard deviation are the statistical-based features that describe the intensity distribution of pixels in IR images. For a grayscaled IR image, the first-order histogram probability P(g) is defined as follows [40]:

$$P\left(g\right)=\frac{L\left(g\right)}{M}$$

(1)

where M is the total number of pixels and $L\left(g\right)$ describes the number of gray levels g. In gray-level images, the total number of the intensity level of pixel L spans into [0, 256]. As the tonal distribution represents the thermal distribution in captured scenes, gray-level infrared images can be processed based on tonal intensities. The general brightness of images is defined by the mean value:

$$\overline{g}=\sum _{g=0}^{L-1}g·P\left(g\right)$$

(2)

Furthermore, the dispersion of a set of data points around their mean value is defined by variance and is given by the following equation:

$${\sigma }_g^2=\sum _{g=0}^{L-1}{(g-\overline{g})}^2·P\left(g\right)$$

(3)

The standard deviation or the square root of the variance has described the spread in IR image data and can give us information about the contrast of IR images. As temperature distribution is a key index of possible defects, the standard deviation can be considered as an important factor for identifying overheated idlers.

3.3. Adaptive Region of Interest Estimation

The original captured videos contain both information on the target and redundant areas; therefore, it would be an advantage if non-ROI regions can be removed before precise (final) analysis. This reduction has to be performed in a way such that no idlers data are lost while the computational burden is reduced. The ROI analysis is defined by a set of techniques that can be used for selecting areas of an image from which the individual or average pixel values are extracted for further analysis. The ROI can be defined manually or by automated methods. The first one is faster but less precise, whereas the second method is more time-consuming but in general more accurate.

The purpose of ROI generation is to automatically extract regions of interest (ROI) from the extracted IR frames that only contain pixels that are related to idlers. During the examination, for capturing RGB and IR videos, the inspection mobile robot was navigated through the free spaces between belt conveyors. The camera system’s point of view (POV) was fixed and pointed toward the belt conveyor. As long as the mobile robot followed a straight line alongside the conveyor belt, the changes in the camera system POV were neglectable.

There are many different methods for tracking objects in the sequence of the frame. Since the camera system POV changes during the inspection were neglectable, we could accurately estimate the approximate location of the idler in the sequence of extracted frames, as shown in Figure 3. By the consideration of small changes in the camera system POV, a rectangular region of the predetermined size, 400 × 600 pixels surrounding the estimated location of idlers with fixed positions, was defined on extracted IR and RGB frames. The size of the defined ROI was large enough to only capture the idler, while it was considerably smaller than the original captured frames; therefore, we could effectively reduce the computational burden.

3.4. The Key Frame Extraction Method

Through the conducted experiment, the inspection mobile robot captured continuous thermal videos from different conveyor systems. Using every individual frame for identification of the overheated idlers is unnecessary, as many frames are almost repeated in a certain time interval. Therefore, in our research, we chose a key frame extraction method to summarize and reduce the size of the extracted IR frames.

Mean Square Error (MSE) is one of the two error metrics which can be used for calculating the cumulative squared error between the reference image and the target image. In this paper, the global similarity between each reference and target frame in relation to their pixel intensities has been measured by the MSE method, where frame $f_i$ and $f_{i-1}$ are the target and reference frames, and m and n are coordination of each pixel in compared frames [41].

$$\mathrm{MSE}=\frac{1}{MN}\sum _{n=0}^M\sum _{m=1}^N{[f_{i-1}(n,m)-f_i(n,m)]}^2$$

(4)

For calculating the degree of similarity between each of the two IR frames, firstly, the reference and target $f_i$ and $f_{i-1}$ are loaded. Afterward, histograms of both frames are taken, and the mean square difference between two histograms is calculated. Through the examination, the parameter MSE ≥ 5 turned out to be sufficient and considered as a threshold for the selection of the keyframes; therefore, when the difference between compared frames is greater than the threshold, the target frame is declared as the next keyframe. This process is repeated until there are no frames left for processing.

3.5. Normalization

The constant changes in color scheme adjustments by the camera can only affect the calibration factor regarding absolute temperature values but not affect the distribution pattern of the temperature. As long as pixel intensity was predefined by the camera with respect to the hottest and coldest point in original frames, we need to normalize the intensity value of a pixel with respect to the brightest and darkest pixel in defined ROIs.

The intensity value of infrared images was normalized to a constant range with respect to pixel intensity distribution in defined ROI. Normalizing the temperature pattern allows us to define a set of parameters that works well for analyzing ROIs with varying temperature ranges. The normalization of ROIs with different seasons can rescale the radiant temperature to the same level between the hottest and lowest in defined ROIs and thus reduce the seasonal difference. Accordingly, the extracted ROIs were normalized using the following equation:

$$N_i=\frac{T{S}_i-T{S}_{min}}{T{S}_{max}-T{S}_{min}}$$

(5)

In Equation (5), $N_i$ is the normalized value of pixel i, where ${\mathit{TS}}_i$ is the intensity value of pixel i. Furthermore, ${\mathit{TS}}_{max}$ and ${\mathit{TS}}_{min}$ can be defined as the maximum and minimum values of pixel intensity in a ROI.

3.6. Correction and Refinement

The automatic thresholding method cannot correctly exclude background regions from the foreground due to their high variance. Therefore, some data normalization steps are usually required before thresholding. To avoid the loss of subtleties in the extracted ROIs, sets of pre-processing techniques were used for improving the general characteristics of frames. The correction and refinement incorporate CLAHE, and Gamma correction was followed by median filtering.

3.6.1. CLAHE Method

Histograms in normalized ROIs would be skewed toward the lower end of the grayscale, and all the image detail can be compressed into the dark end of the IR image histogram. For addressing this issue, histogram-based methods can be used to improve image quality and adjust the contrast.

Histogram equalization (HE) is a simple method for enhancing the contrast of the image by spreading out the intensity range of the image or stretching out the most frequent intensity value of the image. Stretching the intensity values changes the natural brightness of the input image and introduces some undesirable noises [42]. To improve the HE method, Adaptive Histogram Equalization (AHE) [43] was proposed. In the AHE method, the input image is split into smaller images, which are called tiles. The noise, however, often increases when the histogram slope is steep.

CLAHE is an effective contrast enhancement method that effectively enhances the contrast of the image. CLAHE is an improved version of the AHE method that works precisely in the same way, but it clips the histogram at specific values for limiting the amplification before computing the cumulative distributive function (CDF). This change reduces the noise because clipping prevents a CDF from being steep. The computation of CLAHE is performed as:

$$p=\left(p_{max}-p_{min}\right)\ast P\left(f\right)+p_{min}$$

(6)

where p represents the pixel value after applying CLAHE, $p_{max}$ and $p_{min}$ represent the maximum and minimum pixel value of an image, respectively, and $P\left(f\right)$ represents the cumulative probability distribution function [44].

3.6.2. Gamma Correction

Gamma correction can be used to control the overall brightness of images. It is responsible for performing a nonlinear calculation of the pixels intensity of the input image and thereby adjusting the saturation of the image. It is necessary to determine the optimal gamma value; therefore, it should neither be too minimum nor maximum. For enhancing the contrast of the ROIs, an adaptive gamma correction (AGC) technique was applied to the image data set. For having uniform distribution in an image histogram, the optimal value for gamma factor by consideration of $\overline{I}=0.5(L-1)$ as mean intensity can be defined as follows [45]:

$$\gamma =\frac{log\left(\frac{\overline{I}}{(L-1)}\right)}{log\left(0.5\right)}$$

(7)

3.6.3. Median Filtering

In extracted frames where sun reflections were captured on the belt, some unwanted variations were observed within the ROIs. These variations appear in an area with the presence of sun reflection on the belt surface in the form of bright spots that can be wrongly segmented as hot areas. Obviously, it is not possible to exclude them through the ROI estimation process, as they appear at the center of the camera POV. A possible solution to this obstacle is to apply median filtering.

The normalized ROIs after double enhancement undergo median filtering to prepare the images before the thresholding process. The median filter is a nonlinear digital filtering method and is employed to eliminate salt-and-pepper noises in pre-processed ROIs. The median filter can reduce the noise without diminishing the sharpness of the image.

Figure 4 shows the original ROI (on the left) with the unwanted variations and the same ROI after being modified through the pre-processing stages (on the right).

3.7. Thresholding

Overheating in idlers can be recognized as a hotspot in certain areas of ROIs. In IR images with uniform backgrounds, the number of pixels belonging to background or cold areas is much larger than the number of pixels belonging to foreground or overheated objects. We know that an overheated idler’s surface always looks relatively brighter than the background in captured IR images. For distinguishing pixels that are related to the background (cold pixels) and pixels that are related to the foreground (overheated idlers), we classify them based on their distance to the mean value. Our automatic segmentation method worked a base on the outlier detection technique. The main advantage of the proposed method is that we can accurately detect abnormal pixels that lie far away from other observation values. Therefore, in ROIs containing a very complex background and low signal-to-noise ratio (SNR), we can precisely find abnormal pixels and prevent the results to be over or under-segmented.

Outlier detection is a problem of finding patterns in data that are not in the range of normal behavior. In this paper, hot spots in IR images are considered anomalous patterns and treated as outliers. An outlier will thus indicate a temperature abnormality. To identify defective idlers, we apply the IQR method to the extracted histogram features ${\mathcal{F}}_i:=\left\{f_1,f_2,\dots ,f_{255}\right\}$.

IQR is a technique that helps to find outliers in the data which are continuously distributed. IQR is the difference between the first quartile and the third quartile: $IQR=Q3-Q1$ where, $Q1$ and $Q3$ can be defined by Equation (8) [46,47].

$$\begin{array}{c}Q1=\overline{g}-0.675\sigma ;\\ Q3=\overline{g}+0.675\sigma \end{array}$$

(8)

The following thresholds (referred to as fences) are required to be defined for classifying the outliers in two different classes. The outliers can be defined as values that are either among inner and outer fences (mild outliers) or beyond outer fences (extreme outliers). The lower and upper inner fences can be computed as Q1 − 1.5 IQR and Q3 + 1.5 IQR, while the lower and upper outer fences can be defined as Q1 − 3 IQR and Q3 + 3 IQR, respectively [48,49].

Let $\alpha$ represent the pre-processed ROI and $\beta$ represent the extracted binary image from $\alpha$ and T as the threshold value by the proposed method. Furthermore, W is ROI width and H is image height. Since this captured IR is a digital image, x and y are indenting the coordination of pixels.

$$\begin{array}{cc}\hfill & \beta (x,y)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\alpha (x,y)>T\hfill \\ 0\hfill & \mathrm{if}\alpha (x,y)\le T\hfill \end{array}\right.\hfill \\ \hfill & \forall 0\le x<W,0\le y<H\hfill \end{array}$$

(9)

Likewise, all the pixel values of pre-processed $\alpha$ are set to 1 when the pixel values are greater than the computed T. On the other hand, the other pixel values are set to 0 when they are less than the defined T. In order to obtain object image $\gamma$ from image $\alpha$ and $\beta$, the following formula is used:

$$\begin{array}{cc}\hfill & \gamma (x,y)=\left\{\begin{array}{cc}\alpha (x,y)\hfill & \mathrm{if}\beta (x,y)=1\hfill \\ 0\hfill & \mathrm{if}\beta (x,y)=0\hfill \end{array}\right.\hfill \\ \hfill & \forall 0\le x<W,0\le y<H\hfill \end{array}$$

(10)

After the segmentation process, $\gamma$ is the thermal image of the inspected equipment after removing the background.

Through examination, we found out that overheated idlers cannot always be defined as mild outliers. Accordingly, for increasing the chance of true detection, the value of extreme outliers is extracted for each frame and considered as an optimal threshold value for segmenting the overheated idlers Figure 5.

3.8. Canny Edge Detection Implementation

In the previous section, we could accurately partition the input image into homogeneous hotspots and backgrounds using an adaptive thresholding method. The post-segmentation processing algorithms including canny edge detection and blob detection are designed to differentiate the hotspots edges from each. Our objective is to extract the boundary of the segmented hotspots and classify them as separate sources of heat.

The aim of performing edge detection, in general, is to significantly reduce the amount of data in an image, which will increase the computation speed of the approach while preserving the structural properties of the image [50,51,52]. The correctness of detected edges is an important factor that affects the blob extraction step; therefore, both the edge detection and blob extraction stages are tightly connected. To find the shape and size of the extracted hot spots, firstly, we used the canny edge detection method for detecting the boundaries of the segmented hotspots.

The canny edge detector is the most common method for detecting a wide range of edges in images. It uses a multi-stage algorithm consisting of five separate steps: smoothing, gradient finding, non-maximum suppression, double thresholding, and edge tracking by hysteresis for detecting boundaries in images. It determines the spots in images more accurately than other operators. It convolves the segmented frames with a Gaussian filter for reducing the noises and then computes the gradient and gradient direction for each possible edge. Furthermore, the detected image gradients undergo double thresholding to remove edge-like noise [53].

3.9. Blob Detection

Now, because all boundaries of the segmented hotspots are detached, they can be detected and counted using techniques such as blob detection. A blob can be defined as a region inside the calculated boundaries in which the pixels are considered to be similar to each other, while they should be different from the surrounding neighborhoods. Blobs are defined as interest points or interest regions. The interest points are referred to as local extremes in scale-location spaces, which will indicate circular or square regions.

The blob detection method is commonly used in many applications that are related to measuring the object’s shape, location, diameter, etc. For providing complementary information about the number of hot spots which are not obtained from edge detectors, the blob detection algorithm is applied to the canny edge detection results.

We used a blob detector method based on the Laplacian of the Gaussian (LoG). Therefore, an image is convolved by a Gaussian kernel. Furthermore, a multi-scale blob detector with automatic scale selection is then computed using a scale normalized Laplacian operator (see Figure 6). After labeling circular shapes, we could detect and count the different overheated elements in ROIs [54,55].

3.10. Performance Metrics

Six performance metrics—Sensitivity, Specificity, Precision, Accuracy and Matthew’s correlation coefficient (MCC)—were used as evaluation metrics where TP, FP, TN, and FN are true positive, false positive, true negative, and false negative, respectively.

$$\mathrm{Sensitivity}=\frac{\left(TP\right)}{(TP+FN)}$$

(11)

$$\mathrm{Specificity}=\frac{\left(TN\right)}{(TN+FP)}$$

(12)

$$\mathrm{Precision}=\frac{\left(TP\right)}{(TP+FP)}$$

(13)

$$\mathrm{Accuracy}=\frac{(TP+TN)}{(TP+FN)+(FP+TN)}$$

(14)

$$\mathrm{MCC}=\frac{\mathrm{TP}\times \mathrm{TN}-\mathrm{FP}\times \mathrm{FN}}{\sqrt{(\mathrm{TP}+\mathrm{FP})(\mathrm{TP}+\mathrm{FN})(\mathrm{TN}+\mathrm{FP})(\mathrm{TN}+\mathrm{FN})}}$$

(15)

In this paper, true positives refer to the frames where overheated idlers were correctly segmented. On other hand, false positive represents the number of frames where other thermal sources were wrongly segmented as overheated idlers. Furthermore, true negative cases are referred to as the frames where no overheated idler was neither present nor detected. In contrast, a false negative can be described as the number of frames where faulty idlers were not segmented due to the underestimation.

4. Experiments and Data Description

The experiments were carried out by capturing data from a conveyor system located in an opencast mine close to the bunker where transported material was stored.The mining site is located in Jaroszów, 50 km to the west from Wrocław. The basis of our measurement system is a remote-controlled mobile robot with an extensive navigation system figure. The robot is custom built for the Wrocław University of Science and Technology as a universal mobile platform for inspections, as shown in Figure 7 with main features explained in

Table 1. Inspection robot main characteristics.

Locomotion type	Wheeled, skid steering
Navigation system	Autonomous (Internal computer)
	Manual (Pilot using remote computer conection)
Internal software	Robot Operating System (ROS)
Power system	Internal battery, 24 V
Robot gross weight	65 kg
Maximum payload capacity	75 kg

. During the inspection mission, the robotic platform captured various types of data, including RGB images, IR images, sound, LiDAR data, etc. The camera system with a fixed point of view was directed toward the conveyor to better cover the ROI. The main specifications are mentioned as follows:

The analyzed conveyor system in this paper is a mechanical system used for the continuous horizontal transport of raw materials including raw clays, milled clays, and chamotte from the mine pit to the bunker. The considered conveyor was the last section that ends the entire series of conveyors. The analyzed section of the conveyor system always runs without the material, because the material is dumped into appropriate silos just before this section. The belt itself carries the material, which is important for thermal reasons, but the idlers do not experience the additional weight of the material.

The investigated conveyor system was several hundred meters long, and it operates in harsh environmental conditions. The design of the conveyor is classical, as shown in Figure 8. The key problem was to identify overheated idlers as a potential source of the fire. It is worth noting that inspection had not started at the very beginning of the belt conveyor.

Data Description

Based on the manual analysis of the acquired data (additional information in

Table 2. (a) The camera system parameter, (b) Belt conveyor parameters.

(a)
Parameter	Value
Resolution	640 × 480 pixles
Frames per second	25 fps
Observation angle	45${}^{\circ }$
Mounting height	100 cm above shelf
(b)
Parameter	Value
Conveyor length	150 m
Idler diameter	133 mm
Idler spacing	1.45 m
Belt width	800 mm

), we have selected several interesting situations that could be problematic during automatic image analysis. Below, we present some of such “difficult to analyze” pictures. In general, one may group them into several classes, namely: images without a hot idler, images with a hot idler, images with sunlight reflection without a hot idler, images with sunlight reflection with a hot idler, etc., see the examples presented below Figure 9.

Despite the advantage of IR imaging methods, there are still different factors that need to be considered even when conducting an indoor inspection. Generally, the precision of a thermographic measurement is directly related to specific background parameters, i.e., the environmental conditions, the optical properties of the target material, and the possible presence of any nearby object. Objects with a high emissivity value such as greasy, black or reflective objects have quite a high emissivity value, typically as high as 0.97; therefore, they can strongly reflect the IR radiations [56]. In Figure 10, one can notice that in the raw IR images, there are many non-informative heating sources that are not related to the conveyor system: for example, windows or sunlight reflection on the belt (marked by arrows).

The experiment was conducted on a sunny day. In a real scenario, it is impossible to predict what the weather will be like, and since the measurement session was organized in a real mine ahead of time, there was no way of predicting the weather several weeks ahead. However, we managed to provide a solution that can be applied in every condition.

Solar radiation can heat the equipment, especially those with high absorption of the sun’s energy, which can make small thermal differences. In our case (inspecting the idlers), the solar radiation was mostly blocked by the celling; therefore, their effects on the surface temperature of idlers were neglectable. However, one can notice that the belt surface is black, smooth, and shiny, which makes a very probable sunlight reflection problem. Therefore, the IR camera measured the reflected temperature instead of measuring the temperature of the belt itself. This will make image processing difficult, and this is the main reason to limit the analyzed area to conveyor-related only (defining the ROIs).

In Figure 11, one can see another example of hot spots (marked by frames) not related to idlers. It shows that the application of real industrial data is always difficult due to unpredictable sources of noise/unwanted components.

As result, the methodology will provide a false detection of sunlight recognized as a hot area. Examples presented here with a detailed discussion on the detection efficiency and understanding of the source of the problem are necessary before the automatic processing of hundreds of images. In the next sections, we will discuss global efficiency with some indicators of detection quality. Even if there are some problematic examples that are hard to recognize, other known techniques appear much less efficient. Moreover, the proposed technique is automatic and provides results in an objective way that is better than the subjective opinion performed by experts.

5. Results and Data Validation

The performance of the method in the identification of overheated idlers was tested on image data sets that were captured from a conveyor systems. The inspection robot moved alongside the conveyor system two times (back and forth) and captured data. Initially, 6275 frames from data set 1 (moving forward) and 6135 frames from data set 2 (moving backward) were extracted from captured RGB and IR videos. After applying the keyframe extraction, we could reduce the data set size up to 40%; therefore, 2470 frames from data set 1 and 2205 frames from data set 2 were chosen for further analysis. Furthermore, we compared the performance of the proposed method in identification of overheated idlers with Maximum entropy, Yen and Minimum method [26].

5.1. Validation of Detection Results Based on Manual Analysis

The test images were hand labeled. The evaluation was performed by comparing the final segmentation mask through the visual interpretation process. By consideration of the number of thermal sources that can be detected as an overheated idler alongside the conveyor, validation of the segmentation results is important, as other thermal sources can be wrongly segmented as overheated idlers. Therefore, through the validation process, the shape and location of the segmented hotspots in segmented frames were compared to RGB images, as shown in Figure 12.

5.2. Results

The specificity and accuracy value for both data sets were about 0.98. The specificity value indicates the performance of the proposed method regarding identifying true positives and true negatives, while the accuracy value discussed the proportion of correctly predicted samples among the total number of the processed samples. The precision value determines the ratio of correctly predicted positive observations to the total predicted positive observations, which were above 0.66 for both data sets.

We additionally computed the F1-score for both data sets (see

Table 3. Comparison of the performance factors of the proposed method with other selected methods—data set 1.

Measures	Proposed Method	Maximum Entropy	Yen	Minimum Method
Sensitivity	1	1	1	1
Specificity	0.98	0	0	0
Precision	0.62	0.05	0.03	0.03
Accuracy	0.98	0.05	0.03	0.03
F1-score	0.76	0.1	0.05	0.06

and

Table 4. Comparison of the performance factors of proposed method with other selected method—data set 2.

Measures	Proposed Method	Maximum Entropy	Yen	Minimum Method
Sensitivity	0.94	1	1	1
Specificity	0.98	0	0	0
Precision	0.66	0.05	0.05	0.01
Accuracy	0.98	0.05	0.05	0.01
F1-score	0.78	0.1	0.1	0.03

). The F1-score combines the precision and recall of a classifier into a single metric by taking their harmonic mean and measure of accuracy incorporating both the precision and recall. It can be used as a single performance test for positive classifications, which were 0.76 for data set 1 and 0.78 for data set 2. The F1-score of the proposed method was 80% higher than the compared methods, which indicates the low performance of other thresholding methods in the true detection of overheated idlers in the studied data sets.

6. Concluding Remarks

Analyzing the intensity distribution of pixels from the acquired IR images by means of ROI analysis, canny edge detection, blob detection methods and the fusion of segmented IR and RGB frames showed an evident correlation between abnormalities in pixel intensity (temperature patterns) and the existence of hotspots on the surface of the faulty idlers. Furthermore, the proposed diagnostic approach gave promising results, succeeding to diagnose four out of four defective idlers in the studied conveyor system.

In conclusion, robotic-based IR thermography with the combination of thermal image processing techniques was proved to be a potential and reliable method for CM and fault diagnosis of idler modules. The presented approach gave easily interpreted results and fast detection of faulty idlers, utilizing both qualitative and quantitative data from the processed thermal images from the two conveyor systems.

The most important limitation of the proposed method is the fact that any specular object present in the background could cause unwanted gray-level variations that may be conflicting with the actual variations related to hotspots that may cause false alarms. In order to reduce unwanted gray-level variations, sets of pre-proposing algorithms, including CLAHE, adaptive gamma correction, and median filter, were applied to ROIs.

In grayscaled IR images, the changes in temperature are indicated with changes in pixel intensity; therefore, they depict some degree of smoothness during the traversal from one pixel to another, and there is a lack of sudden and sharp change. Hence, most of the conventional automatic methods have different performances in the automatic thresholding of IR images as compared to their performances with other types of images.

There are further limitations referring to emissivity uncertainties, the presence of sunlight reflection on the belt surface of the conveyor systems, and the presence of other heat sources in mining sites that have to be always taken into account for field measurements.

Additional investigation in improving the detection results can be completed based on the definition of adaptive ROI on extracted IR and RGB frames. Through the conducted examination, we found out that in some cases, as long as the mobile robot was moved through the harsh surfaces, obstacle avoidance maneuvers and sudden changes in the robot paths were necessary. The camera system position was fixed during the inspection; therefore, the camera’s point of view (POV) was changed due to sudden modifications in the mobile robot path. For addressing sudden changes in camera POV and improving the detection results, a mapping technique that fuses the camera system with the LiDAR data for estimating the optimal region of interest on captured videos is of further interest to the current research team.

Characterizing idler defects, e.g., bearing defects, can be completed based on a micro-scale analysis combining the current experience with active thermography approaches or with the fusion of acoustic and thermal imaging approaches for a more accurate CM plan. In addition, a more thorough understanding of degradation mechanisms and failure modes in idler modules is critical to improvements in idler design and reliability. Performing robotic-based IR imaging in large-scale conveyor systems with new fusion techniques can be considered in future research.