A New Belt Ore Image Segmentation Method Based on the Convolutional Neural Network and the Image-Processing Technology

Abstract

In the field of mineral processing, an accurate image segmentation method is crucial for measuring the size distribution of run-of-mine ore on the conveyor belts in real time0The image-based measurement is considered to be real time, on-line, inexpensive, and non-intrusive. In this paper, a new belt ore image segmentation method was proposed based on a convolutional neural network and image processing technology. It consisted of a classification model and two segmentation algorithms. A total of 2880 images were collected as an original dataset from the process control system (PCS). The test images were processed using the proposed method, the PCS system, the coarse image segmentation (CIS) algorithm, and the fine image segmentation (FIS) algorithm, respectively. The segmentation results of each algorithm were compared with those of the manual segmentation. All empty belt images in the test images were accurately identified by our method. The maximum error between the segmentation results of our method and the results of manual segmentation is 5.61%. The proposed method can accurately identify the empty belt images and segment the coarse material images and mixed material images with high accuracy. Notably, it can be used as a brand new algorithm for belt ore image processing.

1. Introduction

The particle size distribution of run-of-mine ore exhibits a great influence on the grinding process. Variations in the particle size distribution directly affect the throughput and power consumption of mills, especially autogenous (AG) and semi-autogenous (SAG) grinding mills [1]. Therefore, it is critical to evaluate the size distribution of run-of-mine ore on the conveyor belts in real time [2,3]. The measurement of the particle size distribution by sampling and sieving is considered a common and time-consuming method. The analysis method based on machine vision is considered a non-invasive, fast, and inexpensive technique for rock size measurement [4]. Since the 1980s, many studies have been conducted to evaluate the particle size distribution of materials on a conveyor belt based on machine vision and image processing technology [2,5,6]. Scholars have mainly followed three aspects of exploration. The first aspect includes accurate ore contour detection algorithms. The second aspect is the reasonable evaluation model which is used to convert two-dimensional information of ore into three-dimensional information, and then evaluating energy consumption or particle size distribution. The third aspect is new technology including neural networks, deep learning, and genetic algorithms, etc. In 1988, Lange developed an on-line, real-time system which can capture images of rocks on conveyor belts, and process images to get the chord-length distributions, and then transform chord-length distributions to equivalent sieve sizes. Lange offered a method to distinguish belt ores of different size distributions [2]. Lin and Miller developed an image-based system which used image processing technology to get the chord-length of rocks, and used two kernel functions to calculate the cumulative chord-length distributions of regularly and irregular shaped particles, respectively. The last step was to transform the chord-length distributions into size distributions by the transformation Equation [5]. In 1997, Yen and co-workers used an empirical correction function to solve the coarse particle overlap problem [7]. Before 2000, limited by hardware technology, it was difficult to get sharp images and many algorithms that consumed too much computer performance could not be adopted. The scholars mainly researched the image-based system software and hardware framework, contour detection, reasonable measurement parameters and size transformation functions.

After 2000, with the rapid development of computer hardware and new technologies, the image-based, online, and real-time particle size measurement development made much progress. Singh and Mohan Rao extracted RGB color information, visual texture of particles, and developed a system based on a radial basis neural network. The system was used for ore classification and ore sorting [8]. Al-Thyabat and co-workers evaluated ore image segmentation results by means of Feret’s diameter and equivalent area diameter, and experimented and discussed the effect of camera positions [9]. Levner offered a classification-driven watershed segmentation to segment belt ore images, and adopted machine learning to produce markers and identify ore edges [10]. Outal et al. provided a calibration method for evaluating 3D size distribution, according to the 2D segmentation results [11]. Andersson evaluated the size distribution of particles using ordinal logistic regression [12]. Hamzeloo et al. used different particle equivalent models to evaluate 3D size distribution. These equivalent models included the equivalent area circle, best-fit rectangle, Feret diameter, and maximum inscribed disk [4]. In addition, the impact of the shape of the particles on the product properties was also researched extensively [13,14,15,16]. Until now, many image-based analysis methods have been successfully applied to evaluate the particle size distribution [17,18,19], however there are still three unresolved problems regarding the image-based, on-line particle size analysis methods.

One is that there is no research focus on the problem of empty belt identification. In the course of production, we should not turn on or switch to the empty conveyor belts. Therefore, the accurate recognition of the empty belt is necessary to realize the automatic control and switching of conveyor belts. The second problem is the accurate belt ore image segmentation method, especially for the coarse-fine images. According to the experimental results of previous researches, it cannot be concluded that the proposed method can accurately segment both coarse and fine materials. An accurate image segmentation method is the basis of a particle size distribution measurement system. Although there is a method which uses machine learning to identify the pixels of ore edges, the contour detection based on image segmentation is more stable, accurate, and adaptable. In the future, the ore image segmentation method based on deep learning will be an expected practice. The third problem is related to the overlapping of particles. Even though many studies have been conducted in order to find viable solutions to these problems, most of them rely on empirical correction [4,7,20]. Dynamic image analysis (DIA) is a feasible method to solve the overlap problem [15].

Traditional rock image analysis methods cannot distinguish different types of images, such as empty belt images, mixed material images, and coarse material images, which are distinct. The mixed materials include coarse-fine materials and fine materials. It is difficult to accurately process all three types of belt ore images with an image segmentation algorithm; therefore, our analysis method should be able to accurately classify the images we obtain. In recent years, with the rapid development of computer hardware and deep learning theory, the convolutional neural networks (CNNs) have shown great progress in the field of image recognition and classification [21,22]. Krizhevsky et al. developed AlexNet, which can reach 83.6% top-5 accuracy for the ImageNet dataset [23]. At present, the top-5 accuracy of many convolutional neural networks in image recognition tasks can reach more than 90% for the ImageNet dataset [24,25,26]. Many research studies on rock image recognition and classification based on deep learning have achieved high accuracy [8,27,28].

In this research, the method based on the deep learning method and the image processing technology was developed in order to accurately segment the belt ore images. The strategy was to classify the belt ore images into empty belt, mixed materials and coarse materials first and then use different algorithms for processing mixed materials and coarse materials. We focused on the accuracy of belt ore image segmentation and empty belt identification. Both the conversion model for converting two-dimensional information of ore into three-dimensional information and the impact of the shape of the particles on the product properties are beyond the scope of this article.

2. Details of the Method

The proposed method is divided into three layers. The first layer is a classifier based on a convolutional neural network. The second layer consists of two image processing algorithms based on the OpenCV library. The two algorithms are used to process coarse material images and mixed material images, respectively. The third layer is the statistics layer. The classifier divides the raw images into the empty belt, coarse materials, and mixed materials. If the belt is empty, it gives an alarm; otherwise, it uses the coarse image segmentation (CIS) algorithm to process coarse material images and uses the fine image segmentation (FIS) algorithm to process mixed material images, respectively. Finally, the cumulative area distribution is calculated following the counting segmentation area information. The technical roadmap is shown in Figure 1.

2.1. The Convolutional Neural Network Classifier

2.1.1. The Dataset Preparation

The size of the dataset is considered crucial for evaluating the performance of the trained model. An insufficient dataset causes a low recognition accuracy of the trained model. The dataset used in this study consisted of 2880 images collected from the process control system (PCS) system of a mineral-processing plant in the Yunnan Province, China. The images were taken by eight cameras installed on eight feeding belts with a collection rate of eight photos per minute for each camera. AXIS P3227-LVE cameras from AXIS are used, as well as LED PCS6-LED80 W lamps from Woodgrove. Two belt ore images are taken by each camera continuously every 15 s. The PCS system stores the latest 100 pictures from each camera; therefore, the images in the PCS system are completely updated every 12.5 min. We wrote a Python script to transfer the pictures from the storage folders to specified folders, and the transfer was executed every 13 min, lasting for a week. In the image transfer stage, the goal is to obtain sufficient belt ore images. It is efficient and economical to directly transfer images saved by the PCS system. At the same time, only two feeding belts are in running; therefore, there are a lot of duplicate images in the specified folders. All of the images have a size of 2304 × 1728 px (JPEG file). We planned to select about 3000 sharp, non-repetitive, and representative belt ore images as the original dataset. The 982 empty belt images were picked out including empty belts with water stains, empty belts with small particles, images taken by telephoto lens, and images taken by short focal length lens. We divided the belt ore images with fine content less than 30% into coarse material images. The features of coarse material images are obvious and similar; therefore, the number of coarse material images can be reduced appropriately. The 841 coarse material images were selected. The 1057 mixed material images were selected according to the proportion of fine material. The mixed material images were selected consisting of 100%, 90%, 70%, and 50% fine material. All images were taken from an industrial site and were not created; therefore, the proportion of fine material was an estimation and not an exact value. The 2880 high-quality, representative images were selected from the saved images as the original dataset. After physical verification, 522 px in each image was found to be equal to 30 cm. Several examples of the images are shown in Figure 2.

2.1.2. Model Training

The design of our network is shown in Figure 3. The network consists of two convolution layers, two maxpool layers, two fully connected layers, and three ReLU activation functions. The DELL R730 server was used to train the model. The Windows Server 2012 was used as the operating system. The Intel E5–2609V4 was used as the CPU with a RAM of 32 GB, and the Nvidia P2000 (5 GB) was used as the GPU.

The original dataset was used as the raw data. The input images consisted of three channels, which were resized to 500 × 500 × 3. All input images were required to undergo a two-step pretreatment process. In the first step, the value range of pixels was changed from 0–255 to 0–1. In the second step, the image was normalized by using the empirical mean vector and the empirical std vector. The normalization is described as follows:

$$mean=\left[0.485, 0.456, 0.406\right]$$

(1)

$$std=\left[0.229, 0.224, 0.225\right]$$

(2)

$$result=\left(image-mean\right)/std$$

(3)

The original dataset was completely shuffled: 20% of the images assigned to the test set, 20% of the images assigned to the validation set, and 60% of the images used as the training set. The processed images were input into the neural network for model training. The epoch value was set to 10, the batch size was set to 32, the learning rate was set to 0.001, and the cross-entropy was used to evaluate the training loss. The prediction result was compared with the true label in order to calculate the training accuracy and the validation accuracy of every epoch. The training accuracy and validation accuracy were used to update the weights in the model [29]. The training process is shown in Figure 4. The training accuracy was 99.48%, the validation accuracy was 100%, and the training cross-entropy equaled 0.0146 when the epoch equaled 10. The model was tested with the test set, and the prediction accuracy was 100%.

2.2. CIS and FIS Algorithms

2.2.1. CIS Algorithm

The CIS algorithm processes images classified by the classifier as coarse materials. The CIS algorithm is based on the Python OpenCV library (version 4.1.1). The uneven color distribution on the surface of the coarse ores and the coarse ores covered by fine particles led to the region of coarse ores in the image being divided into many small regions. Therefore, the CIS algorithm should be able to remove features that are similar to the ore edge on the surface of the coarse ore. The CIS algorithm is described in Figure 5.

The CIS algorithm follows five steps. In the first step, the input image is processed using the bilateralFilter function. The setting value of the diameter of each pixel neighborhood equals 25, the sigma parameter of the color space equals 100, and the sigma parameter of the coordinate space equals 25. The purpose of bilateral filtering is to blur the surface of coarse materials while preserving the edge. The parameters of the bilateralFilter function were chosen according to experience and practice; for example, 25, 50, 75, 100. After bilateral filtering, the noise, details, and small color blocks on the surface of the ore are blurred, and the edge of the ore is preserved [30,31]. In the second step, the operations include graying [32], adaptive thresholding [33], and median filtering [34,35]. The setting kernel of the median filter equals 3. The kernel of the median filter should not be too high, to prevent edge interruption. After several tries, the results processed using the kernel with a value of 3 was more suitable for subsequent processing than 5 or 7. After the second step, the color image is binarized and denoised. In the third step, the image is processed using the findContours and drawContours functions. After these operations, the interconnected areas are closed. In the fourth step, the image is processed using the erode and morphologyEx functions [36,37,38]. The setting kernel is a 3 × 3 morph ellipse. The iterations parameter of the erode function is 10 and the iterations parameter of the morphologyEx function is 4. The target of erosion and open operation is to extract markers. The iterations parameters are sensitive, and are chosen by experience and trials. Finally, the watershed transformation is performed based on markers, and the watershed lines are drawn [10,39,40]. For instance, Figure 6 shows how the CIS algorithm processes a coarse material image.

2.2.2. FIS Algorithm

The FIS algorithm processes the images classified by the classifier as mixed materials. The FIS algorithm is based on the Python OpenCV library (version 4.1.1). Both fine and coarse-fine materials are mixed materials. The edge of fine materials is weak; therefore, excessive blurring causes the under-segmentation of fine material images, and insufficient blurring causes the over-segmentation of coarse material images. The FIS algorithm must remove the edge-like features on the surface of coarse ores as much as possible while retaining the outlines of granular particles and fine material. The FIS algorithm is described in Figure 7.

The FIS algorithm follows four steps. In the first step, the input image is processed using the bilateralFilter function. Due to the need to deal with the coarse ores which are contained in the mixed materials, it is necessary to use bilateral filtering. Parameter settings are found to be the same as the CIS algorithm [30,31]. In the second step, the operations include graying [32], adaptive thresholding [33], and two median filterings [34,35]. The setting kernels of the median filters equal 3. After the second step, the color image is binarized and denoised. In the third step, the image is processed using the erode function and median filtering. The setting kernel of the median filter equals 9 and the kernel of the erosion operation is a 5 × 5 morph ellipse [38]. Due to the edge of fine materials being weak, we should not adopt strong morphological operations. Therefore, the iterations were set to one. To ensure complete separation among markers, the kernel of the median filter in the third step was set to 9. Finally, the watershed transformation was performed based on markers and the watershed lines were drawn [10,40]. A mixed material image was processed using the FIS algorithm as shown in Figure 8.

3. Experimental Details

3.1. Image Acquisition and Classification

The experimental dataset covers 12 representative images which consist of four empty belt images labelled as one, four mixed material images labelled as two, and four coarse material images labelled as zero. All images were taken from the PCS system at different times. We selected four empty belt images including an empty belt with water stains, an empty belt with small particles, an image taken by a telephoto lens, and an image taken by a short focal length lens. The features of coarse material images are obvious and similar. The four coarse material images were selected randomly. The four mixed material images were selected consisting of 100%, 90%, 70%, and 50% fine material. The raw images were divided into the following four groups: one empty belt image, one mixed material image, and one coarse material image in each group. In order to compare the segmentation results of the different algorithms, the designated region of raw images was processed by using different algorithms. The size of the designated region was 1202 × 631 (JPEG file). Manual segmentation was used to obtain accurate segmentation images. Images from group one (as shown in Figure 9a) were processed using PCS, CIS, and FIS, respectively, for evaluating each algorithm. Other images were processed and evaluated by using our method.

3.2. Estimation of Segmentation Accuracy

The area (px) of the regions surrounded by contours in the segmented images were counted, and the cumulative area distribution was calculated. Suppose $x_1$, $x_2$, …, $x_n$ are areas of the regions enclosed by the particle segmentation contours in a processed image. Among $x_1$, $x_2$, …, $x_n$, areas of the regions that are smaller than the specified values are $y_1$, $y_2$, …, $y_m$. The cumulative area distribution ($C$) is computed using the following equation:

$$C=\frac{{\displaystyle \sum _{i=1}^m{y}_i}}{{\displaystyle \sum _{j=1}^n{x}_j}}\left(m\le n\right)$$

(4)

We use the cumulative area distribution to evaluate the segmentation accuracy of algorithms. The specified value used to calculate the cumulative area distribution is called the area filter (px). If the number of pixels in the region surrounded by the contour is less than the area filter, the region can pass the area filter; otherwise, the region cannot pass. The contourArea function from the Python OpenCV library (version 4.1.1) was adopted to get the number of pixels in the region surrounded by the contour. The cumulative area distributions of the segmentation results of the empty belt image, mixed material image, and coarse material image are quite different; therefore, different area filters were used for calculating the cumulative area distribution of the segmented images. The empty belt images were evaluated using 2000, 4000, 8000, 10,000, 20,000, 40,000, 80,000, and 100,000 px area filters. The mixed material images were evaluated using 2000, 4000, 6000, 8000, 10,000, 20,000, 30,000, and 40,000 px area filters. The coarse material images were evaluated using 5000, 10,000, 15,000, 20,000, 25,000, 30,000, 35,000, and 40,000 px area filters. The cumulative area distribution and the number of segmentation contours of the image segmented by the PCS system, CIS, and FIS, respectively, were compared with the segmentation result of the manual segmentation image. The segmentation result of the algorithm, which was close to the manual segmentation result, was evaluated as accurate.

4. Experimental Results and Discussions

4.1. The Segmentation Result Analysis of Different Algorithms

Images in Figure 9a were segmented using manual segmentation, PCS, CIS, and FIS, respectively, providing results as shown in Figure 9b–e. By counting the number of segmentation contours in the segmented images, the results are shown in

Table 1. The segmentation contour count of various algorithms.

Classification	Segmentation Contour Count
	Manual	PCS	CIS	FIS
Empty belt	0	371	51	92
Mixed materials	885	942	123	955
Coarse materials	101	408	91	280

. By calculating the cumulative area distribution of the segmented images, the results are shown in Figure 10. Table 1 shows that the segmentation contour counts of the empty belt image by the PCS system, CIS algorithm and FIS algorithm, respectively, are 371, 51, 92. Figure 10a and Table 1 demonstrate that no algorithm can accurately segment the empty belt images. Due to many disturbing factors such as small particles, dirt, and water on the surface of the empty belts, the segmentation algorithms always segments the empty belt images. We only require to recognize and need not segment the empty belt images.

Table 2. The results of empty belt images from groups 2 to 4, as processed by our method.

Groups	Status
2	Alarm
3	Alarm
4	Alarm

shows our method alarms all three empty belt images from groups 2 to 4. Table 2 indicates that using the training model based on the convolutional neural network to process the empty belt images shows better performance and accuracy. From Figure 10c and Table 1, it is observed that the PCS system shows serious over-segmentation in processing the coarse material images, and the segmentation accuracy of the FIS algorithm is found to be between that of the CIS algorithm and the PCS system. The outlines of massive rocks are obvious as compared to those of granular particles and fine material. Many disturbing factors, such as edges and corners, uneven color distribution, shadow, and so on, are found on the surface of massive rocks. An algorithm that can accurately segment the coarse material images must overcome the above interferences, however, the blur and denoise operations used cannot weaken the outlines of coarse ores too much. Figure 9 shows that the CIS algorithm has obvious advantages for segmenting the coarse material images. The blur, denoise, and smooth operations of the CIS algorithm weaken the outlines of ores; therefore, there is a serious under-segmentation in processing mixed material images, especially the fine material images (see Figure 9 and Figure 10b). We classified the images in the original dataset, which are neither coarse material images nor empty belt images, as mixed material images.

As the number of images in the original dataset is very high, it is not feasible to count the particle size distribution of materials on conveyor belts. The method of distinguishing empty, mixed materials, and coarse materials is based on artificial classification. An image that is full of massive ores is classified as a coarse material image. The accurate segmentation of mixed material images is most difficult to achieve. The massive ores show a greater impact on the grinding process. The results calculated by the under-segmentation of fine material images are found to be much coarser than the actual results. Fewer morphological operations enable the FIS algorithm to retain the outlines of granular particles and fine materials. More denoise operations enable the FIS algorithm to remove the interferences on the surface of the massive ores as much as possible. Figure 10b shows that the FIS algorithm shows accurate segmentation in processing mixed material images, especially the fine material images. As the FIS algorithm cannot remove the large non-edge features on the surface of massive rocks such as edges, corners, and color blocks, the FIS algorithm does not show as accurate results as the CIS algorithm in processing the coarse material images (see Figure 10c and Table 1). It is more accurate and reasonable to classify images first and then use different algorithms for processing rather than to process all images with the same algorithm.

4.2. The Processing Result Analysis of Our Method

Images from groups 2 to 4 were processed by using both our method and manual segmentation. Three mixed material images include one fine material image from group 2 and two mixed material images from groups 3 and 4. The results of the three processed empty belt images are shown in Table 2. The results of the three processed mixed material images are shown in

Table 3. The results of mixed material images from groups 2 to 4, processed by our method.

Groups	Methods	Cumulative Area Distribution (%)
		2000	4000	6000	8000	10,000	20,000	30,000	40,000
2 (F)	Our method	55.59	85.64	96.82	100	100	100	100	100
	Manual	58.06	87.6	94.11	98.61	100	100	100	100
3 (M)	Our method	16.44	36.67	50.89	62.29	69.54	89.07	92.08	92.08
	Manual	18	38.09	55.54	66.82	73.67	89.49	92.8	92.8
4 (M)	Our method	26.4	50.64	67.1	77.11	90.78	95.17	100	100
	Manual	28.09	55.66	71.1	80.66	94.77	100	100	100

F: fine material; M: mixed materials.

. The results of the three processed coarse material images are shown in

Table 4. The results of coarse material images from groups 2 to 4, processed by our method.

Groups	Methods	Cumulative Area Distribution (%)
		5000	10,000	15,000	20,000	25,000	30,000	35,000	40,000
2	Our method	8.86	26	41.58	56.73	63.02	72.61	79.91	85.05
	Manual	10.43	29.51	41.94	57.48	64.07	74.07	80.65	85.99
3	Our method	6	15.38	25.81	40.83	57.06	64.08	79.86	84.57
	Manual	7.34	18.55	31.42	43.26	61.66	67.26	85.26	85.26
4	Our method	8.49	22.74	33.68	43.76	51.15	66.5	71	74.34
	Manual	6.92	25.5	37.02	46.46	53.02	63.73	72.86	78.17

. The cumulative area distribution of one fine material image from group 2, one mixed material image from group 3, and one coarse material image from group 2, which were processed by our method, are shown in Figure 11. The column headers in Table 3 and Table 4 (5000, 10,000, etc.) indicate area filters. Table 2 shows that three empty belt images used for testing were alarmed. Our method is found to be accurate and reliable for empty belt identification. From Table 3, it is observed that the maximum error of the cumulative area distribution calculated by using different area filters is 2.71% for fine material, 4.65% for mixed materials from group 3, and 5.02% for mixed materials from group 4. From Table 3, the average error of the cumulative area distribution calculated by using different area filters is 1.07% for fine material, 2.27% for mixed materials from group 3, and 2.89% for mixed materials from group 4. From Table 4, it is observed that the maximum error of the cumulative area distribution calculated by using different area filters is 3.51%, 5.61%, 3.83%, respectively, for coarse materials from groups 2 to 4. From Table 4, the average error of the cumulative area distribution calculated by using different area filters is 1.30%, 3.30%, 2.59%, respectively, for coarse materials from groups 2 to 4. Table 3 and Table 4 indicate that our method can segment both mixed material images and coarse material images with high precision. Our method is considered useful for identifying and segmenting the images taken on industrial conveyor belts.

5. Conclusions

The objective of this study is to develop a method that can accurately segment belt ore images. The accurate image segmentation method is considered important for estimating the size distribution of mineral materials on industrial conveyor belts. For that purpose, 2880 images collected from a process control system on the industrial site were processed as the original dataset. Deep learning and image processing techniques were integrated with the image segmentation method. From the perspective of an application, the accurate recognition of the empty belts is necessary to realize the automatic switch of conveyor belts. The new method can identify the empty belt images with high precision. Moreover, it is difficult for an image segmentation algorithm to achieve accurate segmentation of both coarse materials and mixed materials. This study adopted the convolutional neural network model to solve the automatic classification of belt ore images, and then used the CIS algorithm and the FIS algorithm to accurately segment coarse material images and mixed material images, respectively. The new method makes it feasible and efficient to accurately process various belt ore images. Notably, it can be used as a brand new algorithm for belt ore image processing. The main novelties are as follows:

• This study used a convolutional neural network to identify empty belts.
• The new method adopted the strategy which is to classify the belt ore images first and then use different algorithms for processing different kinds of images.