Laboratory methods are intense and time-consuming—samples are collected from a technological line, prepared, and then analyzed using specific applications. Another aspect is the cost of the system, especially when methods such as X-ray and laser diffraction are used. Different methods offer varying information on the grains, i.e., color, shape, size, type of mineral, share of metal in the ore. In this research, we wanted to determine the level of grinding that could be achieved and to be able to assess the amount of active surface of copper. A machine vision system, thus, was found to be a natural choice and optimal solution. We were able to achieve a level of 80–100 µm of grinding material. The proposed system can extract several types of grains features, which include color, texture, and morphology. The complexity of the copper ore grain structure (localized at different positions and angles) in the prepared sample could result in false outcomes. Thus, we decided to use different features extracted from the spatial and frequency domain to identify texture and morphology (grain shape and size), respectively, in a different step of the algorithm.
The grain size analysis methods described in the previous section have advantages and weaknesses; however, in order to determine the distribution of granularity fractions, we made several assumptions, which allowed us to choose one method of analysis:
3.1. Electromagnetic Mill Construction and Sample Preparation
The research and method described in this paper were performed with an optimized electromagnetic mill, developed and constructed at the Silesian University of Technology. A detailed description of the working principle, measurement, and control system can be found in [
38]. In this mill, material is ground by ferromagnetic grinding media (rods) in a working chamber. The main idea assumes that the mill structure and vertical position of the working chamber, which is filled with material from the top and a stream of transport air from the bottom, is stationary during the grinding process. The generated rotating electromagnetic field moves the grinding media and the feed material is ground by numerous collisions between rods and processed material’s grains. The size of the rods depends on the mill working chamber diameter and a feed, in most cases 1–3 mm of diameter and 10–15 mm of length. The grinding media moves in a chaotic manner inside the working chamber which increases the grinding process. The electromagnetic mill performance depends on specific roads size, operating frequency on the inducted field, material flow rate, material moisture. In comparison to the ball mill, electromagnetic mill requires specific grinding and classification rig with the integrated control system. The developed mill provides a significant reduction in energy consumption and a higher level of technological performance, effectively producing particles with specific parameters, especially with the desired shape such as sharp grain edges or more cubic ones with soft edges regarding the different granularity of the ground material.
In
Figure 2a, a schematic representation of the electromagnetic mill is provided, which includes components such as stream of the feed material (1); working chamber with electromagnetic field inductors and a preliminary classifier (2); working space with the ground material and the grinding media (3); source of air (4); main and additional air stream (5); stream of material that was not ground (6); machine vision grain detection and classification module (7); recycle stream of greater-than-assumed particle size (8); and final product (9), which moves to the cyclone and finally on to the final product tank. In
Figure 2b, the working chamber with moving in electromagnetic field rods is presented. In contrast to grinding with the feed material in this case rods moving on specific vertical lines, except some chaotic collisions. It can be controlled among other methods by correct selection of induction level, number, and position of induction windings.
The feed material was of variable size in the range of 0–2 mm and a minimum of 45 µm as final product. We decided to set the vision system with a minimum required parameters in a wide range of granularities such as camera resolution and lens. In another case, these parameters can be suited to the feed material granularity or target granularity product. Besides the type of dry or wet milling should be taken into account. Results in this paper have been obtained only for dry milling. Our method was tested on a computer vision laboratory stand selected according to the mentioned issues. The system is equipped with a 1624 × 1234 and 2448 × 2048 resolution monochromatic CCD 2/3”camera with global shutter, lens with 50 (49.93) mm focal length and F1.8-16 aperture, extension tube with set of spacer rings, max. 0.5, 1, 2, 5, 10, 20, 40 mm, one and two LED sources of co-axial spot white light with Ø6 mm diameter and a personal computer with developed software for control camera with illumination and grains detection and classification. In
Figure 2c the laboratory stand with the sampler connected to the mill rig is presented. The material is sampled with a predefined frequency directly from a rig and then transported gravitationally with a simple material slope strictly to the field of view of the camera. The detailed laboratory vision stand is presented on
Figure 2d. The proposed 5 MP resolution CCD camera and combination of spacer rings and lens for 1 cm
2 field of view will acquire and possible detection of grain size in the range below 100 μm, whereas for grains with a size of 80 μm, we have about 20 pixels, and for grains of about 45 μm, only 12, where evaluation of correct and real shape is a definitively hard, because the 12 pixels grains can be homogeneous or strictly rugged.
3.2. Pre-Processing Algorithm
Images of the copper ore samples were acquired by the vision system, which was equipped with a camera, two lights mounted on opposite sides, and a predefined camera axis angle between 30–45°; this illumination helped extract information on the edges and top of the grains. The acquired image contains valuable information (such as valleys/hills), which can be used to evaluate quality. Despite the angle illumination, we should deal with some low-luminance problems in the image. If the angle between illumination and sample will be small, the edges of the grains in the sample will be extracted in contrast to a planar region. Thus, the selected angle must be a compromise between overall image luminance and a number of extracted grains boundaries. In consequence, the pre-processing part of the algorithm was developed to improve detection results and processing time by eliminating not correctly prepared samples, before the main process is carried out.
Figure 3a–c shows a low-luminance image, wherein detection of grain size and shape is incorrect. To correct this, image contrast must be enhanced through global and local approaches with respect to grain’s edges. The first approach improves image contrast by extending a dynamic range of intensity using the histogram of the complete image. The local approach, on the other hand, only uses local information inside each separated block. In some cases, the blocks overlap, but the intensity of the whole image is still omitted. Recent literature discusses numerous global and local methods of contrast enhancing, such as based on the bi-histogram equalization median plateau limit [
39]; clipping the histogram, where maximum value of histogram is controlled by clipping histograms higher than the predefined threshold [
40]; combination of Histogram Equalization (HE) and histogram clipping in exposure based sub-image histogram equalization [
41]; and gradient-based local histogram equalization to preserve image texture [
42]; among others.
The authors of this paper used the HE algorithm, which enhances the finer details of low-luminance images. HE is an automatic and fast contrast improvement technique, which operates in the spatial domain on the pixel level of the image. HE changes the mean brightness of the input image to the middle level; in other words, it flattens the density distribution and stretches the dynamic range of gray levels in an image. Global contrast enhancement techniques have one main disadvantage—they increase or decrease contrast in an image at the same level, thus changing details of individual objects on simple, homogenous backgrounds. HE, on the other hand, boosts bright areas or removes dark regions in an image without difference to individual spots. This feature is clearly visible in complex scenes, where there is no uniform background and there exists numerous objects, or one larger object with numerous different items.
Figure 3b,d presents the results of HE of different copper ore fractions. The images of a large number of grains were enhanced by eliminating dark areas and enhancing light ones.
In the next step, the uniformity of the sample surface is examined with the use of the conditional cascade method. The cascade is deemed conditional because the second cascade is only executed when the sample passes the first. We observed that the surface of the sample was not flat due to defects such as valleys and hills, which were created when the material moved from the rig to the vision system. There were several defects even when the sample was mechanically flattened. It was not possible to remove all the defects, but we were able to use samples with small ones.
We used post-HE images as input, divided into four symmetrical, non-overlapping sub-regions. These sub-regions are processed separately in both cascades. As a result of our observations, we divided defects into two types based on their shape and size. The first type is usually visible in the dark areas of an image, which is associated with the formation of valleys of considerable size—the shape depends on the speed of material sliding, which directly depends on the granularity fraction. The regions of defects of this type are usually large in size and elongated in one direction—towards at least two neighboring sub-regions. In smaller granularity fractions, valleys have an irregular shape and are contained in a separate sub-region; however, they are often greater in number. The first type of defect is examined in the first cascade, which is based on the frequency domain with the use of fast Fourier transform. The second cascade, which is based on spatial information with the use of GLCM, detects a second type of defect.
The Fast Fourier Transform (FFT) [
43] is a technique that offers a wide range of applications, such as image analysis, image filtering, motion control, and compression. In general, low frequencies represent smooth parts of the image with small variations, while high frequencies denote rapid changes at a gray level, such as contours, edges, and rough objects. FT can, thus, be used to detect large defects, because they have an effect on low frequencies. A classic FT calculation, which consists of 2DFFT, a power density spectrum, shift to the center, and logarithmic transformation, was performed for symmetrical, non-overlapping sub-regions in this study. Each sub-region was described by a mean magnitude. In most cases, the values differ; thus, we rejected samples wherein the largest difference between two out of four mean magnitudes was greater than 2%. If that was not the case, we examined the sample using the second cascade.
Figure 4 shows a sample with the first type of defect. The bottom-left identifies a sub-region, large valley with a corresponding power spectrum (see
Figure 4c,e), which differs visually for lower frequencies with the numerical difference between the mean magnitudes being 8.2–12.9%.
The second cascade extracts detailed information from the images. Interesting defects in this cascade are represented by small gray value regions of pixels, located as an irregular pattern on the image. In our previous research, we used a simple and fast method, which was based on the calculation of average intensity and standard deviation for each sub-region. These parameter values were compared with nominal sample parameter values; if the calculated values were less than 80% of the nominal ones, the sample was rejected. Unfortunately, average intensity and standard deviation do not fully describe image texture when there are a large number of grains. Thus, the GLCM method [
44] was selected as a reliable method for the description of texture in complex images. This method calculates all the transactions between intensities at specified positions, relative to each other in the image. In other words, GLCM identifies how often a pixel with gray-level value
i occurs either horizontally (0°), vertically (90°), diagonally bottom left to top right (−45°) or diagonally top left to bottom right (−135°) with regard to neighboring pixels with value
j. The GLCM matrix dimension depends directly on the number of gray levels; thus, matrix dimensionality is reduced to the specified number of gray tones, due to a significant sensitivity for the size of the objects in the image.
The top-left value in
Figure 5 corresponds to the number of transactions between the same grey levels 0. Thus, the bottom-right value is a result of transactions (3,3). A part of the GLCM matrix has been presented for sub-regions defined in
Figure 4a. In the bottom-right sub-region, transitions (0,0) and (3,3) marked with red ellipse differ from similar transitions in other sub-regions. This observation is correct and in this case, has a direct relation to the real sample image. However, the GLCM method was used for less-distinctive defects. Thus, correct and in-depth information about texture in the image should be extracted from the GLCM matrix. Haralick proposed 14 different textural features for textural characteristics; we selected four, based on previous research: energy (
GE), correlation (
GC), inverse difference moment (
GH), and variance (
σ). The definition is presented in Equation (1):
where
g is a number of gray levels,
P(
i,
j) is a pixel intensity,
µ is a GLCM mean,
σ is a variance of the intensities of all reference pixels in the relationships that contributed to the GLCM.
Energy is the sum of the squares of values in the GLCM, and is high for images with high homogeneity. Correlation measures the linear dependency of grey levels of neighboring pixels. The high local homogeneity (uniformity) of the image is extracted by inverse difference. Finally, the variance describes the contribution of individual pixel intensity in the image. The variance is smaller when the pixel’s similarity to its neighbors is greater. All the features calculated for each of the four sub-regions must be examined by the first cascade rule. If the features are similar in all four sub-regions, the image is processed using the grain detection procedure.
3.3. Grains Detection Algorithm
The main goal of this paper is effective detection and classification of separated grains of copper ore using images of the ground sample. Well-known object detection techniques can identify grains by their stationary features (surface color, shape, texture, and size), dynamic changes in shape and size, and indirect methods such as time taken to submerge in a liquid (large grains take lesser time than small ones). Our research takes into account a few vital requirements—wide range of detected grains fractions in one sample, especially the largest one; and image decision without any tracking, color/gray level and simultaneous shape analysis.
The images acquired using a machine vision system show grains on the entire surface of the material, clearly visible by a camera; many of the separate grains have similar color or texture, despite the HE. Image segmentation is the right way to identify the grains, especially an adaptive method with region growing. Several algorithms have been proposed based on different uniformity criteria in other industries [
45,
46]. This approach, however, has a few main problems: selecting seed points, similarity criteria, correlation of the fixed or adaptive threshold with the similarity criteria.
Our previous research focused on modifying the Otsu method, which takes into account local changes in texture; it produced good results only for non-connected grains. The method is based on a growing region, starting from a point in the segmented region (e.g., grain), and proceeding to add objects that are similar to the seed, until all such points in the image are examined. The uniformity of the grains was determined using similarity criteria, which was calculated for the current region by using the well-known Relative Standard Deviation (RSD). RSD is defined as the ratio of standard deviation to the mean. In the proposed solution, RSD must be calculated each time for the current region and for the current region with a possible added point. If these two values are similar, then the tested point is added to the current grain. Of course, as a consequence, some of the detected grains are of different size. Each grain is described by a set of shape features—aspect ratio (
FA), compactness factor (
Fc), and Heywood circularity factor (
FH)—which carry information about non-typical grains. They are defined as follows:
where
dmin and
dmax are the smallest and largest diameter respectively,
A is a area,
i1 and
i2 are two second moments of the grain around its principal axes,
P is a perimeter.
This study found the grains to have very low aspect ratio, low value of compactness, and a large value of Heywood factor. Next, the selected grain is divided into a few smaller ones, based on the calculated distance maps. In this method, a significant part of the work is done using distance map calculation, wherein more than 15% of the grains are restored. In most cases, the grains overlap or have a common boundary; thus, region growing is performed, producing merged grains.
Figure 6 presents samples of the merged grains after the segmentation method.
Separation of the various, adjacent homogeneous regions, based only on segmentation, is a crucial task. The reasons why segmentation needs to be improved are: to improve the number of parameters that must be selected (even if they are selected automatically), varying localizations of the grains, different grain sizes, different pixels intensity, and number of grains classified into fractions.
The most challenging issue concerning copper ore detection from real images is correct region recognition for different fractions. In
Figure 7a–c, granularity fractions in the range of 0.25–0.75 are presented at different positions, angles, and sizes. Furthermore, three common situations have been identified: a separated, elongated grain (
Figure 7d); oval grain with a complex surface—bright and dark areas (
Figure 7); and an intermediate shape with four connected grains (
Figure 7f).
Based on this, we characterize grains as a non-uniform region, where pixel intensity has a wide range and varies in the region, edges that are not thin, one-pixel edges, and common boundary between grains. Based on human vision, which is sensitive to edges in the scene [
47], previously identified SRG properties have been improved upon, with focus on region growing in combination with boundary information, extracted with the use of edge detection.
Edge-based methods are suitable to detect step changes in the images, especially linear and points features.
Figure 8a presents two grains from the same sample that show a significant difference between boundaries. The one on the left is sharp, the right irregular. The results of processing of the raw images by edge detection operators are extremely sensitive to image noise levels; thus, images are smoothened with Gaussian and detected with Canny (
Figure 8b). The boundary of the left grain is accurate, while the right is irregular. However, the edge separates the region of the right grain from the background and also sets it apart from the left grain.
Although the Canny operator has been used in numerous edge detection applications, it has two main weaknesses. First, sensitivity to image noise, which must be reduced by Gaussian smoothing. Second, false positives edges with many discontinuities, which should be removed by hysteresis thresholding. One of the first steps during edge detection with Canny is calculation of the first derivative of the image. The calculated gradient operator is sensitive to local gray level changes and can be used as an edge detector after comparison with the predefined threshold value. Besides, the non-maximum suppression algorithm is used to remove pixels that are not on the edge. Another method is calculation of the second derivative of the image by the Laplacian of a Gaussian (
Figure 8c) and zero-crossing detection, which improve the detection of irregular edges. Both, first and second derivatives are sensitive to window size.
Based on the discussions regarding
Figure 6,
Figure 7 and
Figure 8, several general problems have been identified, which have been considered in the proposed algorithm. Moreover, the authors of [
48] identified three segmentation errors, especially in region growing: false object boundaries are detected by segmentation without corresponding to the real edge, real edges are not detected by the segmentation, and lack of coincidence between the segmented boundary and edge in the image. Region growing and edge detection is associated with a threshold value, which only considers the intensity of the pixel without any adjacent pixels. Thresholding typically affects the final result of segmentation by adding pixels outside or removing pixels from inside the region of interest. The final segmented region is reduced or enlarged by the extraneous pixels, which is highly inconvenient due to grain size determination. The problem of correct thresholding can be solved by using a global or local method. Global thresholding methods based on difference between classes, e.g., object and background, are significant; thus, threshold depends on the pixel’s properties (such as intensity). A commonly global thresholding method is based on the Otsu [
49] algorithm. Other methods are based on local information for each pixel, such as intensity, variance, or mean in small regions such as 3 × 3 pixels. Local methods are generally sensitive to individual image characteristics. The main local techniques identified include Niblack [
50], Sauvola [
51], Yanowitz, and Bruckstein [
52].
Because these methods have certain negative features, we proposed a combination of local and global information. Our main goal was accurate grain region detection; however, region growing segmentation caused limitations and excessive grain growth when attempting to identify all pixels in a grain. Therefore, we proposed a method that is based on region growing, where the process is carried to the nearest edge, followed by tracking of the edge in a closed area. As a result, all pixels between the recognized edge and the seed point from which the growth began were automatically assigned to the grain region without checking any of the thresholds.
The region-growing segmentation uses the intensity value of the pixel and a homogeneity checking procedure. While this is correct in definition, it does not work correctly in most real applications. There are four problems that should be considered: selection of the seeds (which have simply been taken from the list of pixels with the highest intensity after reducing adjacent pixels), homogeneity measure, edge detection, and size of the local neighborhood. The proposed method consists of a few main steps, as follows:
Input grayscale image I, Canny calculated binary image of edges E
Initialize threshold T regarding the GLCM matrix with Equation (2) and initialize list of the pixels to be searched P and list of the seeds S
N is a region number; K is a seed pixel number, M is a pixel number in a region
Start growing region RN from pixel SK (seed point, k = 1,..) with TR in the horizontal direction, then vertically
Find the edge EN based on calculations (3), (4) and (6); update TR
Track the edge EN with T
Assign all pixels inside the region RN between SK and the last pixel of EN to the current region; update P and S
If EN completed, initialize T regarding to GLCM, starting growing from SK in vertical direction
Repeat 5–7
Finish growth of region RN
Select next seed pixel SK+1 and repeat 4–10 until all the seeds from S are removed
Go to particle features’ calculation and refining process, if required
First, we decided to combine local and global information. The images contain a large number of grains with uniform intensity, localized in a consistent region; and small number of low-intensity pixels localized on the edges. We used Equation (3), which combines information from intensity and derivatives of the image, instead of a simple intensity value of the pixel:
where
q is the number of pixels in a small neighborhood [3×3] or [5×5],
P(
i,
j) is pixel intensity,
f1 is a first derivative and
f2 is a second derivative of image in pixel (
i,
j). The calculated value is then compared at each step with threshold value
T, which is used as a homogeneity measure. At the start, the value is calculated strictly on the basis of GLCM matrix values, obtained during the quality check step. In contrast to other solutions, we updated the
T value only when the growing region has an edge. This part of the process is presented in
Figure 9a–c.
In this case, the method skips single pixels with high intensity, but with the correct calculation of
T can be used for edges visible in
f2 detection. We decided to modify Niblack’s algorithm, which is simple and fast, but unfortunately, the value around the mean of the neighborhood of the pixel varies. We added extra information to the definition of Niblack’s threshold by consideration of Canny edge appearance in the pixel. This reduces a part of the standard deviation. Threshold is calculated as per Equation (4):
where
k is a parameter set to 0.2 for bright objects and −0.2 for dark objects in classic Niblack’s definition,
σ is standard deviation in the neighborhood of the pixel. We used [3×3] or [5×5] neighboring regions to search for edges—the smaller one in the first round and a larger one as the second stage. We modified values for k, instead of proper edge detections:
where
k after the initialization stage depends only on the presence of edges found using an additional, corresponding Canny edge detector. Finally, the edges are detected by using the proposed consistent condition:
If Equation (6) is found to be true, then threshold
T is updated by the value of
TN. Contrary to the hysteresis method, which uses many thresholds to find the edge, we use one threshold
T that strongly closes the area after detection of the edge. Next, this method tracks edges with defined threshold
T in small neighboring regions of [3×3] or [5×5], when the edge is not continuous and a [3×3] edge detection fails (see
Figure 9d). After each edge pixel detection, all the pixels localized in the triangle between seed point, first edge pixel, and the last one are automatically added to the region, without comparison to
T (see
Figure 9e,f). We identified diverging edges, i.e., too close grains. In
Figure 9g, two pixels marked in red meet Equation (6). In this case, the Euclidean Distance
d (Equation (7)) to seed pixel is calculated for both of them, and a pixel with the least distance is marked as an edge pixel
EP (see
Figure 9h):
This calculation ensures compactness of the grain. In
Figure 9i, a sample with added edge pixel and other pixels belonging to the region of the grain is presented.
3.4. Particle Analysis, Refining and Classification
The proposed segmentation procedure localizes grains in the sample image. Grains, as any object, can be described by several first-, second- and third-order features. With regards to our main goal, which is determining grinding quality, two issues are of importance—the percentage distribution of fractions in the sample, and the shape quality of the grain. For this purpose, we use shape features that characterize the shape and compactness of the grain. Recent literature has found numerous shape features, a few of which are particularly useful for our method, namely, aspect ratio (FA), Heywood circularity factor (FH), and compactness factor (Fc). First, each detected grain is tested using FH and Fc, which should not be too low for correct grains; a perfect value for both is 1. A value below 0.5 suggests that the grain is probably heterogeneous; in other words, the grain consists of two or more grains that should be separated. For this reason, we used a method of refining grains, which is based on grain separation using a simple and fast distance map method.
Finally, the grains must be classified as one of the predefined fractions. In most cases, while milling with an electromagnetic mill, we identified 4-6 fractions of different size e.g., 0.2–0.25 mm. This size is only approximate, based on grain diameter. Considering that the vision system is stationary and the distance between the camera and the sample is constant, we used a grain classification procedure that compared grain size and perimeter to the nominal values of the selected fractions. We could have used a more sophisticated classification method with weighted parameters, even without knowledge of grinded fractions; however, our system offers the possibility of comparison to nominal values—extracted using the sieves method. Finally, we are able to quantify the share of each fraction, in addition to extracting the quality of the detected grains in a statistical form.