A Review of the Application of Computer Vision Techniques in Sustainable Engineering of Open Pit Mines

Abstract

Mineral resources are important industrial raw materials and the cornerstone of ensuring industrial production, especially metal ores. With the continuous development and progress of artificial intelligence technology, it is of great significance to apply artificial intelligence technology to mining. Computer vision technology, as a sensor that collects information like a human “eye”, is becoming increasingly important in ensuring mining safety, improving mining continuity, and reducing environmental interference through computer vision methods. In this context, this paper focuses on general problems of metal mineral resources, the sustainability of exploration, drilling and blasting, transport, personnel safety, and security. It describes the latest progress of computer vision technology in each link and summarizes and looks forward to the key technical methods. It also summarizes and looks ahead to the key technical methods in each area. The research results show that the application of computer-vision-related technologies in related links not only greatly improves production efficiency but also reduces environmental interference and the probability of production safety accidents, effectively ensuring sustainable mining. In the future, to achieve unmanned mining throughout the entire process, it will be necessary to combine computer vision technology with other specialties such as intelligent control and intelligent perception to achieve a technological breakthrough throughout the entire process.

1. Introduction

Mineral resources are known as the “food” of industry in industrial production and are the most basic and important guarantee of industrial development. At the same time, with the development of society, there are increasingly high demands on the safety, economy, and sustainability of the mining process. The mining industry is also gradually developing in the direction of automation [1], unmanned operation [2], and intelligence [3]. There are also increasingly high demands for continuous operation of the production process, as this is a key factor in ensuring production efficiency and safety.

Artificial intelligence (AI) technology plays a crucial role in this regard [4]. Through real-time data analysis and predictive maintenance of sensors, people can effectively monitor and optimize the operating status of mining equipment and reduce equipment failures and downtime, thereby improving the stability and continuity of production, which is conducive to continuous operation of the enterprise. Computer vision (CV) technology, as a part of AI, is not only a key part of this process [5] but also plays an important role due to its intuitive nature. CV technology can be used throughout the entire life cycle of mining, not only effectively reducing the cost of the mine’s production cycle but also significantly improving worker safety and early warning of equipment failures during the production process [6]. As the process of unmanned mining continues to deepen, the number and impact of safety incidents will have the opportunity to be reduced and minimized, which will also help reduce resource waste and environmental pollution and contribute to sustainable development [7].

The development of CV technology was relatively slow until the emergence of neural network technology. However, in recent years, with the continuous improvement of image acquisition quality [8] and the continuous reduction of the cost of industrial cameras, CV technology has also been promoted in the industrial field [9], especially in the mining industry. In order to ensure the safe and stable operation of CV algorithms [10], in recent years, extensive research has been conducted in this field. By processing images captured by industrial cameras through advanced algorithms and providing real-time feedback on potential issues, the efficiency and reliability of industrial production have been significantly enhanced [11]. Early algorithms were primarily categorized into two groups: one based on neural networks such as R-CNN [12] and its variants, and the other based on networks derived from architectures like ResNet [13], VGG [14] (Visual Geometry Group), Inception [15], and DenseNet [16]. Additionally, there were single-stage algorithms, with methods like YOLO [17,18] (You Only Look Once) and SSD [19] (Single Shot MultiBox Detector) leading the way. To address the challenges posed by imbalanced datasets, researchers have explored innovative solutions [20], including the adoption of transfer learning [21] and the use of GANs (generative adversarial networks) [22], both of which have substantially improved data processing capabilities. With ongoing advancements in technology, the integration of attention mechanisms into industrial applications has gained increasing attention [23].

This paper collects a number of papers in the current field, including 13 papers in the exploration session, 13 papers in the drilling and blasting session, 18 papers in the belt transportation session, and 7 papers in the personnel safety and security session. Classified according to the realization tasks, they can be divided into 10 types of tasks, including 16 target detection tasks, 8 image segmentation tasks, 6 anomaly detection tasks, and 5 granularity analysis tasks; the number and percentage of other types of tasks are shown in

Table 1. Table on the number of individual mandate categories.

Main Category	Quantity
Object Detection	16
Image Segmentation	8
Anomaly Detection	6
Granularity Analysis	5
Image Generation	4
Object Localization	3
Model Lightweighting	3
Image Classification	2
Graph Data Analysis	2
Image Stitching	2

and Figure 1, respectively. According to the algorithmic perspective, there are 41 categories of algorithmic innovations, of which the algorithmic classification is mainly divided into 13 categories, including 5 traditional image processing algorithms and 35 deep learning algorithms. Among the deep learning algorithms, there are 17 articles using convolutional neural networks (CNNs) and their variant algorithms, 4 articles using graphical neural networks (GNNs) and their variant algorithms, and 3 articles using target detection and segmentation networks; the number and percentage of the other algorithms are shown in

Table 2. Table of numbers of various algorithmic categories.

Category	Quantity
Convolutional Neural Networks (CNN) and Variants	17
Traditional Image Processing Algorithms	5
Graph Neural Networks (GNN) and Variants	4
Target Detection and Segmentation Networks	3
U-Net and Improvements	2
Support Vector Machines (SVM) and Optimizations	2
Autoencoders (Autoencoder)	1
Transfer Learning	1
Time Series Analysis Algorithms	1
Loss Function Optimization	1
Lightweight Network Design	1
Optimization Algorithms Combined with Neural Networks	1
Image Enhancement Algorithms	1

and Figure 2, respectively.

This paper focuses on the four aspects of mineral exploration, safety production, transportation, and post-mining filling and introduces the sustainable development and application of CV technology in these aspects of mining engineering. Section 2.1 mainly introduces the research status of combining remote sensing images with CV technology to find potential ore mining sites; Section 2.2 describes the current state of research on computerized CV techniques in drill and blast; Section 2.3 mainly introduces the role of CV in the transportation process; and Section 2.4 mainly introduces the role of CV technology in ensuring the safety of workers. Finally, Section 3 summarizes the current development status and puts forward a vision for the future development of CV in the mining field.

2. Current Research Status of CV in Various Aspects of Mining

Open-pit mining is a mining method suitable for ore bodies close to the surface. Its operating process involves multiple links, and continuous operation greatly optimizes production costs and increases profits. At the same time, sustainable development is also a new requirement that the times have placed on the mining industry [24].

Before mining in a mining area, the scale, grade, and mining scale of the ore body need to be assessed through geological exploration and drilling, and environmental protection planning needs to be carried out in combination with sustainability requirements [25]. Mining operations begin with drilling operations, using equipment such as drills to accurately drill holes in the ore layer to create conditions for subsequent blasting and mining. Then, through scientifically designed drilling locations and the rational use of explosives, blasting can efficiently split the rock, facilitating subsequent excavation and transportation. After blasting, the ore is loaded by forklifts or excavators and transported quickly and efficiently to the crushing plant or processing facility by belt conveyors and other transport equipment. This continuous operation not only improves production efficiency but also reduces energy consumption during transportation, contributing to energy conservation and emission reduction [26]. The entire open-pit mining process is shown in Figure 3.

Throughout the mining process, safety in production is of paramount importance, especially the safety of personnel. It can be said that the core element of continuous operation in a mine is the absence of production safety accidents. Therefore, strict mine monitoring and safety management must be implemented. Modern technology ensures the stability of mine production, the good operation of equipment, and the safety of personnel. It guarantees continuous production and sustainable development of the mine [27].

2.1. Current Research Status of CV in Exploration

The exploration process is a key step in determining the location, size, quality, and mining value of mineral resources. It includes various methods such as geological surveys, geophysical exploration, geochemical exploration, and drilling [28]. Geological surveys mainly infer the distribution of underground mineral resources by observing and analyzing surface rocks, minerals, and geological structures [29]; geophysical exploration uses changes in physical fields such as earthquakes, gravity, and magnetism to detect underground structures [26]; and geochemical exploration analyzes the chemical composition of soil, water, and other samples to find chemical anomalies in minerals [30].

After exploration, a mining prospect map is drawn. Mineral prospectivity mapping (MPM) is a kind of chart that shows information such as the distribution of mineral resources, ore body shape, and ore grade. It provides an important reference for the design and planning of mining projects and helps engineers formulate reasonable mining plans and resource utilization strategies [31,32]. Through CV-related technologies, data preprocessing and data enhancement can be carried out effectively, and on this basis, prediction and reasoning can be carried out, and the trend of mineral resources can be judged according to the analysis of earth data [33]. In recent years, as the amount of high-quality known mineral resources has decreased, leading to rising exploration costs, and remote sensing technology has gradually matured, the method of preliminary screening through a deep learning network and then conducting field surveys can effectively reduce costs while also improving the efficiency and accuracy of mining. It reduces costs while also improving the efficiency of continuous operations, reduces interference with the environment, and protects the environment to the greatest extent possible [34].

Early scholars mainly explored by correlation. Xiong et al. [35] used an autoencoder network to train remote sensing images. By stacking the autoencoder network with CRBMs (convolutional restricted Boltzmann machines), they found that geochemical anomalies and the location of Fe deposits are strongly spatially correlated, as shown in Figure 4. For geochemical abnormality maps [35], Liu et al. used a traditional CNN algorithm to improve the accuracy of the judgment to 93% and at the same time provided a judgment that the probability of the existence of ore bodies in some areas is very high [36]. These traditional deep learning algorithms have created a precedent for the combination of CV and mineral resource exploration, and the potential of Fe deposits is discovered through regions with high reconstruction error.

With the development of CV-related technologies, some scholars have narrowed down the scope of exploration by comparing different CV algorithms and analyzing existing data, analyzing the advantages and disadvantages of areas with a high probability of mineral deposits, in order to find the right algorithm. Some scholars have obtained better analysis results by obtaining higher-quality data and processing multidimensional data.

Among them, the following are some of the different algorithms analyzed: Wu et al. used transfer learning and a twin neural network to increase the learning and generalization capabilities of the model and achieved an accuracy rate of 85% in tests conducted in the Junggar Basin in northern Xinjiang [37]. Zuo et al. [38] compared more algorithms and used three different neural network architectures—CNN, GCN, and GAT—to predict the potential trend of mineral deposits. Among them, GNN had the best overall prediction accuracy, as shown in Figure 5. The article also points out that graph deep learning algorithms have an advantage in trend prediction and will have better development prospects in the future with the incorporation of attention mechanisms. Shi et al. [39] used graph neural networks to take into account the characteristics of different ore regions and geological conditions. By modifying the model, they verified that heterogeneous graphs can better simulate mineral prospect maps and improve the accuracy of prediction results compared with traditional neural networks [39]. Some other scholars, such as Xiao et al., used hyperspectral images of iron ore mines to provide a fast, accurate, and low-cost monitoring method for iron ore mines based on a 3D convolutional neural network and a deep residual network, with an overall accuracy of 99.62% [40].

2.2. Current Status of Research on CV in Drilling and Blasting

Drilling and blasting are two key sequential steps in mining engineering, and there is much room for improvement in terms of engineering sustainability. Drilling refers to the use of a drilling rig to drill holes in the rock in order to insert blasting materials. The location and diameter of the holes will greatly affect the blasting effect and safety. Blasting involves placing explosives in the drilled holes and detonating them to break up the rock, thereby creating space for mining the ore while controlling the extent of the breakage and reducing the impact on the surrounding environment [41,42]. With the continuous maturity of robotics and the continuous advancement of unmanned mining, there has been increasing research on using CV to determine the status of the drilled holes [43]. CV can be used to confirm the quality and location of holes after drilling and to conduct real-time inspections of blasting effects after blasting [44], which helps to dynamically optimize the entire continuous operation process, is conducive to ensuring the continuous operation of mining activities, and can effectively dynamically optimize and adjust the drilling position and number of holes based on the blasting effect, reducing the number of secondary blasts and the impact of explosions on the environment, which is conducive to promoting the sustainable development of the mining industry [45].

2.2.1. Current Research Status of CV in the Drilling Process

The use of CV technology in the drilling process is a relatively new and novel technology [46]. At present, In the drilling process, the application of CV technology is relatively limited. It is mainly used in the drilling process without human intervention. The drilling position is calibrated by machine vision to improve the blasting effect [47,48]. In the blasting process, the main method is to analyze the particle size of the broken rock after the explosion [49] or to analyze the edge of the ore by image method, and further analyze the blasting effect [50].

In the drilling process, Valencia used a drone to take pictures of the area after drilling and then used CV to analyze the data. The conclusion was that the combination of CNN and SVM is an effective method for identifying blast holes, as shown in Figure 6 [47]. Guo et al., using a method based on YOLOV5, identified blast holes under various lighting conditions, such as low-light and misty conditions, with an accuracy rate of over 80% [48]. These studies have explored the possibility of further unmanned mining.

2.2.2. Current Research Status of CV in Blasting

In the blasting process, many scholars use CV to analyze the particle size distribution of the broken rock after blasting, in order to facilitate research on blasting fragmentation prediction. Some scholars also use CV to evaluate the blasting effect after blasting.

The following are some of the studies conducted by scholars on the particle size distribution of crushed rock after blasting. Jin et al. used the U-CARFnet algorithm to analyze the particle size of ore fragments after blasting, with an accuracy rate of 97.11% and a segmentation error of only 5.46% [49]. Yang et al. improved the U-Net algorithm to improve the particle segmentation performance of the stockpile, improving the overall accuracy by 1.5% [50]. The following are some studies by scholars on the evaluation of blasting effects after blasting. He et al. [51] used a watershed algorithm based on genetic algorithms and OTSd algorithms to determine the blasting effect based on the fragmentation of the blast. Figure 7 shows that the computer algorithm can segment the image of the rock after blasting and evaluate the blasting effect based on the segmentation effect. Bahraini et al. used an improved YOLOV8 algorithm to evaluate the results after blasting through an intelligent stereo vision method, thereby providing theoretical support for the optimization of blasting parameters [52]. Vu et al. deployed the deep learning model Mask R-CNN, achieving a boundary segmentation coefficient of 82% and 93%, laying the technical foundation for the automatic measurement of blasting fragmentation in open-pit mines [53].

2.3. Current Status of Research on CV in the Transportation Process

After the ore is mined, it needs to be transported to other places for subsequent processing. The efficiency of transportation has a significant impact on the productivity of the mine. The efficient operation of the mine also contributes to the realization of the high-quality sustainable development transformation of the mining industry [54]. According to the different stages of transportation, transportation can be divided into horizontal transportation in tunnels and vertical transportation, which is pulled up from the ground through shafts [55]. If problems such as belt tears and belt deflections during transport occur, production will need to be stopped for repairs [56], which greatly affects the continuity of operations. Monitoring using CV can not only effectively reduce personnel costs but also detect potential problems in real time, improve production safety, and avoid the adverse effects of production safety accidents on the environment and continuous production [57].

2.3.1. Detection of Belt Damage During Transportation

With the development of technology, more and more scholars have conducted more detailed research. If belt damage is encountered in production, it is necessary to stop work for repair, which seriously affects the sustainability of production. The earlier the belt problem is detected, the less impact it will have on other equipment. In terms of belt damage detection, it mainly includes the extension of monitoring links, algorithm innovation, and application in special environments. In terms of link extension, the method of image stitching is mainly used; some scholars have also explored the problems of poor lighting and harsh environments. On the one hand, this improves the accuracy of recognition; on the other hand, the model is also lightweight, which improves the detection speed.

The following is an introduction to image stitching as used by scholars. Gao uses the IOI algorithm to process the images obtained by the camera by fusing the segmented images and completes online detection with an accuracy rate of 97.67% and an average splicing time of less than 500 ms. A schematic diagram of the equipment design is shown in Figure 8 [58]. Liu et al. [59] designed a method that uses a temporal convolution network (TCN) to extract the temporal features of multiple consecutive video frames. The belt damage recognition method that incorporates the spatio-temporal features of the image is more than 20% more accurate than the method that only extracts the spatial features of the image. Zhang et al. proposed a centernet object detection algorithm based on centroid detection, which achieved an accuracy of 97% on the conveyor belt damage dataset and a test speed of 32.4 FPS [60]. The following is an introduction to how scholars have improved detection speed through model lightweighting. Wang et al. used a dark channel algorithm to eliminate the interference of fog on belt images, which improved the recognition accuracy to 98.4% in harsh environments and reduced the recognition delay to 52.3 ms [61].

2.3.2. Belt Offset During Transportation

During the transportation process, the belt may shift for a variety of reasons. When the offset is significant, the transported object may dislodge from the belt, which has a substantial impact on the sustainability of the transportation process. This problem has been the subject of extensive research by numerous scholars. Some scholars have employed bilateral positioning to ascertain whether the belt has shifted, while others have modified network algorithms to enhance the recognition accuracy of CV methods, thereby facilitating deviation detection.

Here are some studies by scholars who obtained data from both sides to determine whether the belt is offset. Wu et al. proposed a double-baseline positioning judgment method using Canny edge detection and Hough transform algorithms, which can accurately detect belt deviation up to 99.45% [62]. Zhang et al. constructed a conveyor belt deviation detection system using a line laser generator and the Labview platform to construct a conveyor belt deviation detection system [63], and Zhang et al. used a dual-branch lightweight network, FastBeltNet, to achieve an accuracy of 80.49% and a processing speed of 99.89 frames/second, effectively balancing the accuracy and speed of industrial production [64]. The following are some scholars’ research on belt offset detection by improving CV methods. Zhang first replaced Darknet53 with EfficientNet, and the recognition and detection speed of YOLOV3 reached 42 frames/second, with average accuracy reaching 97.26% [65]. The experimental schematic is shown in Figure 9 [65]. Next, he used the improved network YOLOV5 to achieve a balance between detection accuracy and speed, with a detection accuracy up to 90%, and a detection speed of up to 67 frames/second [66]. Zhao et al. [67] integrated an enhanced squeeze and excitation (ESE) module into C2f to promote feature extraction of rollers and belts, and achieved a prediction accuracy of 98.1% and a detection speed of 46 frames/second by enhancing the bidirectional feature fusion and attention mechanism. The detection scene is shown in Figure 10.

2.3.3. Detection of Foreign Objects During Transportation

The presence of foreign bodies also poses a serious threat to the service life of the conveyor belt and the safe transportation of goods. In the event of an accident, it will seriously affect the continuity of production. However, CV methods based on deep learning can effectively identify the situation and improve transportation safety.

Here are some scholars who have used CV methods. Zhang achieved a maximum detection accuracy of 93.73% and a detection speed of 70.1 frames/second by modifying YOLOV4 [68]. Luo et al. achieved a detection accuracy of 94.9% and a maximum prediction speed of 90.2 FPS in harsh environments using an improved YOLOv5 model [69]. Dai et al. introduced a multi-layer perceptron (MLP) network optimized by the gray wolf algorithm to identify large targets, achieving a laboratory recognition accuracy of 98.8% and an on-site test accuracy of over 95% [70]. Zhang et al. achieved integrated detection of various tasks by grafting a segmentation network into the object detection network YOLOv5, with a detection accuracy of up to 97% and a detection speed of up to 90 frames/second. The effect map of the on-site measurement is shown in Figure 11 [71].

2.4. Application of CV in Ensuring Personnel Safety During Mining

Personnel safety is of the utmost importance in mines. At the same time, monitoring personnel can also greatly ensure production safety. If a personnel safety accident occurs, it will result in a shutdown of production for at least half a year and up to a year, which has a significant negative impact on the continuity of the entire production process [72]. In the past, safety officers were needed to constantly patrol and check the working status of workers and the wearing of relevant safety equipment. However, with the development of computer technology, CV can largely replace the work of safety officers, while improving the quality of monitoring and ensuring uninterrupted monitoring, which can greatly contribute to safe production.

Some scholars have found that the probability of an accident is directly proportional to the fatigue level, so they have adopted a method that combines CV algorithms with traditional fatigue detection methods (FMD) to monitor workers [73]. For example, Chen et al. improved the accuracy of the model by 18.68% using a least squares support vector machine (LSSVM) optimized by the marine predator algorithm (MPA) [74].

When working in underground mines, the lighting is not good most of the time, and due to the dust, visibility is often poor. It is particularly important to use image enhancement methods to improve the accuracy of recognizing human behavior [75]. Xu et al. used image enhancement methods on the dataset, which improved recognition by 68.26% compared to the dark primary color prior dehazing algorithm, and in practical applications, it improved by 31.1% compared to the dark primary color prior dehazing algorithm, which greatly enhanced the recognition of people in dim environments [76].

In work safety accidents, a large proportion of accidents are caused by workers not wearing or incorrectly wearing safety helmets. Therefore, if the wearing of safety helmets by workers can be effectively monitored, the occurrence of such accidents can be effectively reduced [77]. For example, Yao et al. proposed an improved YOLOv5s model (AMCFF-YOLOv5s) that significantly improves the accuracy of detecting the status of miners in mine hoisting cages from 89.2% to 97.6% by introducing a k-means clustering algorithm, a BiFPN feature fusion network, a CBAM attention module, and a CIoU loss function [78]. It can effectively identify whether miners are wearing helmets and whether they have fallen [78].

3. Conclusions

3.1. Summary

The role of CV technology in the context of unmanned and intelligent mining is of paramount importance. Its efficacy in ensuring the continuity of operations within the mine is also a key consideration. The technology has been shown to reduce production costs and enhance safety, which is considered to be the most significant benefit.

This paper sorts out the role played by different CV technologies in different aspects of mining from four major aspects. In terms of mineral resource exploration, CV technology can narrow the exploration scope by assisting in drawing mineral prospect maps. In terms of drilling and blasting, CV technology can confirm the quality and location of the drill holes and the blasting effect so as to dynamically optimize them, reduce the number of secondary drilling and blasting operations, improve the continuity of operations, and reduce the impact on the environment. In belt transportation, it can monitor the belt and transported ore from multiple directions and angles, detect potential problems in advance, and reduce the risk of safety accidents causing work stoppages. In terms of personnel safety detection, CV technology can protect the lives of workers by integrating multiple angles with traditional fatigue detection.

3.2. Future Prospects

With the continuous advancement of machine vision technology, future unmanned mining will achieve a higher level of automation and intelligence, promoting the sustainable development of the mining industry. Detecting and classifying targets based on machine vision technology is currently the most important technology for safeguarding personnel safety, and it is also the basis for realizing unmanned mining in the field in the future. Through advanced image recognition and analysis capabilities in open-pit mining, assisted by robots with intelligent control, it can improve the efficiency and precision of resource extraction and reduce the damage to the environment, while real-time monitoring and optimization of all aspects of the mining process will not only improve the efficiency and safety of mining but also reduce the incidence of accidents by reducing the exposure of personnel in hazardous environments.

Therefore, we believe that in the future development of the current field, we should focus on thinking about how to use the data from computer vision as fundamentalized data and combine it with the corresponding control technology and equipment, so as to predict the potential veins of mines, automatically optimize the design of blasting, intervene in real time for the problems occurring in the belt, and so on. In addition, combined with high-speed communication technologies such as 5G, machine vision will support more efficient remote monitoring and decision support, driving the mining industry in a more sustainable and environmentally friendly direction.