A Fast and High-Accuracy Foreign Object Detection Method for Belt Conveyor Coal Flow Images with Target Occlusion
Abstract
:1. Introduction
2. Image Acquisition of Multi-Layer Occluded Foreign Objects
2.1. Experimental Platform and Parameters
2.2. Image Data Construction
3. Detection Model of Multi-Layer Occluded Foreign Objects
3.1. YOLOv5s Network Model
3.2. Soft Non-Maximum Suppression
Algorithm 1: Soft–NMS (Non-maximum suppression) |
3.3. SimOTA Label Allocation Optimization
3.4. Improvement of Loss Function
3.4.1. Slide Loss
3.4.2. Inner–SIoU
3.5. Model Pruning
4. Experimental Results and Analysis
4.1. Model Evaluation Indexes
4.2. Ablation Experiment of Accuracy Improvement Model
4.3. Comparative Experiment of Model Compression Method
4.4. Comparative Experiment of Different Detection Algorithms
4.5. Visual Analysis of Detection Results in Different Scenarios
5. Conclusions
- (1)
- For the coal flow foreign object occluded image dataset’s construction, the target occlusion degree, occlusion type, and dense degree are comprehensively considered, and the synthetic occlusion is used. The synthetic occluded images and the real occluded images together constitute the coal flow foreign object occluded image dataset in a low-illumination and dust fog environment.
- (2)
- The YOLOv5 algorithm is improved by the Soft–NMS, SimOTA, Slide Loss, and Inner–SIoU modules. The problem of missed and false detection caused by dense and occluded targets is solved, and the detection accuracy of the model used for small and occluded targets is improved. The mAP0.5 of the accuracy improvement model reaches the detection level of YOLOv8s, which is higher than YOLOv5n, YOLOv5s, YOLOv5m, YOLOv7, YOLOv7–X, and YOLOv8n.
- (3)
- The accuracy improvement model is pruned through the Group–Taylor method. After pruning, the mAP0.5 of the model is greater than 90%, and its model parameters, FLOPs, and size are also greatly reduced. The detection speed is faster on the GT 1050Ti, GT 1030, and Intel (R) Core (TM) i5–11400F processors, which meets the real-time processing speed requirements.
- (4)
- The mAP0.5 of the proposed method on the self-built multi-layer occluded coal flow foreign object dataset is 91.30%, and the mAP0.5 is lower than YOLOv5n, YOLOv5s, YOLOv5m, YOLOv7, YOLOv7–X, YOLOv8n, YOLOv8s, and YOLOv8m. However, both the model’s lightweight level and detection speed are superior to the above, and the detection effect of most occluded targets is good in various scenarios.
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Chen, X.J. Current status and development trend of intelligent technology of underground coal mine transportation system. J. Mine Autom. 2022, 48, 6–14+35. [Google Scholar] [CrossRef]
- Wang, H.J.; Wang, H.L. Status and prospect of intelligent key technologies of belt conveyor. Coal Sci. Technol. 2022, 50, 225–239. [Google Scholar] [CrossRef]
- Hao, S.; Zhang, X.; Ma, X.; Sun, S.Y.; Wen, H.; Wang, J.L. Foreign object detection in coal mine conveyor belt based on CBAM-YOLOv5. J. China Coal Soc. 2022, 47, 4147–4156. [Google Scholar] [CrossRef]
- Ge, S.R.; Hao, S.Q.; Zhang, S.H.; Zhang, X.F.; Zhang, L.; Wang, S.B.; Wang, Z.B.; Bao, J.S.; Yang, X.L.; Yang, J.J. Status of intelligent coal mining technology and potential key technologies in China. Coal Sci. Technol. 2020, 48, 28–46. [Google Scholar] [CrossRef]
- Wu, S.P. Rescarch on Detcction Method of Foreign Object on Coal Conveyor Belt Based on Computer Vision. Master’s Thesis, China University of Mining and Technology, Xuzhou, China, 2019; pp. 1–60. [Google Scholar]
- Wang, J.C.; Li, L.H.; Yang, S.L. Experimental study on gray and texture features extraction of coal and gangue image under different illuminance. J. China Coal Soc. 2018, 43, 3051–3061. [Google Scholar] [CrossRef]
- Li, B.; Wang, X.W.; Pang, S.Z.; Gao, X.Y.; Wang, L.Y.; Ding, E.F.; Bao, Q.B. Image characteristics analysis and experimental study of coal and gangue. Coal Sci. Technol. 2022, 50, 236–246. [Google Scholar] [CrossRef]
- Cao, X.G.; Hao, P.Y.; Wang, P.; Liu, S.Y.; Wu, X.D.; Huo, X.Q. Research on method of acquiring high quality coal gangue images under multi-factor illumination condition. Coal Sci. Technol. 2023, 51, 455–463. [Google Scholar] [CrossRef]
- Li, M.; Yang, M.L.; Liu, C.Y.; He, X.L.; Duan, Y. Illuminance adjustment method for image-based coal and gangue separation. J. China Coal Soc. 2021, 46, 1149–1158. [Google Scholar] [CrossRef]
- Mao, Q.H.; Li, S.K.; Hu, X.; Xue, X.S. Coal mine belt conveyor foreign objects recognition method of improved YOLOv5 algorithm with defogging and deblurring. Energies 2022, 15, 9504. [Google Scholar] [CrossRef]
- Fan, H.W.; Liu, J.P.; Cao, X.G.; Zhang, C.; Zhang, X.H.; Li, M.; Ma, H.W.; Mao, Q.H. Multi-scale target intelligent detection method for coal, foreign object and early damage of conveyor belt surface under low illumination and dust fog. J. China Coal Soc. 2024, 1–12. [Google Scholar] [CrossRef]
- Fan, H.W.; Zhang, C.; Cao, X.G.; Liu, J.P.; Zhang, X.H.; Zhao, H. An image dust removal and enhancement method in low illumination environment based on dark-bright channel segmentation and fusion. J. China Coal Soc. 2024, 49, 2167–2178. [Google Scholar] [CrossRef]
- Gao, Y.S.; Zhang, B.Q.; Lang, L.Y. Coal and gangue recognition technology and implementation based on deep learning. Coal Sci. Technol. 2021, 49, 202–208. [Google Scholar] [CrossRef]
- Xu, Z.Q.; Lv, Z.Q.; Wang, W.D.; Zhang, K.H.; Lv, H.M. Machine vision recognition method and optimization for intelligent separation of coal and gangue. J. China Coal Soc. 2020, 45, 2207–2216. [Google Scholar] [CrossRef]
- Guo, Y.C.; Wang, X.; He, L.; Liu, P.Z. Research on coal and gangue recognition method based on TW-RN optimized CNN. Coal Sci. Technol. 2022, 50, 228–236. [Google Scholar]
- Wu, S.P.; Ding, E.J.; Yu, X. Foreign body identification of belt based on improved FPN. Saf. Coal Mines 2019, 50, 127–130. [Google Scholar] [CrossRef]
- Lv, Z.Q. Research on Foreign Body Image Recognition of Coal Mine Belt Transport under Complex Environment. Master’s Thesis, China University of Mining and Technology, Xuzhou, China, 2020; pp. 1–60. [Google Scholar]
- Hu, J.H.; Gao, Y.; Zhang, H.J.; Jin, B.Q. Research on the identification method of non-coal foreign object of belt conveyor based on deep learning. Ind. Mine Autom. 2021, 47, 57–62, 90. [Google Scholar] [CrossRef]
- Ren, Z.L.; Zhu, Y.C. Research on foreign objects recognition of coal mine belt transportation with improved CenterNet algorithm. Control Eng. China 2023, 30, 703–711. [Google Scholar] [CrossRef]
- Wang, Y.B.; Wang, Y.J.; Dang, L.F. Video detection of foreign objects on the surface of belt conveyor underground coal mine 12 based on improved SSD. J. Ambient Intell. Humaniz. Comput. 2023, 14, 5507–5516. [Google Scholar] [CrossRef]
- Chen, D.Q.; Xu, J.Y.; Kou, Q.Q.; Zhang, H.W.; Han, C.G.; Yu, B.; Qian, J.S. Lightweight network based on residual information for foreign body classification on coal conveyor belt. J. China Coal Soc. 2022, 47, 1361–1369. [Google Scholar] [CrossRef]
- Chen, Z. Research on Foreign Objects Detection of Mine Belt Conveyor Based on Deep Learning. Ph.D. Thesis, China University of Mining and Technology, Xuzhou, China, 2023; pp. 1–59. [Google Scholar]
- Ding, Z.H.; Chen, G.D.; Wang, Z.; Chi, W.Z.; Wang, Z.H.; Fan, Y.L. A real-time multilevel fusion recognition system for coal and gangue based on near-infrared sensing. IEEE Access 2020, 8, 178722–178732. [Google Scholar] [CrossRef]
- Li, D.Y.; Ren, H.W.; Wang, G.F.; Wang, S.; Wang, W.S.; Du, M. Coal gangue detection and recognition method based on multiscale fusion lightweight network SMS-YOLOv3. Energy Sci. Eng. 2023, 11, 1783–1797. [Google Scholar] [CrossRef]
- Zhang, M.C.; Cao, Y.H.; Jiang, K.; Li, M.X.; Liu, L.X.; Yu, Y.; Zhou, M.S.; Zhang, Y. Proactive measures to prevent conveyor belt Failures: Deep Learning-based faster foreign object detection. Eng. Fail. Anal. 2022, 141, 106653. [Google Scholar] [CrossRef]
- Chen, Y.M.; Sun, X.; Xu, L.; Ma, S.C.; Li, J.; Pang, Y.S.; Chen, G. Application of YOLOv4 algorithm for foreign object detection on a belt conveyor in a low-illumination environment. Sensors 2022, 22, 6851. [Google Scholar] [CrossRef] [PubMed]
- Yao, R.B.; Qi, P.; Hua, D.Z.; Zhang, X.; Lu, H.; Liu, X.H. A Foreign Object Detection Method for Belt Conveyors Based on an Improved YOLOX Model. Technologies 2023, 11, 114. [Google Scholar] [CrossRef]
- Bodla, N.; Singh, B.; Chellappa, R.; Davis, L.S. Soft-NMS improving object detection with one line of code. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 5561–5569. [Google Scholar] [CrossRef]
- Ge, Z.; Liu, S.T.; Li, Z.M.; Yoshie, Q.; Sun, J. Ota: Optimal transport assignment for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 303–312. [Google Scholar] [CrossRef]
- Ge, Z.; Liu, S.T.; Wang, F.; Li, Z.M.; Sun, J. Yolox: Exceeding yolo series in 2021. arXiv 2021, arXiv:2107.08430. [Google Scholar] [CrossRef]
- Yu, Z.P.; Huang, H.B.; Chen, W.J.; Su, Y.X.; Liu, Y.H.; Wang, X.Y. Yolo-facev2: A scale and occlusion aware face detector. arXiv 2022, arXiv:2208.02019. [Google Scholar] [CrossRef]
- Zhang, H.; Xu, C.; Zhang, S.J. Inner-iou: More effective intersection over union loss with auxiliary bounding box. arXiv 2023, arXiv:2311.02877. [Google Scholar] [CrossRef]
- Molchanov, P.; Mallya, A.; Tyree, S.; Frosio, I.; Kautz, J. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 11264–11272. [Google Scholar] [CrossRef]
- Torch-Pruning. Available online: https://github.com/VainF/Torch-Pruning (accessed on 8 April 2024).
- Lee, J.; Park, S.; Mo, S.W.; Ahn, S.S.; Shin, J.W. Layer-adaptive sparsity for the magnitude-based pruning. arXiv 2021, arXiv:2010.07611. [Google Scholar] [CrossRef]
- Li, H.; Kadav, A.; Durdanovic, I.; Samet, H.; Graf, H.P. Pruning filters for efficient convnets. arXiv 2017, arXiv:1608.08710. [Google Scholar] [CrossRef]
- Yann, L.C.; John, S.D.; Sara, A.S. Optimal brain damage. Adv. Neural Inf. Process. Syst. 1989, 2, 598–605. [Google Scholar]
Device | Parameters |
---|---|
Belt conveyor | Belt length: 11 m, belt width: 0.8 m, motor frequency range: 0~50 Hz |
Belt speed control computer | Operating system: Windows XP, MotorControl software 1.0 |
Industry camera | Hikvision MV–CA016–10UC–USB3.0 area array camera |
Shot | HFO420M, focal length: 4 mm |
Camera controller | Operating system: Windows 10, Hikvision client software MVS 4.3.1 |
Light source | LED strip lamp: 50 cm long, white light source, manually adjusted |
Illuminometer | 31/2-bit LCD display, measuring range: 0~200,000 Lux, manually adjusted |
Platform | Parameters |
---|---|
Operating system | Windows 10 |
GPU | NVIDIA GeForce RTX 4080, Memory size: 16 GB |
CPU | Intel(R) Core(TM)i7–13700KF 3.40GHz |
Deep learning framework | Pytorch 2.0.0, CUDA 11.8.0 |
Platform | Parameters |
---|---|
GPU0 | NVIDIA GeForce RTX 4080, Pytorch 2.0.0, CUDA 11.8.0 |
GPU1 | GTX 1050Ti, Pytorch 1.10.1, CUDA 10.2.89 |
GPU2 | GT 1030, Pytorch 1.10.1, CUDA 12.2.89 |
CPU3 | Intel(R) Core(TM)i5–11400F 2.60 GHz, Pytorch 1.10.1 |
Illumination (Lux) | Real Occlusion Case | Synthetic Occlusion Case |
---|---|---|
80 | ||
80–V5 | ||
185 | ||
280 |
Real Occlusion Case | Synthetic Occlusion Case | |
---|---|---|
Blur | ||
Dust fog | ||
Dust fog–V3 |
Model | Soft–NMS | SimOTA | Slide Loss | Inner–SIoU | mAP0.5/% | mAP0.75/% | mAP0.5:0.95/% |
---|---|---|---|---|---|---|---|
1 | 93.9 | 85.2 | 74.0 | ||||
2 | √ | 94.2 | 88.7 | 77.5 | |||
3 | √ | √ | 95.3 | 90.6 | 79.6 | ||
4 | √ | √ | √ | 95.9 | 91.1 | 79.5 | |
5 | √ | √ | √ | √ | 96.0 | 91.4 | 79.8 |
Pruning Method | mAP0.5/% | mAP0.75/% | mAP0.5:0.95/% | Parameters /105 | FLOPs /109 | Model Size /MB | Compression Degree |
---|---|---|---|---|---|---|---|
Model 5 | 96.00 | 91.40 | 79.80 | 70.26 | 15.80 | 14.40 | 1.00 |
+Random [34] | 90.10 | 75.60 | 64.50 | 4.87 | 1.20 | 1.30 | 14.34 |
+Lamp [35] | 90.30 | 75.50 | 64.20 | 4.87 | 1.20 | 1.30 | 14.34 |
+L1 [36] | 90.00 | 75.90 | 64.60 | 4.20 | 1.10 | 1.20 | 16.74 |
+Group–Hessian [37] | 91.30 | 76.60 | 65.30 | 4.20 | 1.00 | 1.20 | 16.74 |
+Group–Taylor | 91.30 | 78.20 | 65.90 | 4.20 | 1.00 | 1.20 | 16.74 |
Pruning Method | FPS | |||
---|---|---|---|---|
RTX 4080 | GTX 1050Ti | GT 1030 | CPU | |
Model 5 | 108.03 | 36.83 | 14.46 | 7.63 |
+Random | 105.71 | 61.93 | 37.32 | 29.20 |
+Lamp | 106.36 | 62.10 | 38.27 | 29.60 |
+L1 | 104.46 | 62.79 | 39.43 | 32.59 |
+Group–Hessian | 105.82 | 63.15 | 39.83 | 32.61 |
+Group–Taylor | 109.37 | 66.31 | 41.90 | 33.03 |
Method | mAP0.5/% | Parameters/105 | FLOPs/109 | Model Size/MB | FPS(CPU) |
---|---|---|---|---|---|
YOLOv5n | 92.50 | 17.70 | 4.20 | 3.90 | 18.51 |
YOLOv5n–Group–Taylor | 91.00 | 6.73 | 1.60 | 1.70 | 23.60 |
YOLOv5s | 93.90 | 70.26 | 15.80 | 14.40 | 7.91 |
YOLOv5m | 94.30 | 208.73 | 47.90 | 42.20 | 3.78 |
YOLOv7 | 94.30 | 365.07 | 103.20 | 74.80 | 1.65 |
YOLOv7–X | 94.40 | 708.14 | 188.10 | 142.10 | 1.12 |
YOLOv8n | 95.40 | 30.07 | 8.10 | 6.20 | 12.67 |
YOLOv8s | 96.00 | 111.27 | 28.40 | 22.50 | 5.84 |
YOLOv8m | 96.70 | 258.43 | 78.70 | 52.00 | 2.67 |
Proposed method | 91.30 | 4.20 | 1.00 | 1.20 | 33.03 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Fan, H.; Liu, J.; Yan, X.; Zhang, C.; Cao, X.; Mao, Q. A Fast and High-Accuracy Foreign Object Detection Method for Belt Conveyor Coal Flow Images with Target Occlusion. Sensors 2024, 24, 5251. https://doi.org/10.3390/s24165251
Fan H, Liu J, Yan X, Zhang C, Cao X, Mao Q. A Fast and High-Accuracy Foreign Object Detection Method for Belt Conveyor Coal Flow Images with Target Occlusion. Sensors. 2024; 24(16):5251. https://doi.org/10.3390/s24165251
Chicago/Turabian StyleFan, Hongwei, Jinpeng Liu, Xinshan Yan, Chao Zhang, Xiangang Cao, and Qinghua Mao. 2024. "A Fast and High-Accuracy Foreign Object Detection Method for Belt Conveyor Coal Flow Images with Target Occlusion" Sensors 24, no. 16: 5251. https://doi.org/10.3390/s24165251