STSN-Net: Simultaneous Tooth Segmentation and Numbering Method in Crowded Environments with Deep Learning
Abstract
:1. Introduction
- A GCN for automatic bone window segmentation was proposed to generate masks of regions to train the RSNN. This process leveraged dental segmentation annotations for initial calibration, followed by feature extraction and mask refinement via GCN.
- A modified DSN was proposed, which made use of a dilated convolution module in certain specific layers to enlarge the receptive field and employed a deformable convolution block to compensate for the diversity in tooth morphology and distribution among individuals.
- The proposed RSSN aimed to accurately segment the tooth region within the images, simultaneously suppressing the background noise and resulting in refined predictions that substantially reduced analytical errors.
- The high precision and strong practicability of the proposed STSN-Net were demonstrated. The proposed two-stage structure initially deployed a GCN for bone window extraction, followed by a multinetwork structure that performed object detection and refinement segmentation, respectively, and simultaneously. Furthermore, STSN-Net was adeptly optimized and adapted for real-world clinical applications in embedded devices.
2. Materials and Methods
2.1. Dataset TSNDS
2.2. Data Augmentation
2.3. The Pipeline of the Proposed STSN-Net
2.4. Automatic Segmentation of Mandibular Morphology
2.5. STSN-Net Structure
2.6. Detection Sub-Network
2.7. Refinement Segmentation Sub-Network
2.8. Loss Function
3. Results
3.1. Evaluation Metrics
3.2. Experiment Results
4. Discussion
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Chandrashekar, G.; AlQarni, S.; Bumann, E.E.; Lee, Y. Collaborative deep learning model for tooth segmentation and identification using panoramic radiographs. Comput. Biol. Med. 2022, 148, 105829. [Google Scholar] [CrossRef]
- Kato, C.N.; Barra, S.G.; Tavares, N.P.; Amaral, T.M.; Brasileiro, C.B.; Mesquita, R.A.; Abreu, L.G. Use of fractal analysis in dental images: A systematic review. Dentomaxillofac. Radiol. 2020, 49, 20180457. [Google Scholar] [CrossRef]
- Mureșanu, S.; Almășan, O.; Hedeșiu, M.; Dioșan, L.; Dinu, C.; Jacobs, R. Artificial intelligence models for clinical usage in dentistry with a focus on dentomaxillofacial CBCT: A systematic review. Oral Radiol. 2023, 39, 18–40. [Google Scholar] [CrossRef]
- Schroder, Â.G.D.; Gonçalves, F.M.; Germiniani, J.d.S.; Schroder, L.D.; Porporatti, A.L.; Zeigelboim, B.S.; de Araujo, C.M.; Santos, R.S.; Stechman-Neto, J. Diagnosis of TMJ degenerative diseases by panoramic radiography: Is it possible? A systematic review and meta-analysis. Clin. Oral Investig. 2023, 27, 6395–6412. [Google Scholar] [CrossRef]
- Rozylo-Kalinowska, I.; Rozylo-Kalinowska, I. Panoramic Radiography in Dentistry. In Imaging Techniques in Dental Radiology: Acquisition, Anatomic Analysis and Interpretation of Radiographic Images; Springer: Cham, Switzerland, 2020; pp. 43–56. [Google Scholar]
- Hlongwa, P.; Moshaoa, M.A.L.; Musemwa, C.; Khammissa, R.A.G. Incidental Pathologic Findings from orthodontic pretreatment panoramic radiographs. Int. J. Environ. Res. Public Health 2023, 20, 3479. [Google Scholar] [CrossRef]
- Heller, Z.A.; Hogge, M.; Ragan, M.R.; Portnof, J.E. Applications of Cone Beam Computed Tomography Scans in Dental Medicine and Potential Medicolegal Issues. Dent. Clin. 2024, 68, 55–65. [Google Scholar] [CrossRef]
- Bilgir, E.; Bayrakdar, İ.Ş.; Çelik, Ö.; Orhan, K.; Akkoca, F.; Sağlam, H.; Odabaş, A.; Aslan, A.F.; Ozcetin, C.; Kıllı, M.; et al. An artifıcial ıntelligence approach to automatic tooth detection and numbering in panoramic radiographs. BMC Med. Imaging 2021, 21, 124. [Google Scholar] [CrossRef] [PubMed]
- Çelik, B.; Çelik, M.E. Automated detection of dental restorations using deep learning on panoramic radiographs. Dentomaxillofac. Radiol. 2022, 51, 20220244. [Google Scholar] [CrossRef] [PubMed]
- AlQarni, S.; Chandrashekar, G.; Bumann, E.E.; Lee, Y. Incremental Learning for Panoramic Radiograph Segmentation. In Proceedings of the 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Glasgow, UK, 11–15 July 2022; pp. 557–561. [Google Scholar]
- Leite, A.F.; Gerven, A.V.; Willems, H.; Beznik, T.; Lahoud, P.; Gaêta-Araujo, H.; Vranckx, M.; Jacobs, R. Artificial intelligence-driven novel tool for tooth detection and segmentation on panoramic radiographs. Clin. Oral Investig. 2021, 25, 2257–2267. [Google Scholar] [CrossRef] [PubMed]
- Yüksel, A.E.; Gültekin, S.; Simsar, E.; Özdemir, Ş.D.; Gündoğar, M.; Tokgöz, S.B.; Hamamcı, İ.E. Dental enumeration and multiple treatment detection on panoramic X-rays using deep learning. Sci. Rep. 2021, 11, 12342. [Google Scholar] [CrossRef] [PubMed]
- Kim, C.; Jeong, H.; Park, W.; Kim, D. Tooth-Related Disease Detection System Based on Panoramic Images and Optimization Through Automation: Development Study. JMIR Med. Inform. 2022, 10, e38640. [Google Scholar] [CrossRef] [PubMed]
- Yin, X.X.; Sun, L.; Fu, Y.; Lu, R.; Zhang, Y. U-Net-Based medical image segmentation. J. Healthc. Eng. 2022, 2022, 4189781. [Google Scholar] [CrossRef]
- Estai, M.; Tennant, M.; Gebauer, D.; Brostek, A.; Vignarajan, J.; Mehdizadeh, M.; Saha, S. Deep learning for automated detection and numbering of permanent teeth on panoramic images. Dentomaxillofac. Radiol. 2022, 51, 20210296. [Google Scholar] [CrossRef] [PubMed]
- Tuzoff, D.V.; Tuzova, L.N.; Bornstein, M.M.; Krasnov, A.S.; Kharchenko, M.A.; Nikolenko, S.I.; Sveshnikov, M.M.; Bednenko, G.B. Tooth detection and numbering in panoramic radiographs using convolutional neural networks. Dentomaxillofac. Radiol. 2019, 48, 20180051. [Google Scholar] [CrossRef] [PubMed]
- Lee, J.H.; Han, S.S.; Kim, Y.H.; Lee, C.; Kim, I. Application of a fully deep convolutional neural network to the automation of tooth segmentation on panoramic radiographs. Oral Surg. Oral Med. Oral Pathol. Oral Radiol. 2020, 129, 635–642. [Google Scholar] [CrossRef] [PubMed]
- Ahn, Y.; Hwang, J.J.; Jung, Y.H.; Jeong, T.; Shin, J. Automated mesiodens classification system using deep learning on panoramic radiographs of children. Diagnostics 2021, 11, 1477. [Google Scholar] [CrossRef]
- Chen, M.; Zhao, C.; Tian, X.; Liu, Y.; Wang, T.; Lei, B. Placental super micro-vessels segmentation based on resnext with convolutional block attention and U-Net. In Proceedings of the 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Virtual, 1–5 November 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 4015–4018. [Google Scholar]
- Bui, T.H.; Hamamoto, K.; Paing, M.P. Automated Caries Screening Using Ensemble Deep Learning on Panoramic Radiographs. Entropy 2022, 24, 1358. [Google Scholar] [CrossRef]
- Siuly, S.; Li, Y.; Wen, P.; Alcin, O.F. SchizoGoogLeNet: The googlenet-based deep feature extraction design for automatic detection of schizophrenia. Comput. Intell. Neurosci. 2022, 2022, 1992596. [Google Scholar] [CrossRef]
- Beeche, C.; Singh, J.P.; Leader, J.K.; Gezer, N.S.; Oruwari, A.P.; Dansingani, K.K.; Chhablani, J.; Pu, J. Super U-Net: A modularized generalizable architecture. Pattern Recognit. 2022, 128, 108669. [Google Scholar] [CrossRef]
- Agarwal, M.; Gupta, S.K.; Biswas, K. Development of a compressed FCN architecture for semantic segmentation using Particle Swarm Optimization. Neural Comput. Appl. 2023, 35, 11833–11846. [Google Scholar] [CrossRef]
- Hu, Z.; Wang, B.; Li, X. Densitytoken: Weakly-Supervised Crowd Counting with Density Classification. In Proceedings of the ICASSP 2023–2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 4–10 June 2023; IEEE: Piscataway, NJ, USA, 2023; pp. 1–5. [Google Scholar]
- Sivaprakash, P.; Sankar, M.; Chithambaramani, R.; Marichamy, D. A Convolutional Neural Network Approach for Crowd Counting. In Proceedings of the 2023 4th International Conference on Smart Electronics and Communication (ICOSEC), Trichy, India, 20–22 September 2023; IEEE: Piscataway, NJ, USA, 2023; pp. 1515–1520. [Google Scholar]
- Chandio, A.A.; Asikuzzaman, M.; Pickering, M.R.; Leghari, M. Cursive text recognition in natural scene images using deep convolutional recurrent neural network. IEEE Access 2022, 10, 10062–10078. [Google Scholar] [CrossRef]
- Xue, C.; Huang, J.; Zhang, W.; Lu, S.; Wang, C.; Bai, S. Image-to-Character-to-Word Transformers for Accurate Scene Text Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 45, 12908–12921. [Google Scholar] [CrossRef] [PubMed]
- Islam, S.M.; Mondal, H.S. Image enhancement based medical image analysis. In Proceedings of the 2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT), Kanpur, India, 6–8 July 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 1–5. [Google Scholar]
- Subramanian, A.K.; Chen, Y.; Almalki, A.; Sivamurthy, G.; Kafle, D. Cephalometric Analysis in Orthodontics Using Artificial Intelligence—A Comprehensive Review. BioMed Res. Int. 2022, 2022, 1880113. [Google Scholar] [CrossRef] [PubMed]
- Alshagathrh, F.M.; Househ, M.S. Artificial Intelligence for Detecting and Quantifying Fatty Liver in Ultrasound Images: A Systematic Review. Bioengineering 2022, 9, 748. [Google Scholar] [CrossRef] [PubMed]
- Huang, S.; Su, Z.; Liu, S.; Chen, J.; Su, Q.; Su, H.; Shang, Y.; Jiao, Y. Combined assisted bone age assessment and adult height prediction methods in Chinese girls with early puberty: Analysis of three artificial intelligence systems. Pediatr. Radiol. 2023, 53, 1108–1116. [Google Scholar] [CrossRef] [PubMed]
- Coşkun, M.F.; Akar, G.B. Crowd Multi Prediction: Single Network for Crowd Counting, Localization and Anomaly Detection. In Proceedings of the 2023 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA, 6–8 January 2023; IEEE: Piscataway, NJ, USA, 2023; pp. 1–3. [Google Scholar]
- Sheng, J.; Zhang, Z. Summary of Pedestrian Traffic Statistics Research. In Proceedings of the 2020 15th IEEE Conference on Industrial Electronics and Applications (ICIEA), Kristiansand, Norway, 9–13 November 2020; IEEE: Piscataway, NJ, USA, 2020; pp. 43–47. [Google Scholar]
- Zhi, X.; Zhao, B.; Wang, Y. A Hybrid Framework for Text Recognition Used in Commodity Futures Document Verification. In Proceedings of the 2021 6th International Conference on Computational Intelligence and Applications (ICCIA), Xiamen, China, 11–13 June 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 143–148. [Google Scholar]
- Kılıc, M.C.; Bayrakdar, I.S.; Çelik, Ö.; Bilgir, E.; Orhan, K.; Aydın, O.B.; Kaplan, F.A.; Sağlam, H.; Odabaş, A.; Aslan, A.F.; et al. Artificial intelligence system for automatic deciduous tooth detection and numbering in panoramic radiographs. Dentomaxillofac. Radiol. 2021, 50, 20200172. [Google Scholar] [CrossRef]
- Sheng, C.; Wang, L.; Huang, Z.; Wang, T.; Guo, Y.; Hou, W.; Xu, L.; Wang, J.; Yan, X. Transformer-Based Deep Learning Network for Tooth Segmentation on Panoramic Radiographs. J. Syst. Sci. Complex. 2023, 36, 257–272. [Google Scholar] [CrossRef]
- Wang, J.; Wu, Y.; Mei, Y. Insulator instance segmentation based on deep learning network Mask RCNN. In Proceedings of the 2022 China International Conference on Electricity Distribution (CICED), Changsha, China, 7–8 September 2022; IEEE: Piscataway, NJ, USA, 2022; pp. 1424–1427. [Google Scholar]
- Gunawan, F.; Hwang, C.L.; Cheng, Z.E. ROI-YOLOv8-Based Far-Distance Face-Recognition. In Proceedings of the 2023 International Conference on Advanced Robotics and Intelligent Systems (ARIS), Taipei, Taiwan, 30 August–1 September 2023; IEEE: Piscataway, NJ, USA, 2023; pp. 1–6. [Google Scholar]
Category | Train | Validation | Test | Total |
---|---|---|---|---|
Permanent Dentition | 1115 | 228 | 228 | 1571 |
Primary Dentition | 422 | 86 | 87 | 595 |
Total | 1500 | 300 | 316 | 2116 |
Subcategory | Train | Validation | Test | Total |
---|---|---|---|---|
Missing Teeth | 427 | 87 | 88 | 602 |
Dental Crowding | 837 | 171 | 172 | 1180 |
Impacted Teeth | 528 | 108 | 108 | 744 |
Teeth with Fillings | 384 | 78 | 79 | 541 |
Teeth with Restorations | 328 | 67 | 67 | 462 |
Base Size (1333 × 800) | F1 (1.0×) | F1 (1.2×) | F1 (1.4×) | F1 (1.6×) |
---|---|---|---|---|
Baseline | 97.36% | 97.53% | 97.64% | 97.88% |
+ Deformable convolution | 98.09% | 98.14% | 98.19% | 98.23% |
+ Dilated convolution | 98.08% | 98.13% | 98.14% | 98.16% |
+ Region segmentation | 98.11% | 98.12% | 98.15% | 98.22% |
Mask R-CNN | 97.49% | 97.62% | 97.82% | 97.89% |
YOLOv8 | 96.49% | 96.78% | 96.89% | 97.02% |
The proposed network | 98.30% | 98.32% | 98.38% | 98.49% |
Metrics | Mask R-CNN | YOLOv8 | STSN-Net |
---|---|---|---|
mAP (object detection) | 96.58% | 95.61% | 98.90% |
mAP (instance segmentation) | 95.88% | 94.60% | 98.10% |
DICE | 94.98% | 94.02% | 96.29% |
Base Size (1333 × 800) | F1 (1.0×) | F1 (1.2×) | F1 (1.4×) | F1 (1.6×) |
---|---|---|---|---|
Baseline | 85 | 89 | 94 | 100 |
+ Deformable convolution | 100 | 110 | 115 | 120 |
+ Dilated convolution | 95 | 98 | 100 | 110 |
+ Region segmentation | 90 | 95 | 99 | 104 |
The proposed network | 110 | 115 | 118 | 121 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wang, S.; Liang, S.; Chang, Q.; Zhang, L.; Gong, B.; Bai, Y.; Zuo, F.; Wang, Y.; Xie, X.; Gu, Y. STSN-Net: Simultaneous Tooth Segmentation and Numbering Method in Crowded Environments with Deep Learning. Diagnostics 2024, 14, 497. https://doi.org/10.3390/diagnostics14050497
Wang S, Liang S, Chang Q, Zhang L, Gong B, Bai Y, Zuo F, Wang Y, Xie X, Gu Y. STSN-Net: Simultaneous Tooth Segmentation and Numbering Method in Crowded Environments with Deep Learning. Diagnostics. 2024; 14(5):497. https://doi.org/10.3390/diagnostics14050497
Chicago/Turabian StyleWang, Shaofeng, Shuang Liang, Qiao Chang, Li Zhang, Beiwen Gong, Yuxing Bai, Feifei Zuo, Yajie Wang, Xianju Xie, and Yu Gu. 2024. "STSN-Net: Simultaneous Tooth Segmentation and Numbering Method in Crowded Environments with Deep Learning" Diagnostics 14, no. 5: 497. https://doi.org/10.3390/diagnostics14050497