Augmented Dataset for Vision-Based Analysis of Railroad Ballast via Multi-Dimensional Data Synthesis
Abstract
:1. Introduction
2. Objective and Scope
3. Field and Lab Ballast Data Preparation
3.1. Overview
3.2. Methodology
3.2.1. Field Data Acquisition
3.2.2. Laboratory Data Collection
3.2.3. Data Processing and Labeling
4. Synthetic Data Generation
4.1. Overview
- Limited accessibility. Acquiring a high-quality ballast dataset for training robust deep learning models requires access to dedicated in-service railroad sites, high-end equipment, and coordination with specialized departments with domain knowledge.
- Limited scalability. The processes of gathering, processing, and annotating field data are time-consuming and labor-intensive.
- Limited flexibility. Extracting specific forms of data, such as individual ballast particles within a 3D reconstructed point cloud, can be extremely challenging and may be unattainable from images of actual ballast in real-world scenarios.
- Ballast data. The ballast dataset provides high-resolution (2500 pixel × 2500 pixel) RGB images captured from a top-down perspective under various lighting conditions (Figure 1a–c). It also contains a 3D heightmap (Figure 1d) corresponding to the base plane, a 3D textured point cloud (Figure 1f) encompassing all the ballast particles within the scene, and individual, complete meshes for each ballast particle. Figure 1d visualizes the generated 3D heightmap, where the height of each pixel to the base plane is encoded with a colormap ranging from dark blue to red, where red pixels indicate the highest points in the physical world. The diversity of the synthetic data allows for the emulation of a wide range of commonly used ballast imaging devices, including but not limited to line scan cameras, area scan cameras, and 3D laser scanners.
- Ballast labels. Within a generated scene, each synthetic ballast particle is assigned a unique particle ID ranging from 1 to the number of particles in the scene. The label of a synthetic 2D ballast image (2D particle masks) paired with the ballast data is a map from each pixel in the 2D ballast image to its corresponding ballast particle’s unique ID. Figure 1e visualizes the 2D particle mask for a top-view RGB image, where the red polygons are the outlines of each ballast particle, and the color of each pixel inside the red polygons encodes its unique ballast particle ID. Similarly, the label of a generated 3D ballast point cloud is a map from each 3D point in the point cloud to its corresponding ballast particle’s unique ID. Figure 1g visualizes the label of a 3D point cloud by rendering each point in the point cloud with a unique color encoding its unique ballast ID, indicating that points with the same color belong to the same ballast particle. The gray cuboids in Figure 1g show the 3D bounding box of each ballast particle. Furthermore, from the unique particle ID, the completed mesh (Figure 1h) that each 3D point belongs to can be linked. These auto-generated annotations allow multi-dimensional ballast instance recognition and segmentation.
- Ballast statistics. In practice, statistical aggregate features can provide valuable insights into the geotechnical characteristics of the ballast sample, like fouling conditions. Therefore, the actual particle size distribution (PSD) (Figure 1i), as well as a series of morphological metrics including sphericity, the flat and elongated ratio (FER), and angularity for particles over 3/8 in. (9.5 mm), is calculated and appended for each generated scene.
4.2. Three-Dimensional Mesh Libarary of Actual Ballast Particles
4.3. Scene Simulation and Ballast Data Rendering
4.3.1. Physical Simulation
4.3.2. Rendering
4.4. Dataset Acquisition from the Generated Scenes
4.4.1. Two-Dimensional Images and Masks
4.4.2. Three-Dimensional Point Cloud and Annotations
5. Evaluation of Results
5.1. Comparison between Train on Real Test on Real (TRTR) and Train on Synthetic Test on Real (TSTR)
5.2. Comparison between Train on Real Test on Real (TRTR) and Train on Augmented Test on Real (TATR)
6. Summary and Conclusions
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Freight Rail & Climate Change–AAR. Available online: https://www.aar.org/issue/freight-rail-climate-change/ (accessed on 13 February 2024).
- Al-Qadi, I.L.; Rudy, J.; Boyle, J.; Tutumluer, E. Railroad Ballast Fouling Detection Using Ground Penetrating Radar—A New Approach Based on Scattering from Voids. e-J. Nondestruct. Test. 2006, 11, 8. [Google Scholar]
- Stark, T.D.; Wilk, S.T.; Swan, R.H. Sampling, Reconstituting, and Gradation Testing of Railroad Ballast. Railr. Ballast Test. Prop. 2018, 2018, 135–143. [Google Scholar] [CrossRef]
- Al-Qadi, I.L.; Xie, W.; Roberts, R. Scattering Analysis of Ground-Penetrating Radar Data to Quantify Railroad Ballast Contamination. NDT E Int. 2008, 41, 441–447. [Google Scholar] [CrossRef]
- Jol, H.M. Ground Penetrating Radar Theory and Applications; Elsevier: Amsterdam, The Netherlands, 2008; ISBN 978-0-08-095184-3. [Google Scholar]
- Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An Image Is Worth 16 × 16 Words: Transformers for Image Recognition at Scale. arXiv 2020, arXiv:2010.11929. [Google Scholar]
- Luo, J.; Huang, H.; Ding, K.; Qamhia, I.I.A.; Tutumluer, E.; Hart, J.M.; Thompson, H.; Sussmann, T.R. Toward Automated Field Ballast Condition Evaluation: Algorithm Development Using a Vision Transformer Framework. Transp. Res. Rec. 2023, 2677, 423–437. [Google Scholar] [CrossRef]
- Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; Guo, B. Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Computer Vision (ICCV), Montreal, QC, Canada, 10–17 October 2021; pp. 9992–10002. [Google Scholar] [CrossRef]
- Cai, Z.; Vasconcelos, N. Cascade R-CNN: High Quality Object Detection and Instance Segmentation. arXiv 2019, arXiv:1906.09756. [Google Scholar]
- Luo, J.; Ding, K.; Huang, H.; Hart, J.M.; Qamhia, I.I.A.; Tutumluer, E.; Thompson, H.; Sussmann, T.R. Toward Automated Field Ballast Condition Evaluation: Development of a Ballast Scanning Vehicle. Transp. Res. Rec. 2023, 2678, 24–36. [Google Scholar] [CrossRef]
- Wu, X.; Liang, L.; Shi, Y.; Fomel, S. FaultSeg3D: Using Synthetic Data Sets to Train an End-to-End Convolutional Neural Network for 3D Seismic Fault Segmentation. Geophysics 2019, 84, IM35–IM45. [Google Scholar] [CrossRef]
- Lalaoui, L.; Mohamadi, T.; Djaalab, A. New Method for Image Segmentation. Procedia-Soc. Behav. Sci. 2015, 195, 1971–1980. [Google Scholar] [CrossRef]
- Barth, R.; IJsselmuiden, J.; Hemming, J.; Henten, E.J.V. Data Synthesis Methods for Semantic Segmentation in Agriculture: A Capsicum annuum Dataset. Comput. Electron. Agric. 2018, 144, 284–296. [Google Scholar] [CrossRef]
- Toda, Y.; Okura, F.; Ito, J.; Okada, S.; Kinoshita, T.; Tsuji, H.; Saisho, D. Training Instance Segmentation Neural Network with Synthetic Datasets for Crop Seed Phenotyping. Commun. Biol. 2020, 3, 173. [Google Scholar] [CrossRef]
- Huang, H. Field Imaging Framework for Morphological Characterization of Aggregates with Computer Vision: Algorithms and Applications. Ph.D. Thesis, University of Illinois at Urbana-Champaign, Champaign, IL, USA, 2021. [Google Scholar]
- Goodfellow, I.J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative Adversarial Networks. Commun. ACM 2020, 63, 139–144. [Google Scholar]
- Cheng, M.-Y.; Khasani, R.R.; Setiono, K. Image Quality Enhancement Using HybridGAN for Automated Railway Track Defect Recognition. Autom. Constr. 2023, 146, 104669. [Google Scholar] [CrossRef]
- Zheng, S.; Dai, S. Image Enhancement for Railway Inspections Based on CycleGAN under the Retinex Theory. In Proceedings of the 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), Indianapolis, IN, USA, 19–22 September 2021; pp. 2330–2335. [Google Scholar]
- Liu, S.; Ni, H.; Li, C.; Zou, Y.; Luo, Y. DefectGAN: Synthetic Data Generation for EMU Defects Detection with Limited Data. IEEE Sens. J. 2024, 24, 17638–17652. [Google Scholar] [CrossRef]
- Hess, R. Blender Foundations: The Essential Guide to Learning Blender 2.5; Routledge: New York, NY, USA, 2010; ISBN 978-0-240-81431-5. [Google Scholar]
- Man, K.; Chahl, J. A Review of Synthetic Image Data and Its Use in Computer Vision. J. Imaging 2022, 8, 310. [Google Scholar] [CrossRef]
- Karoly, A.I.; Galambos, P. Automated Dataset Generation with Blender for Deep Learning-Based Object Segmentation. In Proceedings of the IEEE 20th Jubilee World Symposium on Applied Machine Intelligence and Informatics (SAMI), Poprad, Slovakia, 2–5 March 2022; pp. 000329–000334. [Google Scholar] [CrossRef]
- Ding, K.; Luo, J.; Huang, H.; Hart, J.M.; Qamhia, I.I.A.; Tutumluer, E. Augmented Dataset for Multidimensional Ballast Segmentation and Evaluation. IOP Conf. Ser. Earth Environ. Sci. 2024, 1332, 012019. [Google Scholar] [CrossRef]
- Dutta, A.; Zisserman, A. The VIA Annotation Software for Images, Audio and Video. In Proceedings of the 27th ACM International Conference on Multimedia, Nice, France, 21–25 October 2019; ACM: New York, NY, USA, 2019; pp. 2276–2279. [Google Scholar]
- Russell, B.C.; Torralba, A.; Murphy, K.P.; Freeman, W.T. LabelMe: A Database and Web-Based Tool for Image Annotation. Int. J. Comput. Vis. 2008, 77, 157–173. [Google Scholar] [CrossRef]
- Sekachev, B.; Manovich, N.; Zhiltsov, M.; Zhavoronkov, A.; Kalinin, D.; Hoff, B.; TOsmanov; Kruchinin, D.; Zankevich, A.; Sidnev, D.; et al. Opencv/Cvat: V1.1.0 2020. Available online: https://github.com/cvat-ai/cvat/issues/2392 (accessed on 28 July 2024).
- Lynn, T. Launch: Label Data with Segment Anything in Roboflow. Available online: https://blog.roboflow.com/label-data-segment-anything-model-sam/ (accessed on 28 July 2024).
- Koohmishi, M.; Palassi, M. Evaluation of Morphological Properties of Railway Ballast Particles by Image Processing Method. Transp. Geotech. 2017, 12, 15–25. [Google Scholar] [CrossRef]
- Coumans, E. Bullet Physics Simulation. In Proceedings of the SIGGRAPH ‘15: Special Interest Group on Computer Graphics and Interactive Techniques Conference, Los Angeles, CA, USA, 9–13 August 2015; p. 1. [Google Scholar] [CrossRef]
- Kobbelt, L.; Campagna, S.; Seidel, H. A General Framework for Mesh Decimation; RWTH Aachen University: Aachen, Germany, 1998. [Google Scholar]
- Lambe, T.W.; Whitman, R.V. Soil Mechanics; John Wiley & Sons: Hoboken, NJ, USA, 1991; ISBN 978-0-471-51192-2. [Google Scholar]
- Iraci, B. Blender Cycles: Lighting and Rendering Cookbook, 1st ed.; Quick Answers to Common Problems; Packt Publishing: Birmingham, UK, 2013; ISBN 978-1-78216-461-6. [Google Scholar]
- Valenza, E. Blender 2.6 Cycles: Materials and Textures Cookbook; Packt Publishing Ltd.: Birmingham, UK, 2013; ISBN 978-1-78216-131-8. [Google Scholar]
- Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; Zitnick, C.L. Microsoft COCO: Common Objects in Context. In Proceedings of the Computer Vision—ECCV 2014: 13th European Conference, Zurich, Switzerland, 6–12 September 2014; Proceedings, Part V. pp. 740–755. [Google Scholar] [CrossRef]
- Platzer, M.; Reutterer, T. Holdout-Based Empirical Assessment of Mixed-Type Synthetic Data. Front. Big Data 2021, 4, 679939. [Google Scholar] [CrossRef]
- Snoke, J.; Raab, G.M.; Nowok, B.; Dibben, C.; Slavkovic, A. General and Specific Utility Measures for Synthetic Data. J. R. Stat. Soc. Ser. A 2018, 181, 663–688. [Google Scholar]
- Kindratenko, V.; Mu, D.; Zhan, Y.; Maloney, J.; Hashemi, S.H.; Rabe, B.; Xu, K.; Campbell, R.; Peng, J.; Gropp, W. HAL: Computer System for Scalable Deep Learning. In Proceedings of the PEARC’20: Practice and Experience in Advanced Research Computing 2020: Catch the Wave, Portland, OR, USA, 27–30 July 2020; pp. 41–48. [Google Scholar]
- Park, N.; Mohammadi, M.; Gorde, K.; Jajodia, S.; Park, H.; Kim, Y. Data Synthesis Based on Generative Adversarial Networks. Proc. VLDB Endow. 2018, 11, 1071–1083. [Google Scholar] [CrossRef]
- Wang, L.; Zhang, W.; He, X. Continuous Patient-Centric Sequence Generation via Sequentially Coupled Adversarial Learning; Database Systems for Advanced Applications—24th International Conference, DASFAA 2019, Proceedings; Springer: Berlin/Heidelberg, Germany, 2019; Volume 11447 LNCS, p. 52. ISBN 978-3-030-18578-7. [Google Scholar]
- Beaulieu-Jones, B.K.; Williams, C.; Greene, C.S.; Wu, Z.S.; Lee, R.; Byrd, J.B.; Bhavnani, S.P. Privacy-Preserving Generative Deep Neural Networks Support Clinical Data Sharing. Circ. Cardiovasc. Qual. Outcomes 2019, 12, e005122. [Google Scholar] [CrossRef] [PubMed]
- Chin-Cheong, K.; Sutter, T.; Vogt, J.E. Generation of Heterogeneous Synthetic Electronic Health Records Using GANs. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, BC, Canada, 8–14 December 2019; ETH Zurich: Zürich, Switzerland, 2019. [Google Scholar]
- Zhu, M. Recall, Precision and Average Precision; Department of Statistics and Actuarial Science, University of Waterloo: Waterloo, ON, Canada, 2004. [Google Scholar]
- Che, Z.; Cheng, Y.; Zhai, S.; Sun, Z.; Liu, Y. Boosting Deep Learning Risk Prediction with Generative Adversarial Networks for Electronic Health Records. In Proceedings of the 2017 IEEE International Conference on Data Mining (ICDM), New Orleans, LA, USA, 18–21 November 2017; pp. 787–792. [Google Scholar]
- Yang, F.; Yu, Z.; Liang, Y.; Gan, X.; Lin, K.; Zou, Q.; Zeng, Y. Grouped Correlational Generative Adversarial Networks for Discrete Electronic Health Records. In Proceedings of the 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Bioinformatics and Biomedicine (BIBM), San Diego, CA, USA, 18–21 November 2019; pp. 906–913. [Google Scholar] [CrossRef]
- Borrego, J.; Dehban, A.; Figueiredo, R.; Moreno, P.; Bernardino, A.; Santos-Victor, J. Applying Domain Randomization to Synthetic Data for Object Category Detection 2018. arXiv 2018, arXiv:1807.09834. [Google Scholar]
Sources | Fouling Conditions | Number of Images |
---|---|---|
Railroad field sites | Clean | 51 |
Fouled | 93 | |
Lab-engineered samples | Clean | 142 |
Aggregate quarries | Clean | 137 |
Fouling Index (FI) | Number of Scenes | Number of 2D RGB Images | Total Number of 2D Ballast Instances | Average Number of 3D Points | Total Number of 3D Ballast Instances |
---|---|---|---|---|---|
7 | 32 | 120 | 73,954 | 3,045,332 | 21,123 |
14 | 30 | 103 | 52,625 | 3,045,153 | 19,075 |
23 | 22 | 103 | 28,990 | 2,970,323 | 14,347 |
30 | 20 | 111 | 31,552 | 2,968,787 | 12,773 |
39 | 16 | 109 | 24,415 | 1,882,340 | 7496 |
Total | 120 | 546 | 211,536 | 2,863,712 | 74,814 |
Experiment Group | Trained with Real Ballast Dataset | Trained with Synthetic Ballast Dataset | ||
---|---|---|---|---|
TRTR | YES | NO | 32.1% | 34.3% |
TSTR | NO | YES | 21.2% | 23.9% |
TATR | YES | YES | 32.9% | 34.6% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Ding, K.; Luo, J.; Huang, H.; Hart, J.M.; Qamhia, I.I.A.; Tutumluer, E. Augmented Dataset for Vision-Based Analysis of Railroad Ballast via Multi-Dimensional Data Synthesis. Algorithms 2024, 17, 367. https://doi.org/10.3390/a17080367
Ding K, Luo J, Huang H, Hart JM, Qamhia IIA, Tutumluer E. Augmented Dataset for Vision-Based Analysis of Railroad Ballast via Multi-Dimensional Data Synthesis. Algorithms. 2024; 17(8):367. https://doi.org/10.3390/a17080367
Chicago/Turabian StyleDing, Kelin, Jiayi Luo, Haohang Huang, John M. Hart, Issam I. A. Qamhia, and Erol Tutumluer. 2024. "Augmented Dataset for Vision-Based Analysis of Railroad Ballast via Multi-Dimensional Data Synthesis" Algorithms 17, no. 8: 367. https://doi.org/10.3390/a17080367