A Robust Tool for 3D Rail Mapping Using UAV Data Photogrammetry, AI and CV: qAicedrone-Rail

Abstract

Rail systems are essential for economic growth and regional connectivity, but aging infrastructures face challenges from increased demand and environmental factors. Traditional inspection methods, such as visual inspections, are inefficient and costly and pose safety risks. Unmanned Aerial Vehicles (UAVs) have become a viable alternative to rail mapping and monitoring. This study presents a robust method for the 3D extraction of rail tracks from UAV-based aerial imagery. The approach integrates YOLOv8 for initial detection and segmentation, photogrammetry for 3D data extraction and computer vision techniques with a Multiview approach to enhance accuracy. The tool was tested in a real-world complex scenario. Errors of 2 cm and 4 cm were obtained for planimetry and altimetry, respectively. The detection performance and metric results show a significant reduction in errors and increased precision compared to intermediate YOLO-based outputs. In comparison to most image-based methodologies, the tool has the advantage of generating both accurate altimetric and planimetric data. The generated data exceed the requirements for cartography at a scale of 1:500, as required by the Spanish regulations for photogrammetric works for rail infrastructures. The tool is integrated into the open-source QGIS platform; the tool is user-friendly and aims to improve rail system maintenance and safety.

1. Introduction

Rail systems are critical infrastructures due to their role in economic growth, efficient passenger transport and energy efficiency [1,2]. They ensure the safe, reliable movement of goods and people and enhance connectivity between urban and rural areas, contributing to regional development [3,4].

Adequate rail monitoring is vital to ensure proper functioning and reduce maintenance costs. Most rail infrastructures across Europe were built several decades ago and are subject to an increasing demand as the population grows. Moreover, this type of infrastructure is exposed to extreme weather events and climate change, accelerating its deterioration [5,6]. Regular maintenance and advanced monitoring techniques are crucial to address these challenges, extending the lifespan of rail systems and enhancing their reliability and safety [7].

Until a few years ago, visual inspection was the usual methodology for rail inspection, while rail geometry was determined by classical surveying [7]. In both cases, technicians would walk the tracks, exposing themselves to high risk due to train circulation, and in some cases, traffic interruption was required. The process was time-consuming and costly. The obtained data were also limited by the capabilities of the technician and the security measures. In recent years, different methodologies have been implemented for rail measurement, including laser technology [8,9,10] and image-based approaches [11].

Nowadays, UAVs have emerged as the best alternative to obtain rail data in a cost-efficient manner and without risk [12]. UAVs are used for rail track detection and inspection [13], and they can be equipped with different sensors, with LiDAR [14,15] and RGB cameras [16] being the most commonly used.

In the context of the image-based approach, combinations of artificial intelligence (AI) and computer vision (CV) have been used for rail detection and inspection from UAV, aerial and terrestrial data. Some studies focus on the detection of rail tracks from high-resolution remote sensing images using different filter-based approaches [17]. Taking UAV images as input data, Sahebdivani et al. [18] developed a methodology for track detection in 3D using the derived UAV point clouds, Guclu et al. [19] used Gabor filters to determine rails positions that were to be followed by the UAV and Mammeri et al. [20] used a U-Net model for railway segmentation.

The accurate extraction of railways is a crucial step for the generation of Digital Twins (DTs), virtual models that improve capabilities for the monitoring, analysis and maintenance of physical infrastructures and that require accurate and detailed data [21,22]. The creation of a DT starts with the accurate data extraction of the real-world object; the first step consists of defining the rails in 3D coordinates with adequate precision. Although we can be led to think that most railways have been correctly mapped since their construction, the age of the infrastructures that have been exposed to all weather conditions as well as high loads, land subsidence or even earthquakes lead us to consider the possibility that the existing data are no longer usable, especially regarding rail altimetry [23].

In this work, a robust methodology for the extraction of rails in 3D from aerial imagery is presented. The input for the tool is a photogrammetric flight of the area of interest. Once the photogrammetric process is completed, AI, in particular YOLOv8, is applied for the detection and segmentation of rails and railways in the image. Then, CV and a Multiview approach, together with the generated Digital Surface Model (DSM), are applied to robustly extract the rails in 3D. The resulting data are suitable for the generation of products such as DTs and for infrastructure monitoring and analysis tasks.

Although several works have focused on the automatic detection of rails from images, in many cases, the results are limited to image coordinates, while the planimetric accuracy is not evaluated and the altimetry is frequently overlooked. These methodologies are also greatly affected by occlusions and do not provide a comprehensive solution for rail mapping [18,24].

The presented methodology offers significant improvements compared to traditional aerial image-based approaches in terms of occlusion handling and precision. While building on established techniques such as photogrammetry and artificial intelligence, we developed a novel algorithm that incorporates Multiview processing to overcome the limitations of computer vision alone. This Multiview approach not only enhances the accuracy of rail mapping but also effectively mitigates occlusion problems, producing accurate results both in planimetry and altimetry.

The tool is designed to be user-friendly and open access, enabling the complete generation of railway track cartography. It is fully integrated into the open-source and free software QGIS 3.28.9 in the form of two plugins. The segmentation pretrained model based on YOLOv8 detection is also provided.

2. Materials and Methods

The workflow of the methodology can be divided into three main steps. First, after the images were acquired, the photogrammetric process was computed to generate 3D data and an orthomosaic. Next, a semantic segmentation pretrained model based on YOLOv8 was applied to the original images to detect rail and railway polygons. Rail refers to each one of the individual tracks or steel bars, while the railway encompasses the two rails and the space between them. Finally, the photogrammetric data, images and polygons were loaded into the qAicedrone tool, where the optimization of the railways was carried out. The methodology was fully integrated into QGIS [25], an open-source software, through two dedicated plugins (Photogrammetry Tools and qAicedrone). The workflow is presented in Figure 1.

2.1. Input Data

The data required for the process consists of a photogrammetric flight of the area of interest. In order to obtain the optimal accuracy for the rail mapping task, the Ground Sample Distance (GSD) of the images must be smaller than 2 cm/pixel. This resolution corresponds approximately to one-third of the rail head’s width (7 cm), guaranteeing that at least one pixel is entirely contained within the rail.

The forward overlap is set to 70–80% (i.e., direction of flight), and the side overlap is 50–60% (i.e., the opposite direction). The UAV must be equipped with an RGB camera capable of taking images with the required GSD.

Appropriate georeferencing of the images should be carried out using Real-Time Kinematic (RTK), Ground Control Points (GCPs), or a combination of both. In order to obtain optimal results in the form of highly accurate rail mapping, a combination of the methodologies is recommended, using control GCPs for flight control and accuracy testing. Detailed information on UAV flight planning can be found in [26].

Rail mapping will likely involve covering large areas, which may present a challenge in terms of the battery life of most UAVs. Fixed-wing UAVs are recommended for data acquisition as they are capable of covering larger areas than multirotor UAVs. A UAV similar to the Matrice 300 RTK equipped with a Zenmuse P1 RGB camera used in the present work is recommended.

In case it is not possible to cover the whole area of study in one flight, the following approach is recommended: take-off and landing bases will be separated at twice the distance that can be covered in one flight. From each base, two flights will be performed, one forward and one backward. A visualization of this planification is shown in Figure 2.

2.2. Photogrammetric Process and Requirements

The photogrammetric process must be carried out prior to rail detection and mapping. It can be performed using a photogrammetric software such as Agisoft Metashape 2.0.0 (Agisoft LLC, St. Petersburg, Russia). Other photogrammetric software can be employed for this task; however, some data format transformation may be required in order to load the photogrammetric information to the QGIS plugins used in the next steps of the process.

The following outputs need to be obtained: a georeferenced point cloud (for visualization only) (Figures S1 and S2), a DSM, a Digital Terrain Model (DTM), images corrected from lens distortion and an orthomosaic. GCPs must be employed to assure the accuracy matches the needs of the project.

2.3. Rail and Railway Detection on Images

The rail and railway detection are presented as problems of object detection and segmentation. YOLOv8 [27] is considered the state-of-the-art algorithm for object detection. YOLOv8 is capable not only of detecting but also accurately segmenting the detections.

The different variants of YOLOv8, for instance, segmentation (i.e., yolov8n-seg, yolov8s-seg, yolov8m-seg, yolov8l-seg and yolov8x-seg) were tested in order to find the best performing alternative. Among all different versions, YOLOv8x-seg was found to provide the best results and was chosen for segmentation.

Model training was required prior to using the tool. The model was trained using a total of 3872 aerial images with an approximate GSD of 1 cm and a resolution of 640 × 640 px. The rails and railways were manually delineated for each image using Roboflow [28], obtaining approximately 24,000 instances of rail and 12,000 instances of railways (Figure 3).

After several tests, the optimal parameters for training were obtained. The learning rate was set to 0.01, with a 0.0005 decay and 0.937 momentum, the chosen optimizer was Stochastic Gradient Descent (SGD), and training was carried out for 25 epochs. The initial resolution of 640 × 640 pixels was maintained. A general mean Average Precision of 0.805 was obtained at 0.50 (mAP50). For classes, the mAP50 was slightly better for rail, with a value of 0.82, while for railways, an mAP50 of 0.79 was achieved (Figure 4).

The weights obtained after training were used for inference in the presented tool. Each image of the project was first divided into tiles, with a similar resolution and GSD to those used in training. Detection and segmentation were then computed for each tile, obtaining a polygon for each detection.

Detection was carried out prior to the QGIS process. The geometry of the resulting polygons, stored in YOLO format, needed to be transformed into Well-Known Text (WKT) format. To carry out the transformation, a Python script named PredictWktFormat.py was developed. The script is available at https://github.com/UCLM-PAFyC/UCLM-PAFyC-AicedronePythonTools (accessed on 12 December 2024).

2.4. Rail Detection and Optimization in 3D Using qAicedrone

The core of the rail detection and optimization methodology relies on two QGIS plugins specifically developed for this project. The first plugin, Photogrammetry Tools, allows us to work with the previously obtained photogrammetric data; specifically, it performs the required transformations between the image space and the object space and deals with image distortions. The second plugin, qAicedrone, performs the optimization of the rail segmentation. This plugin applies constraints to ensure that the 2D segmentations from the images are accurately transformed into 3D data, including the delimitation and parametrization of the railways (Figure 5). The tool’s workflow is outlined in the following steps.

2.4.1. Project Preparation

The project begins by creating a structured environment within QGIS using the following plugins: Photogrammetry Tools and qAicedrone. These tools form the basis of data processing and rail segmentation. All intermediate and final results are stored in a SpatiaLite database. This database allows for efficient data management during processing.

Firstly, a Photogrammetry Tools project is created, and the photogrammetric data previously obtained are loaded. Afterwards, the qAicedrone project is established, linked to the Photogrammetry Tools project. This ensures seamless integration between photogrammetric data processing and segmentation, allowing for more refined rail modeling in 3D space. Once both projects are prepared, the rail detection and optimization steps are carried out. The steps of qAicedrone are visible in Figure 6.

2.4.2. Loading YOLO Detections

The main challenge for 3D data semantic segmentation lies in the unstructured nature of the 3D point clouds. In order to apply neural networks effectively, a transformation is required to provide the 3D data with a regular structure, allowing it to be processed using standard 2D architectures. Projection-based methods provide a solution to this issue. Of the different types of projection-based methods, the Multiview representation was found to be particularly useful for the specific task of this project. Multiview representation employs multiple views of the 3D data, and a 2D semantic segmentation is performed for each individual view. Afterwards, the multiple segmentations are analyzed and compared, taking advantage of the redundancies, to obtain the final segmentation of the 3D data.

The YOLO detections, formatted in WKT, are imported using the qAicedrone tool’s “Import AI Results” function. This process automates the transformation of detected polygons from image space to object space using Photogrammetry Tools. For the railway detections, each polygon’s main axis is defined, and a uniform polygon width of 0.5 m is applied to maintain consistency in the analysis (Figure 7).

At this stage, the study area is segmented into 5 × 5 m tiles. This tiling strategy enables the treatment of short railway segments as straight lines, ignoring curvatures and therefore simplifying the computational complexity for further processing.

Validation is carried out by ensuring all rail objects lie within a railway; otherwise, the rail is discarded.

2.4.3. Preprocess Detections

The objective of this step is to obtain a unique 2D solution for each rail segment by eliminating redundancies. Using the “Preprocess AI Results” option, rail segments are first grouped based on spatial overlap (within a 0.07 m buffer) and angle (below 3 degrees). The objective here is to consolidate multiple detections across different images into a single consolidated representation.

After grouping, RANSAC algorithm is employed to fit lines to each group of segments with a tolerance of 0.2 m. Segments shorter than 0.5 m are discarded to maintain data quality.

This process is concluded with a first outcome of detected rails without redundancies; however, the results have no altimetry and need to be improved in order to be usable (Figure 8).

2.4.4. Optimization of Rails

The optimization process deals with the challenges of the Multiview representation for 3D data semantic segmentation. The main challenge of this approach is the reprojection of the labels to the 3D space [29]. Although a first reprojection is carried out using the photogrammetric data, further processing is required to ensure the 2D data accurately match the 3D data. This process is especially relevant for rail mapping as the rails are thin elements, and any error in reprojection could lead to misidentification of the rail in the 3D data and finally affect the accuracy of the final results, especially the altimetry.

The rail optimization process is computed by using the option “3.1 CV and Photogrammetry solution from AI results”. It enhances the accuracy of rail segment detection by applying computer vision techniques and photogrammetry-based corrections. This step ensures that the detected rail segments are properly aligned in 3D space, addressing challenges such as occlusions and image distortions.

• Creation of segment subimages: For each rail segment, a 0.5 m buffer is applied, creating a rectangular bounding box (Figure 9a). The images overlapping with the footprint of the rectangle over the DSM are selected. Images with low overlap (overlap polygon shorter than 0.75 m) are discarded in order to prevent unnecessary noise in the data.
• Occlusion detection: The system checks for potential occlusions by comparing the segment’s edges with the corresponding DSM. If an occlusion is detected (where the image’s projected point is above the DSM point), the segment is discarded.
• Obtaining the segment in the undistorted image by CV: The undistorted image is cropped using the segment bounding box (Figure 9b). Then, the generated image is transformed to a GSD of 1 px, and a homography is applied to ensure the theorical central axis matches the central line of the image. Lines in the image are detected using ED_lib [30].
  All lines with lengths below a threshold (1 m, by default) or with points further than 0.2 m from the central axis of the image are discarded.
  An iterative process is employed to select the most relevant lines as follows:
  • The longest line is selected.
  • All lines within 0.04 m of the longest line are identified.
  • A final line is computed using RANSAC, applied to the points from all the identified lines, with a tolerance equal to the segment selection threshold (0.04 m).
  If multiple lines remain, a final selection ensures that only the most reliable is retained as follows:
  • The longest line is selected.
  • All lines with an angle below 2 degrees (°) to the selected line are identified.
  • Lines with a distance to the main one greater than 0.04 m are discarded.
  • A final line is computed by applying RANSAC to all the points in the selected lines.

The coordinates of the resulting line are obtained in the undistorted image coordinate system (Figure 9c).

In most cases, the initial detections result in lines at both sides of the rail with important gaps. After processing, the result is a single line in the center of the rail with no gaps (Figure 10).

4.

Determination of coordinates in object space: At this point, a segment has been correctly determined in 2D in all images where it is visible. However, the redundancies need to be addressed in order to obtain unique coordinates of the segment in the 3D object space. All the images containing part of the segment are taken into account. A geodetic local coordinate system is defined using the projection center of the image closest to the barycenter of all images. This system is isometric, which avoids problems of different scales.

The 3D coordinate determination follows an iterative process. In each iteration, one image is designated as the primary image, a second image is selected as the secondary, and the remaining images are considered tertiary. The process is as follows:

• The edges of the segment in the primary image (p in Figure 11) are determined as p1 and p2. The process is carried out for each one of these points.
• For the secondary image (s in Figure 11), the projective plane of the segment is determined using the projective center of the image and the two edges of the segment (s1 and s2 in Figure 11a).
• The 3D coordinates of each point (Pi) are obtained as the intersection of the projective ray of the primary image and the projective plane defined by the secondary image (Figure 11a).
• The precision for the point is obtained using the tertiary images (t1, t2 and t3 in Figure 11). Ideally, with no error, the projection of the 3D points in the tertiary images would belong to the axis of the rail in that image. The error is computed as the distance in pixels between the projection of the point in the image and the segment in the image. Each projection of the point in a tertiary image is considered an outlier if the error is above a threshold (3 pixels by default) (Figure 11b).
  The outcome of this process is a set of points that could be part of the rail segment, as well as the number of inliers and the RMSE for each one. These coordinates are transformed into the project coordinate system.
• The obtained point set is ordered using the RMSE and inlier number.
• The final segment is obtained separately in 2D and altitude. All points with an RMSE below 1 pixel and with less than the maximum number of outliers (3 by default) are considered. RANSAC is applied to the selected points with a default tolerance of 0.02 m and a minimum number of inliers. Although the same process is carried out for planimetry and altimetry, different parameters can be chosen.

The presented methodology for the determination of the 3D coordinates is a vital part of the presented algorithm as it processes the CV and photogrammetric results, removing possible outliers and providing a robust tool.

2.4.5. Segment Merging

The process requires adequately merging the detected segments according to their belonging to the same rail. This step is essential for ensuring geometric continuity and consistency in the final 3D model. Merging is conducted based on the spatial proximity and alignment of segment orientations, which are critical for resolving ambiguities, particularly in areas with a high density of rail crossings. The success of this process depends on carefully selected parameters, which must be tailored to the specific characteristics of the dataset. It is recommended to carry out a manual inspection of the previous results in order to facilitate the present task.

Before initiating the merging, a proximity threshold must be defined to determine which segments are considered candidates for merging. By default, segments within a maximum distance of four tiles (equivalent to approximately 15 m) are considered. This assumption is based on the low likelihood of a rail being invisible for more than three continuous tiles.

Next, segments with significant overlap and close proximity are analyzed. If two segments meet all the following criteria, they are considered duplicates, and the shorter segment is removed:

• The angle between the segments is less than 2°.
• The segments overlap. This is determined by projecting the edges of one segment orthogonally onto the line defined by the other segment. Overlap is confirmed if at least one projected point lies within the second segment.
• The orthogonal distance between the segments is less than 0.105 m (1.5 times the width of the rail head), measured from the edge points.
• Once duplicates are removed, the process continues by merging segments that are confirmed to belong to the same rail. This is carried out under the following conditions:
• The angle between the segments is less than 2°.
• The orthogonal distance between the lines defined by the segments is below a threshold that increases relative to the distance between tiles. For consecutive tiles, this threshold is 0.13 m.

This process is carried out iteratively in both directions along the rail. If multiple candidates meet the criteria in a given direction, the segment with the shortest distance is selected.

A new field is added to the attribute table, assigning a unique rail ID to each group of merged segments, ensuring that all segments belonging to the same rail share the same identifier. Finally, altimetric data are integrated into each rail by using DSM and DTM values. These values enable a comparison with the altimetric data obtained from the Multiview photogrammetry, ensuring consistency in the 3D model.

2.5. User Interaction

Since the tool is integrated within QGIS, it benefits from the software’s intuitive user interface, allowing for easy interaction with geospatial data. This enables users to not only visualize both the intermediate and final results but also to overlay them with additional layers, such as orthomosaics or DSM, for better interpretation. The system provides access to attributes of the extracted rail segments, including their 3D coordinates, which can be readily reviewed and analyzed.

Furthermore, QGIS offers robust functionality for manual data modification, allowing users to make necessary corrections to any intermediate results, ensuring that possible errors are not propagated to the next steps. This is particularly useful in some steps, such as (i) validating the quality of the detected rail segments and eliminating erroneous instances before proceeding with data merging and (ii) manually merging segments that were not correctly linked by the automatic process.

Two options have been incorporated into the plugin: one for the deactivation of incorrect detections, and a second one for merging different segments. This editing tool only requires users to identify the segments and press “join”. The database information will be automatically updated.

The flexibility of QGIS ensures that users can maintain high accuracy in their analyses by customizing the results as needed.

3. Experimental Results

A photogrammetric UAV flight was carried out at the access point to the Joaquín Sorolla railway station in Valencia, Spain (Figure 12). The area contains approximately 45 km of rails. Due to the proximity of the station, the area includes several railways with complex crossings and areas of occlusion.

At the time of this study, plans were underway to bury the railway tracks. To support these works, it was necessary to generate detailed cartography of the area with a particular focus on the tracks. The cartography must meet the requirements established by the Spanish Railway Infrastructure Administrator (ADIF) and the normative NAP 1-2-2.0 by [31] for this type of work, with a scale of 1:500 and a GSD of 0.07 m.

The goal of this experiment was to evaluate the capacity of drones and the developed methodology for the automatic mapping of railway tracks at the required scale of 1:500. The map specifically created for the project by the company awarded the contract served as the ground truth for validation.

All processing was carried out using a computer equipped with a CPU Intel Core i7-6800K CPU with 3.40 GHz, 64 Gb of RAM and an NVIDIA GeForce RTX 3080 GPU. However, a dedicated CPU is not required, and most computational demands arise from processing with Agisoft Metashape.

3.1. Ground Truth Data

The cartography generated for the railway track burial project is used as ground truth data. This cartography, produced by the company responsible for the work (Figure 13), was created through field topographic surveying using GNSS and a photogrammetric flight with a GSD of 0.07 m.

In order to map the rails, the coordinates of a sample of points were obtained by on-site topographic surveying. In total, the coordinates of 4132 rail points were obtained. These points are separated distances that range from 30 m in simple areas to a few meters in more complex areas (Figure 13b). The rails were reconstructed using these points. Other points outside of the rails were also measured with GNSS techniques, and they will be used as check points for the photogrammetric process.

The cartography meets the requirements established by the ADIF according to the normative NAP 1-2-2.0 by the Spanish Railway Infrastructure Administrator (ADIF) [32] with a scale of 1:500 and a GSD of 0.07 m.

3.2. Photogrammetric Process

The photogrammetric process was carried out following the requirements for a cartography with a scale of 1:500. Data acquisition was conducted using a UAV Matrice 300 RTK equipped with a Zenmuse P1 RGB camera featuring a 35 mm focal length and a pixel size of 4.39 μm. The flight was executed at an altitude of 84.2 m, achieving a GSD of 9.62 mm. A total of 2188 images were captured during the flight. The flight plan was designed with a nadir camera angle, with the image overlap set to 80% in the forward direction and 60% in the side direction.

The GSD of the photogrammetric flight is significantly smaller than required due to drone flight altitude restrictions, which must remain below 120 m. To maintain a safety margin, a maximum altitude of 100 m is typically applied. This limitation, combined with the capabilities of commonly used cameras, results in a resolution above the requirements of the project.

For georeferencing, 10 ground control points (GCPs) were employed (Figure 14), which is double the recommended number [32], resulting in an accuracy of less than 10 mm (

Table 1. RMSE of control points and check points.

Type	Count	X Error (cm)	Y Error (cm)	Z Error (cm)	XY Error (cm)	Total (cm)
Control	10	0.22	0.39	0.88	0.44	0.99
Check	136	4.00	4.28	2.80	5.79	6.43

). The photogrammetric data were processed using Agisoft Metashape Professional version 2.0.0., obtaining the 3D point cloud, DSM, DTM, undistorted images and orthomosaic.

Geometric validation was carried out using 136 check points (Figure 15). The coordinates of the validation points were determined using precise surveying techniques, employing a Global Navigation Satellite System (GNSS), as explained in Section 3.1 Topographic surveying was carried out to achieve a precision of 7 cm, corresponding to the required scale of 1:500. The mean RMSE for the check points was 6.43 cm (Table 1). This value is below the threshold of 7 cm, and therefore, it is considered acceptable.

3.3. Validation

Three different methodologies were used to validate the obtained results.

Firstly, the ground truth points measured on-site were used to determine the altimetric and planimetric accuracy of the results.

Despite the usability of these ground truth points, their discrete nature limits the possibility to use them to evaluate the totality of the results. In order to overcome this problem, the planimetry was compared to digitalized rails to obtain the detection process accuracy and recall. For the altimetry, the obtained results were fitted to the rails’ ideal shape.

In both cases (altimetry and planimetry), the results were compared with those obtained by AI, without further processing, in order to measure the importance of the methodology.

3.3.1. Determination of Metric Accuracy

In order to validate the metric accuracy of the results, the detected rails were compared against a ground truth composed by the 4132 points measured on-site using the GNSS technique (as explained in Section 3.1).

For each of the points in the ground truth, the closest distance to the detected rails was computed. The mean distance for the whole set of points is 1.9 cm with a standard deviation of 1.6 cm (

Table 2. Planimetric and altimetric errors.

	Mean (cm)	Standard Deviation (cm)
Planimetry	1.97	1.60
Altimetry	4.00	0.86

).

For each 5 m rail segment, the mean height was obtained using the photogrammetric Multiview approach. The mean standard deviation, as the average of the standard deviation of the points, was also obtained for each segment. The altimetric accuracy was computed to calculate the difference between each point and the mean height of the closest 5 m segment. A mean difference of 4 cm was obtained with a standard deviation of 0.86 cm (Table 2).

Both in planimetry and altimetry, the error is well below the desired GSD of 7 cm, and therefore, the accuracy of the methodology is considered adequate for the task.

3.3.2. Validation of Detection Process in Planimetry

For this process, the rails from the cartography are used as ground truth to validate the precision and the recall of the detection process.

A tolerance of 7 cm, corresponding to the rail head width and the desired GSD, was established. Each point in the detected rails is considered correct if the distance to the closest ground truth rail is below 7 cm. The length of rails within the tolerance was computed, and the performance of the detection process was evaluated using True Positives (TPs, correctly detected rails), False Positives (FPs, incorrectly detected rails) and False Negatives (FNs, missed detections), all measured in meters. Precision was calculated as the proportion of correctly identified rail detections out of all detections made by the model (TP/(TP + FP)). Recall measures the proportion of correctly identified rails out of all actual rails present in the dataset (TP/(TP + FN)) (

Table 3. Evaluation results of obtained planimetric coordinates (TP: True Positive; FN: False Negative; FP: False Positive).

	Instances (km)	TP (km)	FN (km)	FP (km)	Recall	Precision
AI	45.37	50.93	8.34	13.93	0.82	0.73
qAicedrone	45.37	40.12	4.93	0.64	0.89	0.98

).

As the presented methodology pretends to improve the rails obtained directly by computer vision, the results of the rail optimization process were compared to the AI detections after these segments were grouped and fitted to lines using RANSAC (as explained in Section 2.4.3) (Figure 8), and the recall and precision are presented in Table 3.

A recall of 0.89 and a precision of 0.98 were obtained using the qAicedrone tool, greatly improving the AI results, with a recall of 0.82 and a precision of 0.73.

A visual validation of the results is presented. For areas of low complexity, the only visible errors are small gaps along the rails. These gaps are caused by occlusions (e.g., posts, wiring and other structures) that hamper AI detection. Due to their size and frequency, they are considered a minor problem. In most cases, small gaps can be ignored as they will be assigned to the same rail in the merging process (Figure 16a).

In the case of areas with complex crossings, bigger gaps can be encountered (Figure 16b), with the extreme case of undetected areas occurring where the angles are too narrow (Figure 16c).

The visual results show correspondence with the numeric results presented in Table 3. A low number of FNs is shown, while the number of FPs is almost negligible.

The highest planimetric distance between each final segment and its corresponding initial AI segment (determined as the closest segment) was calculated. The average maximum distance for the entire dataset is 0.14 m. This discrepancy is significant, as many of the initial AI segments will not align with the actual rail, which typically has a width of 0.07 m. If left uncorrected, this error would pose a major issue in the accurate determination of altimetry.

3.3.3. Validation of Altimetry

An accurate definition of altimetry is crucial to ensure the effectiveness of the presented method as vertical measurements are particularly sensitive to changes caused by land movement and subsidence from heavy loads. Achieving reliable altimetric data is challenging, requiring high precision in both the generation of 3D data and the reprojection process from 2D to 3D. Ensuring this level of accuracy is critical for detecting and analyzing structural changes over time. The analysis of the altimetric accuracy also serves as a measurement of the planimetric accuracy, as errors in planimetry will lead to detected rail points to fall outside of rail heads, causing high altimetric errors.

As altimetric data, the mean height and standard deviation for each 5 m rail segment will be used.

In order to validate the capacity of our method, the final altimetric values were compared to the initial altimetry derived from the AI pre-processed detections and the altimetry derived from the ground truth cartography.

To determine the altimetry of the AI detections, the points corresponding to each AI-detected 5 m segment, after grouping and RANSAC fitting (Section 2.4.3) (Figure 8), were selected. Each point was reprojected to the DSM to obtain its height. The mean height and standard deviation for each segment were obtained, following the same process as that used for the final detections. The planimetric coordinates of the AI detections are not refined, and inaccuracies are to be expected.

For each final segment, the closest AI segment is considered its correspondence.

An overview of the altimetric values is presented in

Table 4. A comparison of altimetric data for the initial AI detections and the qAicedrone results (values are shown in meters).

	Initial AIResults	Final qAicedrone Results
Mean height	12.564	12.633
Std	0.059	0.011

. The mean standard deviation is 5.9 cm for AI data, while for the final qAicedrone results, the deviation drops to 1.1 cm. According to these results, the presented tool greatly increases the accuracy in the altimetry of the AI detections as the standard deviation of the results decreases drastically after processing. The large difference between the initial and final solutions highlights the importance of the qAicedrone corrections.

A visualization of the altimetric profile for one particular rail is shown in Figure 17. The graph shows that initial detections are affected by outliers that clearly do not match with expected gradual changes in rail altimetry. The results after processing show greater robustness with smaller disturbances.

The ideal altimetric profile of rails consists of straight lines and parabolas. To assess the quality of the altimetric data, the measured points were fitted to these geometrical shapes for accuracy verification. Firstly, the rails were segmented into sections of 30 m. A total of 1543 sections were taken into account. For each section, 60 points, at a constant separation, were obtained together with their 3D information.

The points of each section were adjusted to straight lines and parabolas by Least Square Fitting. The initial weights assigned to the points in the adjustment process were based on the estimated precision of the 3D coordinates obtained during the earlier stages of point determination. This weighting approach ensures that points with higher precision have a greater influence on the fitting, thereby improving the accuracy of the final geometric representation.

The RMSE of the Least Square Fitting was obtained for each section. The best result for each section was taken into account, whether it was a straight line or a parabola. In the majority of cases, the parabola provided the best results (65%); in 9% of cases, the data were better fitted to a straight line; and for the rest of the sections (26%), the RMSE was equal for both adjustments.

The mean RMSE for all sections is 2.2 cm. The number of outliers was considered acceptable given the nature of the data: 78% of adjustments had no outliers, while in 96% of cases, the number of outliers was below three (

Table 5. Altimetric validation results.

Number of total sections	1543
Mean RMSE (best option)	2.2 cm
Percentage of sections with zero outliers (best option)	78%
Percentage of sections with less than three outliers (best option)	96%

).

4. Discussion

This paper presents a new tool for robust rail detection and mapping in 3D. The methodology combines object segmentation with photogrammetry and a Multiview approach to robustly determine both the planimetry and altimetry of rails. The tool has been evaluated under real conditions.

In the first steps of the process, a preliminary detection of rails was carried out using AI, particularly YOLOv8. Although these detections can be useful as a first approximation, they do not provide metric quality and cannot be used for accurate rail mapping. The use of computer vision for accurate line detection in the images was applied to improve the accuracy of the coordinates, especially in 3D, where a small 2D inaccuracy can cause the point to be outside of the rail and, consequently, the altimetry to have an error above 15 cm (corresponding to the rail height).

The obtention of accurate altimetric data is considered crucial for the utility of the tool. The height of the points was obtained by photogrammetry using a Multiview approach, and we took advantage of the redundancies.

The evaluation of the tool was carried out in an extremely complex scenario, corresponding to a train station with a high number of crossings, occlusions and other phenomena that hamper accurate rail mapping from aerial data. Some simpler areas with single railways were also present. In this sense, we consider that the presented results can be extrapolated to a wide range of environments, including those with several tracks, multiple crossings and occlusions.

The metric accuracy of the results was computed by a comparison with ground truth data, and the error was below the required GSD of 7 cm both in planimetry and altimetry.

In terms of planimetry, a high recall and considerable precision were obtained, with only small gaps in the detected lines posing minor issues. For the altimetric data evaluation, the tool significantly outperformed YOLO-based detections. The altimetric precision, obtained by fitting the extracted data to straight lines or parabolas, returned an acceptable RMSE of 2.2 cm, approximately 2.5 times the GSD of the images. This level of accuracy is insufficient for tasks requiring millimetric precision, such as the topographic positioning of rails prior to construction or high-accuracy monitoring. However, it is above the requirements for rail mapping at a 1:500 scale, as required by the ADIF [30]. Additionally, it offers notable benefits in terms of time efficiency and automation compared to the more precise but labor-intensive on-site surveying methods. The tool was proven to drastically refine the results obtained by a simple approach based on object detection or even by manual digitalization of the rails. In particular, the tool is capable of providing accurate and reliable altimetric data. This information is especially important because the height of the rail is an important parameter to monitor as it is subject to change over time.

Other studies have employed aerial data and artificial intelligence for railway detection and mapping. Wang et al. [24] introduced NL-LinkNet-SSR for rail detection in images, achieving a similar precision but lower recall in the detection than the proposed method. However, the approach is limited to image coordinates without an assessment of planimetric or altimetric accuracy. Sahebdivani et al. [18] utilized 3D point clouds derived from UAV images, achieving a distance of 3 cm to an ideal model; however, the accuracy was not validated against ground truth data, and the tool was not designed for fully automated railway mapping. Soilán et al. [10] employed terrestrial LiDAR sensors, obtaining planimetric accuracy with errors below 3 cm. Nonetheless, their study did not include an evaluation of altimetric accuracy. Glira et al. [33] proposed a photogrammetric 3D mobile mapping system for railway tracks, utilizing a forward-facing camera mounted on the tracks and equipped with a low-cost GNSS receiver. While their methodology is innovative and offers numerous possibilities, it yields an approximate error of 25 cm, which is significantly higher than that achieved by the proposed method.

One notable advantage of the methodology is its flexibility; users can replace YOLOv8x with other segmentation models, such as U-Net [34,35], ERTNet [36], NL-LinkNet-SSR [24] or even filter-based approaches [17], enhancing the detection pipeline without disrupting the tool’s core functionality. Although the development of a custom architecture for rail detection is beyond the scope of this work, specific architecture could be integrated seamlessly. The detections generated by other architecture will also be refined by qAicedrone, improving the robustness of the results.

One of the main challenges when mapping rails and other infrastructures from aerial data is the presence of occlusions, which can vary from bridges to vehicles or posts. The use of Multiview allowed us to reduce this issue to a minimum, especially for moving elements, such as trains. In future studies, we will advance in the removal of these occlusions by improving the merging of unconnected segments. In cases where it is not possible to avoid occlusion automatically, minimal manual work, using the available options in the qAicedrone plugin, would allow the user to obtain optimal results.

Additionally, the reflective surface of rails poses a challenge for 3D reconstruction as it hinders accurate feature detection and matching. Incorporating lines, accurately extracted through computer vision techniques, contributes to improving the precision of the final 3D data.

The primary limitation of the tool lies in detection gaps in highly complex areas with several rail crossings and narrow angles. As in the case of occlusions, this issue can be mitigated through minimal manual work. It is important to note that, although the tool is automatic, a certain degree of parameter tuning can help to obtain optimal results. Additionally, the tool has implementations that allow manual corrections, which can help improve the results without a significant time investment.

Although the presented tool is an open-source tool, the proprietary software Metashape 2.0.0 was used for photogrammetric data processing. Current efforts are underway to adapt Graphos, an open-source alternative for photogrammetric processing [37], so it can work seamlessly with the presented tool. Graphos is already integrated into OpenDroneMap 3.5.4 as NodeGraphos, facilitating cloud-based photogrammetric processing, which will enhance accessibility and facilitate the workflow for future users.

5. Conclusions

This study introduced an innovative tool for robust and accurate rail detection and 3D mapping, demonstrating its potential to address the challenges of mapping railways in complex environments. By using a combination of AI-based segmentation and photogrammetry with CV and a Multiview approach, the methodology is capable of providing results that meet the requirements for 1:500 scale mapping. The flexibility of the tool allows for integration with different segmentation architectures, ensuring its adaptability and potential for future enhancements. Additionally, the use of open-source frameworks makes the tool accessible and scalable.

While the tool offers significant advantages, such as time efficiency and reliable altimetric data, its limitations in handling detection gaps and complex rail crossings highlight areas for improvement. Minimal manual intervention and parameter tuning remain necessary in specific scenarios, where users are encouraged to carry out the manual removal of outliers and tune the parameters to obtain the optimal result. If required, the authors can be consulted for further guidance. Future developments will aim to address occlusion-related challenges and further refine the integration with open-source photogrammetric solutions. These advancements, combined with the methodology presented, position this tool as a valuable contribution to rail mapping for infrastructure monitoring and management.