Toward the Enhancement of Rail Sustainability: Demonstration of a Holistic Approach to Obstacle Detection in Operational Railway Environments

Abstract

Rail transport plays a crucial role in promoting sustainability and reducing the environmental impact of transport. Ongoing efforts to improve the sustainability of rail transport through technological advancements and operational improvements are further enhancing its reputation as a sustainable mode of transport. Autonomous obstacle detection in railways is a critical aspect of railway safety and operation. While the widespread deployment of autonomous obstacle detection systems is still under consideration, the ongoing advancements in technology and infrastructure are paving the way for their full implementation. The SMART2 project developed a holistic obstacle detection (OD) system consisting of three sub-systems: long-range on-board, trackside (TS), and Unmanned Aerial Vehicle (UAV)-based OD sub-systems. All three sub-systems are integrated into a holistic OD system via interfaces to a central Decision Support System (DSS) that analyzes the inputs of all three sub-systems and makes decision about locations of possible hazardous obstacles with respect to trains. A holistic approach to autonomous obstacle detection for railways increases the detection area, including areas behind a curve, a slope, tunnels, and other elements blocking the train’s view on the rail tracks, in addition to providing long-range straight rail track OD. This paper presents a demonstration of the SMART2 holistic OD performed during the operational cargo haul with in-service trains. This paper defines the demonstration setup and scenario and shows the performance of the developed holistic OD system in a real environment.

1. Introduction

Rail is often considered as one of the most sustainable modes of transport, and this viewpoint is supported by several factors such as the following: rail transport is highly energy efficient compared to other modes like road or air transport as trains can carry large numbers of passengers or goods using relatively less energy per unit of distance traveled; rail transport typically produces fewer emissions per passenger or per ton of freight compared to cars, trucks, or airplanes; railways help alleviate road congestion by moving significant volumes of people or goods efficiently along dedicated tracks, reducing the need for additional road infrastructure and the associated environmental impacts; rail infrastructure tends to have a longer lifespan and requires less maintenance compared to roads, resulting in a lower environmental impact over time; and railways often form the backbone of public transit systems, encouraging people to choose more sustainable modes of transportation over private cars. In addition to the previously mentioned factors, there is also a safety benefit as rail transport is generally safer than road transport, leading to fewer accidents and associated environmental damages. However, there is always room for improvement in terms of rail safety. Namely, collisions between trains and obstacles on or adjacent to railway tracks pose significant risks to both passengers and the environment. Detecting these obstacles in a timely manner is crucial for preventing accidents and minimizing environmental damage [1].

Autonomous obstacle detection (OD) plays a pivotal role in enhancing railway safety by preventing collisions, reducing human error, and improving operational efficiency [2,3]. It is a critical component of modern railway systems aiming for safer, more efficient, and more reliable transport networks. The majority of OD systems in railways are deployed as on-board systems installed on trains themselves, consisting of different on-board sensors such as cameras and lidar sensors [4,5,6,7,8]. These on-board systems are designed to continuously monitor the tracks ahead and detect any obstacles or hazards that may pose a risk to the train’s safe operation. However, current OD systems neglect the fact that even the perfect on-board OD system that reliably detects obstacles ahead of the train is not sufficient for covering the necessary long-range obstacle detection as it cannot “see” potential obstacles in the areas that are not in the field of view (FoV) of the on-board system such as the areas around curves. One possible solution to this problem has been suggested with the recently completed project SMART2, funded by Europe’s Rail (formerly Shift2Rail) [9]. It is the inclusion of two OD systems, a trackside (TS) and an Unmanned Aerial Vehicle (UAV)-based system, used to complement the on-board OD system and to allow a holistic approach to OD in railways.

In this paper, the innovative SMART2 OD holistic system is presented with emphasis on dynamic field tests to demonstrate the system performance in the operational railway environment from the last year of the SMART2 project. The demonstration activities involved the showcasing of the OD system functionality, accuracy, and reliability in detecting obstacles in real-time scenarios. As explained in the following, these activities involved obtaining permissions for demonstration, the selection of a demonstration site, preparation and setup, scenario design, a live demonstration of the data recording and data processing for the purpose of obstacle detection, and distance calculation. Detailed descriptions of the SMART2 solutions for obstacle detection and distance calculation can be found in a number of publications such as [10,11]. This paper focuses on the results of the demonstration of this SOLUTIONS, and on the analysis of the OD system performance and limitations based on the obtained demonstration results.

The rest of this paper is organized as follows. In Section 2, a short overview of the sub-systems of the SMART2 holistic OD system is provided. Section 3 presents all elements of the demonstration setup in the operational environment. The demonstration results are presented and discussed, respectively, in Section 4 and Section 5. The paper ends with conclusions (Section 6).

2. Related Work

Recent advancements in Artificial Intelligence (AI) and sensor technology have resulted in significant work on OD in railways [12,13]. Validating these autonomous obstacle detection (OD) systems in real-world scenarios is crucial for ensuring their reliability, safety, and practicality. However, while there is an extensive number of published results of research and development in obstacle detection systems for railways, there has been relatively less focus on their validation and demonstration in operational railway environments. The majority of the papers published so far that include the evaluation of developed OD systems have presented evaluation results based on the data publicly available on the Internet [14,15] or on custom-made datasets [16,17]. The latter involves off-line-generated results applied on datasets recorded in real environments, for example, using a specially designed test train vehicle such as in [18]. Some of the performed real-time field tests have been reported in the form of articles/news [19], without providing any details of the employed methodology. Several reasons contribute to this gap in the literature such as the complexity of operational environments, which require significant effort and resources, and safety concerns that make obtaining permissions for trials and experiments in live railway settings very challenging. The recently completed SMART2 project has contributed to closing the gap between the development and practical implementation of obstacle detection systems in operational railway environments by dedicating significant resources to real-world validation and demonstration activities. A description of the performed SMART2 field tests, including a description of the evaluation methodology, is provided in the following, along with a discussion on the evaluation results.

3. SMART2 Holistic Obstacle Detection (OD) System for Railways

An overview of the SMART2 holistic OD system concept is presented in Figure 1. The SMART2 concept is a possible solution to the problem that even a faultless on-board OD system cannot reliably solve, as it cannot “see” potential obstacles in the area that are not in its field of view (FoV). The SMART2 solution is based on complementing the on-board system using UAV-based and trackside (TS) sub-systems to increase the detection area by including areas that are not visible to the on-board system such as areas behind the curves, slopes, tunnels, and other elements blocking the train’s view on the rail tracks. It is assumed that the additional TS and UAV sub-systems are located so as to have these areas in their fields of view (FoVs), as illustrated in Figure 1. The requested detection area in the SMART2 project was up to 2 km, as this range covers the braking distances of different types of trains, both freight and passenger ones.

The SMART2 on-board OD sub-system is a multi-sensory on-board system integrated in a custom-made housing that protects the sensors, holds them in the correct alignment, and enables the easy mounting/dismounting of the unit to/from a locomotive. The multi-sensor SMART2 on-board system consists of different vision sensors enabling obstacle detection under day and night illumination conditions. Three RGB zoom cameras and a thermal camera are accompanied by a LADAR vision unit (LAser Detection Active Ranging), developed within the SMART2 project, which is based on Short-Wave InfraRed (SWIR) vision technology. This SWIR-based vision sensor augments the capabilities of the RGB and thermal cameras and increases the performance of the on-board OD sub-system, improving the functional reliability in different illumination and weather conditions, including challenging ones.

The key component of the trackside OD sub-system is an advanced 3D laser optic (lidar) sub-system that enables optimization of the operational parameters of the OD system (detection range, scanning angles, scanning resolution, and scanning frequency). This type of TS sub-system, developed within the scope of the SMART2 project, is intended for OD at level crossings.

The SMART2 airborne sub-system consists of a nested Unmanned Arial Vehicle (UAV) (drone), which is placed on a pillar next to the rail track at strategic locations such as tight curves, gorges, or landslide prone locations. This UAV-based OD sub-system is intended for obstacle detection in the areas outside the FoV of the on-board OD sub-system such as areas beyond curves. The UAV (drone) selected for the SMART2 UAV-based OD sub-system was a DJI Phantom 4 RTK [20], with an RGB camera as the main perception sensor.

As illustrated in Figure 1, the on-board, TS, and UAV-based OD sub-systems are sending information about obstacles detected in their FoVs to a cloud-implemented Decision Support System (DSS). The SMART2 DSS consists of decision-making algorithms and corresponding cloud infrastructure, which handles security aspects, the integration of data, data access and flow, storing, and also provides technical and graphical user interfaces (GUIs). A high-level view of the DSS cloud infrastructure is shown in Figure 2. As can be seen, it consists of several components with different responsibilities such as providing APIs for receiving metadata from the OD sub-systems (the on-board, trackside, and UAV-based sub-systems), providing information on warnings and failures from the OD sub-systems, providing databases for collecting relevant data from the OD sub-systems as well as from other external data (weather, train related data, digital railway map), guaranteeing sufficient data protection and access to the cloud infrastructure, and providing cloud resources for the implementation of decision-making algorithms that refer to making the final decision on obstacles on the rail tracks.

Starting from three independent inputs from the OD sub-systems, the SMART2 DSS makes the final decision about the obstacles on the rail tracks, their locations in front of the train, and distances to the train, and because of this, the DSS is also called the obstacle detection decision-making (ODDM) system. An entry view of the DSS GUI (Figure 3) shows the user dashboard displaying the sub-system status information, the connectivity status of the logged-in user, a map with the current train position, a table for decision scenarios, and UAV starting information. The table for decision scenarios shows the detection results of the OD sub-systems per individual object, as shown in Figure 3, where each detected object is marked as whether it is within the region of interest (ROI), containing the rail tracks and the vicinity of the tracks, or not.

4. Demonstration Setup in Operational Environment

The assessment of the SMART2 holistic OD system took place through a series of dynamic field tests in an operational railway environment held in Summer 2022. These dynamic field tests were executed to assess the functionality of the fully integrated OD system. In the dynamic tests, all sub-systems were set-up as in real-scenario environmental situations, as illustrated in Figure 4: the on-board sub-system mounted on the train, TS set-up on a level crossing, and UAV observing sections of railway tracks such as those beyond the curves.

The OD on-board demonstrator was mounted on a locomotive series 441/444 owned by Serbia Kargo (“Cpбиja Kapго”), which operated with attached wagons along the Serbian segment of Pan European Corridor X, specifically between the cities of Nis and Ristovac, en route to Thessaloniki, Greece.

4.1. General Methodology

Considering the architecture of the obstacle detection system, the complexity of the different sub-systems, and the interactions between them in the different use cases, it was necessary to develop a methodology for evaluating the overall performance of the system, with respect to requirements and KPIs that have been identified and/or defined for designing and implementing the system.

Typical options for evaluating novel railway/transport technologies before extensive and demanding trials in real environments include simulations, laboratory testing, testing in a representative environment (such as on a dedicated test track), and testing under controlled conditions in an operational environment. Considering the characteristics of the OD system, an approach combining testing in a representative environment (that provides and/or replicates specific conditions) and testing in an operational environment was considered.

The first task of the evaluation methodology was to design a test program that allowed gathering and/or generating data to assess the performance of the technology demonstrator against the requirements. Performing testing in an operational environment, and to a lesser extent in a representative environment, not only enabled the evaluation of the results to determine whether the prototype system meets all the requirements, but also provided a wider perspective of the potential viability and performance of the system in real railway environments, enabling the potential impact of a large-scale introduction of such a system to be better assessed.

The evaluation methodology, which comprised nine key phases, is illustrated in Figure 5. The first two steps involved the analysis of the requirements identified and/or defined for the OD system (general ones and those that apply to the demonstrator) with respect to the fourth step, that is, to define the tests that would generate the data for the evaluation of the performance of the demonstrator. Parallel to the first, second, and fourth steps, the third step involved the generation of datasets, which were used in the development of the detection technologies; these data were also needed for designing the test scenarios, which was the fifth step. The sixth step was the implementation of the test program, during which the seventh step (collecting the test data) occurred. The eighth and ninth steps were the analysis of the data from the tests against the metrics for the tests and the evaluation of the performance of the system, respectively.

4.2. Obtaining Permission for Dynamic Field Tests

Every dynamic test was conducted subsequent to obtaining the necessary permits. These permits encompassed the usage of a locomotive from Serbia Cargo, the installation of the SMART2 OD on-board system onto the test locomotive, and the authorization for testing the OD demonstrator in the operational environment. The Infrastructure Manager—Infrastructure of Serbian Railways issued the permit for testing in the operational environment of the OD demonstrator.

Permits were secured following the procedural guidelines. As per the established procedure, in March 2022, SMART2’s partner, the University of Nis, initiated the process by submitting a formal request to Serbia Cargo for the utilization of a locomotive from the 441/444 series. The request was substantiated with comprehensive documentation, including a detailed description of the OD on-board demonstrator and a thorough account of the procedures involved in its mounting and dismounting onto/from the locomotive. Specifically, the submitted documents comprised the following:

• Technical drawings depicting the sensor housing and the mounting elements for the OD demonstrator;
• Detailed specifications outlining the on-board demonstrator, its constituent components, and the mounting elements;
• Results from random vibration analyses conducted on both the sensor housing and the mounting elements, in accordance with EN 61373:2010–rolling stock equipment–shock and vibration tests for a Category 1 Class B device [21];
• Verification that the on-board demonstrator was designed in compliance with the EN 50155:2007 standard [22];
• A three-dimensional Computer-Aided Design (CAD) model illustrating the on-board demonstrator when installed on the vehicle;
• Photographs showcasing the manufactured sensor housing and mounting elements;
• Comprehensive specifications of the dynamic test protocols and the scenarios employed in testing.

A committee of safety experts and engineers from Serbia Cargo was formed with the task to review the submitted documentation. As a result of this procedure, the committee requested that the trial OD demonstrator assembly should be mounted onto a 441/444 series locomotive and that a trial run should be conducted to finally approve a permit for tests in an operational environment. The mounting of the OD on-board demonstrator and the test run were successfully completed in May 2022, following which Serbia Cargo issued permits for further testing in operational conditions.

Additional documentation was supplied to the Infrastructure of Serbian Railways for the purpose of obtaining the Infrastructure Manager’s permit for testing the holistic OD system in an operational railway environment. This documentation included the following:

• A description outlining the protocols and scenarios for dynamic tests;
• Specifications of the SMART2 Unmanned Aerial Vehicles (UAVs) and the trackside (TS) demonstrator, as well as information about the TS demonstrator mounting;
• Flight plans for the UAVs;
• The planned UAV mission takeoff and landing positions and the installation position of the TS system in relation to the level crossing.

Based on the documentation submitted for UAVs and the TS system, the Infrastructure of Serbian Railways issued permits for UAV usage and data recording, as well as for the installation of the TS system. Furthermore, based on a successful trial, the Infrastructure of Serbian Railways issued a permit for testing in an operational environment.

4.3. Test Track for Holistic Obstacle Detection Technology Demonstration in Operational Environment

The SMART2 OD technology demonstration in a railway operational environment took place on the Serbian segment of Corridor X, extending from the city of Nis (Red Cross station) toward North Macedonia (Ristovac station), as illustrated in Figure 6. The entire test track spans a length of 120 km, where cargo trains are permitted to reach a maximum speed of 80 km/h. For all operational tests involving SMART2 OD, the on-board demonstrator was mounted to the test locomotives at the Red Cross station, serving as the starting point for all operational run tests.

The train, with the on-board OD mounted on its locomotive, then progressed toward Nis Marshalling Yard, where cargo wagons were coupled. With attached wagons, the train then embarked on its in-service run toward the border of North Macedonia. This route and procedure were integral to the comprehensive evaluation of the SMART2 technology in an operational railway environment.

The enlarged section on Figure 6, showing the area between the stations Kocane and Pecenjevce, was specifically chosen as part of the track where the evaluation scenario is defined. This selection was made based on the recommendation of experts from the Infrastructure of Serbian Railways and on the fact that the chosen segment contains sections that are appropriate for the demonstration of all three OD sub-systems. Namely, the chosen segment features a long straight portion between Pukovac and Lipovica along with multiple secured and unsecured level crossings, an S-curve after station Lipovica, and a highly frequented level crossing located in the curve just before Pecenjevce. The selected track segment underwent digitalization to create a precise digital map containing accurate global coordinates of the track and various infrastructure elements. The digital mapping process involved recording the track coordinates using an RTK GNSS rover mounted on the midsection of the frontal profile of the locomotive (Figure 7a). Data from the track construction documentation, supplied by the Infrastructure of Serbian Railways, also contributed to the creation of the digital map (Figure 7b). GPS coordinates of the track were recorded during the in-service train run in May 2022 (Figure 7c), accompanied by 4K footage of the track from the train cabin (Figure 7d). This approach ensured the creation of a highly accurate digital map down to the centimeter level, depicting the precise positions of all infrastructure elements and the exact track profile.

4.4. Scenario for OD Technology Demonstration in Operational Environment

Building upon the adopted evaluation methodology, metrics, and procedures, as well as upon the setup in the railway operational environment, a comprehensive evaluation scenario was formulated to showcase the SMART2 OD developed technologies in a pertinent setting. As previously outlined, the section between stations Kocane and Pecenjevce was chosen for the scenario setup, particularly focusing on the segment between Pukovac station and the Pecenjevce crossing, illustrated in Figure 8.

In a defined scenario, the train progressed from Pukovac toward Pecenjevce, and SMART2 UAVs together with the TS system and obstacles were strategically positioned along the train’s path. The train was equipped with a 4G modem to facilitate communication with the SMART Decision Support System (DSS). The demonstration scenario involved the use of two UAVs, as depicted in Figure 8, contributing to the comprehensive evaluation of SMART2 technologies in a real-world operational railway environment.

The first UAV was responsible for monitoring the area behind the curve following the Lipovica crossing. Positioned at the Lipovica crossing UAV takeoff point, the UAV initiated its mission based on the DSS signal. Namely, based on the train’s GPS location and on the digital map, the DSS initiated the UAV takeoff when the train was approaching a curve so as to view possible objects on the rail track out of the train’s visibility range. The UAV commenced its flight from the Lipovica crossing traversing along the train’s path toward Pecenjevce, as illustrated in Figure 9. Upon reaching the endpoint of its mission, the UAV hovered in place while the train passed, providing oversight of the curve following the Lipovica crossing. The flight length during the mission of the first UAV was planned to be 1 min with a distance traveled of 510 m.

The second UAV was assigned to survey the area preceding the Pecenjevce crossing. Its takeoff point was strategically placed at the Pecenjevce crossing. Initiated by the DSS signal, the UAV embarked on its mission by flying from the Pecenjevce crossing, moving in the opposite direction to the train’s path toward Lipovica, as depicted in Figure 10. While in flight, the UAV observed the area ahead of the Pecenjevce crossing. The flight length of the second UAV was planned to be 35 s, with a distance traveled of 208 m.

The SMART2 TS system, which was based on the lidar sensor, was positioned at the Pecenjevce crossing overlooking the level crossing, as shown in Figure 11, where the TS system is marked with a red rectangle.

In addition to the 4G modem installed on the train, two extra 4G modems were strategically positioned at the Lipovica and Pecenjevce level crossings, as depicted in Figure 8. These additional modems enabled communication between the UAVs, the TS system, and the Decision Support System (DSS).

For the evaluation scenario, four distinct classes of obstacles were selected and carefully placed on the tracks in close proximity to marked infrastructure elements. The precise positioning allowed for accurate distance measurements from the moving train, given that the positions of infrastructure elements were known from the SMART2 digital map of the test section. The positions of the obstacles in Figure 8 are denoted in a km + m format, following the notation used in the track construction data from the Infrastructure Manager—Infrastructure of Serbian Railways. The selected obstacles and their positions along the track were as follows:

• A bicyclist moving across the level crossing at Brestovac (268 km + 318.00 m);
• A fallen tree situated directly on the track, located 265.81 m ahead of the Lipovica level crossing (270 km + 580.49 m);
• A car moving across the Lipovica level crossing (270 km + 846.30 m);
• A pedestrian moving across the track behind the curve, 449.4 m after the Lipovica level crossing (271 km + 295.70 m);
• Cars and pedestrians moving across the Pecenjevce level crossing (273 km + 221.32 m).

The movable obstacles from the list above were members of the SMART2 project team (pedestrians) or vehicles (car and bicycle) driven by the SMART2 project members. A static object, a fallen tree, was placed on the rail tracks and safely moved away from the rail tracks by the SMART2 project team members. The total length of the track covered between the first and the last obstacle was 4903.32 m, surpassing the required obstacle detection range of 2 km. This defined scenario allowed for a comprehensive evaluation of the SMART2 OD system as it enabled the detection of obstacles that were not directly within the line of sight of the on-board system. Notably, the S-curve after the Lipovica level crossing is situated in an urban area with multiple pedestrian crossings, and the Pecenjevce level crossing is very busy and located just 50 m behind the curve.

5. Results of Demonstration in Operational Environment

The testing of the SMART2 solution for a holistic approach to OD in an operational railway environment was performed with a 444-017 locomotive, with a gross mass of 917 t and length of 483 m (Figure 12).

Table 1. Results from the dynamic testing of the SMART2 OD system in an operational environment.

Location	Sensor	Ground Truth, m	Detection, m	Error, %
	RGB1	-	-	-
	RGB2	292	person: 317.4	person: 8
			bicycle: -	bicycle: -
	SWIR	368	person: 391.4	person: 6.4
			bicycle: 381.3	bicycle: 3.6
	Thermal	261	person: 244.8	person: −6.2
			bicycle: 240.3	bicycle: −7.9
	RGB1	- *	-	-
	RGB2	-	-	-
	SWIR	579	535.9	7.4
	Thermal	-	-	-
	RGB1	Person 1: 204 Person 2: 192 Car: 204	Person 1: 214.3 Person 2: 170.7 Car: 223.5	Person 1: 5 Person 2: 10.8 Car: 9.4
	RGB2	Person 1: 212	Person 1: 247	Person 1: 16.5
		Person 2: 199	Person 2: 214	Person 2: 7.7
		Car: 212	Car: 246.3	Car: 16.2
	SWIR	Person 1: 896	Person 1: 900.5	Person 1: 0.5
		Car: 1180	Car: 1332.8	Car: 12.9
	Thermal	Person 1: 398	Person 1: 358.3	Person 1: −9.8
		Person 2: 233	Person 2: 299.4	Person 2: 28.5
		Car: 470	Car: 428.3	Car: −8.9
	UAV1	1200	1204.3	0.4
	RGB1	Person 1: 225	Person 1: 210.5	Person 1: 6.4
		Person 2: 100	Person 2: 102.7	Person 2: −2.7
		Person 3: 106	Person 3: 109.5	Person 3: −3.3
		Car: 54	Car: 51.8	Car: 5.1
		Truck: 48	Truck: 49.1	Truck: −2.3
	RGB2	Person 1: 220	Person 1: 255	Person 1: −15.9
		Person 2: 162	Person 2: 192	Person 2: −18.5
		Person 3: -	Person 3: -	Person 3: -
		Car: -	Car: -	Car: -
		Truck: -	Truck: -	Truck: -
	SWIR	Person 1: 161	Person 1: 170	Person 1: −5.6
		Person 2: 130	Person 2: 140.1	Person 2: −11.2
		Person 3: -	Person 3: -	Person 3: -
		Car: -	Car: -	Car: -
		Truck: -	Truck: -	Truck: -
	Thermal	Person 1: 180	Person 1: 166.9	Person 1: 7.3
		Person 2: 185	Person 2: 178.7	Person 2: 3.4
		Person 3: -	Person 3: -	Person 3: -
		Car: -	Car: -	Car: -
		Truck: -	Truck: -	Truck: -
	UAV2	Car: 1800	Car: 1804.93	Car: 0.3
	Trackside	608	object detected	-

* Not detected by sensor.

shows the results of the detection of obstacles positioned along the test track as defined in the testing scenario. The ground truths and detection distances were reported in relation to the moving train. The table reports the results of the first detection by a particular sensor, that is, the distance is reported when the obstacle was detected by a particular sensor for the first time (i.e., the distance was estimated using the frame in which the obstacle was detected for the first time). Table 1 also reports the error in distance estimation, where a positive error indicates that the detection distance is longer than the ground truth distance and a negative error indicates that the estimated distance is shorter than the ground truth.

For the sake of visibility, the considered obstacles are illustrated in Table 1, namely, illustrations of obstacles are overlaid onto the original photographs of the test sites. The related original images recorded by the SMART2 OD sensors, overlaid by processing results, are presented in Section 5.

6. Discussion

The results reported in Table 1 acquired during the dynamic tests in an operational environment demonstrate the SMART2 OD system’s capability to detect objects and potential obstacles using the integrated perception sensors of all sub-systems. The potential obstacles were detected by at least one perception sensor, even in situations where the potential obstacle was not in the FoV of the on-board system.

6.1. Bicycle at Brestovac Level Crossing

The bicycle and the person pushing the bicycle at the Brestovac level crossing (position 268 + 318) were not detected by the RGB1 camera. The person pushing the bike moved from the level crossing before the RGB1 sensor entered its effective range. The same person was successfully identified while standing next to the level crossing, as shown in Figure 13a. The RGB2 camera failed to detect the bicycle in the same scenario point, despite successfully identifying the person pushing the bike Figure 13b. This discrepancy may stem from various factors such as lighting conditions, RGB camera settings, or specific characteristics of the object. The demonstration of the SMART2 OD system in the operational environment showed that to address this issue, it is important to use different vision technologies such as SWIR or thermal, as well as different zoom factors in camera settings. The SWIR and thermal camera, as it is more resilient to lighting and environmental conditions, successfully detected both the person and the bicycle in the same scenario point, as shown in Figure 14a (SWIR) and Figure 14b (thermal). Due to a larger objective and in turn zoom of the acquired images, the SWIR camera could detect the person and the bicycle at a much larger distance.

6.2. Fallen Tree in Front of Lipovica Level Crossing

The fallen tree in front of the Lipovica level crossing (position 270 + 580.49) was not detected by the RGB1, RGB2, or thermal cameras as the tree was removed from the track due to safety reasons before the noted cameras entered their effective range. The results presented in Figure 15a show that the SWIR camera with a larger zoom and thus a larger range could detect a relatively small tree of 2 m in height, as well as persons occluded by vegetation, even at a range larger than 500 m. With larger objects such as cars, the SWIR camera could detect objects positioned at the next scenario point (Lipovica level crossing), even at 1.2 km, as shown in Figure 15b. Even people at the Lipovica level crossing could be detected at a distance of 900 m, as shown in Figure 15c.

6.3. Car and Pedestrians Crossing the Track at Lipovica Level Crossing

The car and pedestrian crossing the track at the Lipovica level crossing (position 270 + 846.3) was successfully detected by all the sensors of the SMART2 OD on-board system. As already discussed in the previous subsection, the SWIR camera detected the car crossing the track and the pedestrians at very large distances of 1.2 km and 900 m, respectively. The larger detection error of the thermal camera for Person 2 is a consequence of the position of said person in the image as they are barely visible due to the occlusion of a crossing car and Person 1 (Figure 16).

6.4. Pedestrian behind the Curve

The strategically positioned UAV could detect potential obstacles while flying over the rail track in front of the moving train, as shown in Figure 17, with very good accuracy as the error in distance estimation was only 0.4%. The UAV mission start, initiated by the DSS based on the moving train position, enabled detection in sections of the track that were not in the FoV of the moving train and its on-board obstacle detection system. This greatly enhances safety, as the flying-over UAV can inspect critical sections before the train. The flying-over UAV can even monitor the behavior of the moving object as it has an unobstructed view from above. Figure 17a shows the person entering an area where collision with the train is possible, while Figure 17b shows the same person leaving the noted area.

6.5. Cars and Pedestrians at Pecenjevce Level Crossing

The obstacles at the Pecenjevce level crossing (position 273 + 221.32) were detected by multiple sensors of the SMART2 OD on-board system. Person 3, the car, and the truck were not detected by the RGB2, SWIR, or thermal cameras due to a larger zoom level of the noted sensors and thus a narrower FoV. As the level crossing is directly behind the curve, the vegetation blocks the view at the crossing, thus limiting detection at larger distances, as shown in Figure 18. The movement of the camera needed to focus the FoV on the track highly depends on the track configuration. The view of the track is often blocked by the track configuration (gorges, passages, etc.), vegetation, infrastructure elements, or buildings in urban areas. It was proved during multiple tests performed in the frame of the SMART2 project that using multiple cameras with different zoom levels for an on-board OD system achieves better results than directing the camera view at the track using gimbals, for instance.

Moreover, a hovering UAV can greatly enhance safety on level crossings as it can monitor a large section of the track, as well as the surroundings of a level crossing, as shown in Figure 19. The benefits of using a holistic approach are also quite obvious from the analysis of the results, as the UAV view on the track is not blocked by the track configuration.

The reported results collectively highlight the system’s effectiveness in detecting objects across diverse situations in operational environments and utilizing different types of sensors, reaffirming the reliability and versatility of the SMART2 OD system.

The demonstration in the operational environment highlighted that, through the strategic placement of UAVs and TS sensors, it is feasible to cover distances much larger than the requested 2 km [16]. This illustrates that, with appropriate coordination among the three sub-systems, comprising the on-board system, the UAV-based system, and the trackside system, the long-range detection of more than 2 km ahead of the train can be achieved. Importantly, the SMART2 OD system demonstrated that the on-board system alone possesses the capability to detect obstacles at up to 1.3 km. This emphasizes the effectiveness and autonomy of the on-board system in detecting potential obstacles, contributing to enhanced safety measures in railway environments. The demonstration of the SMART2 holistic OD system was performed during good weather conditions by chance. From the test previously performed with individual systems, and while working on system integration, it was proved [8] that the thermal camera and the SWIR camera are overall much more suitable than the RGB cameras for providing reliable detection in challenging environmental and lighting situations. Figure 20a shows a object detection result of the SWIR camera for images taken in complete darkness at a distance larger than 500 m. The person in the image was no longer visible to the naked eye, but the SWIR camera was still able to detect the person, only from the illumination of the surroundings from the locomotive headlights. The thermal camera also provides reliable detection in almost all external environmental conditions, enabling object detection at any time. Figure 20b shows a detection result from images that were also taken in complete darkness. Even though the performance of the system is degraded in challenging environmental or lighting conditions, the overall OD system performance is still better than the human counterpart, i.e., the locomotive driver.

The functionality and correctness of the trackside sensor were tested under different environmental and illumination conditions. The sensor functioned normally in all these cases, even when exposed to direct sunlight. The trackside sensor is laser-based, and it was unaffected by environmental and illumination conditions during all the performed tests.

Furthermore, the new generation of professional UAVs can withstand bad weather, and, due to advanced obstacle detection capabilities, can even perform flight campaigns during the night. As their vision sensors can be the same as for the on-board system (RGB, Thermal, SWIR), it is reasonable to assume that they will have identical performances to the sensors of the onboard system.

7. Conclusions

Autonomous OD systems have the potential to make railways safer and more efficient. Testing OD systems in an operational railway environment is crucial not only for ensuring safety, but also for validating their performance, reliability, and feasibility, as well as for providing feedback for their continuous improvements. In spite of the importance of testing OD systems in operational environments, there is a lack of information about such testing. Therefore, a particular contribution of this paper is the description of all steps of the demonstration of an innovative OD system in a real-world railway environment starting from obtaining permissions for the demonstration and selecting a demonstration site, through the scenario design to processing recorded data for the purposes of obstacle detection and distance estimation.

The demonstrated OD system is a holistic system developed within the SMART2 project that consists of three OD sub-systems: on-board, UAV-based, and trackside. The analysis of the results of the conducted demonstration in a relevant railway environment led to the conclusion that the SMART2 OD system detects objects and potential obstacles using all three sub-systems in a complementary manner, thus enlarging the detection distance range significantly. The obstacles from the demonstration scenario were detected with at least one of the perception sensors of the SMART2 OD system, which proves this system’s reliability. In most cases, the distance estimation error was below 10%, which is considered very good, bearing in mind the effective range of the system. The total length of the covered test track was almost 5 km, which proves that the SMART2 OD demonstrator can detect obstacles at a range greater than 2 km. This was one of the main requirements of the SMART2 project, namely, that the detection range was sufficiently long so as to be larger than the train’s breaking distance. However, further research is necessary in terms of the SMART2 OD system reliability and integrity during challenging environmental and lighting conditions. The research should be focused on using UAVs with SWIR or thermal sensors, as well as on proving that although the system performance degrades in such conditions, it is still better than in the case of a human driver.