Augmented Reality Interface for Adverse-Visibility Conditions Validated by First Responders in Rescue Training Scenarios

Abstract

Updating the equipment of the first responder (FR) by providing them with new capabilities and useful information will inevitably lead to better mission success rates and, therefore, more lives saved. This paper describes the design and implementation of a modular interface for augmented reality displays integrated into standard FR equipment that will provide support during the adverse-visibility situations that the rescuers find during their missions. This interface includes assistance based on the machine learning module denoted as Robust Vision Module, which detects relevant objects in a rescue scenario, particularly victims, using the feed from a thermal camera. This feed can be displayed directly alongside the detected objects, helping FRs to avoid missing anything during their operations. Additionally, the information exposition in the interface is organized according to the biometrical parameters of FRs during the operations. The main novelty of the project is its orientation towards useful solutions for FRs focusing on something occasionally ignored during research projects: the point of view of the final user. The functionalities have been designed after multiple iterations between researchers and FRs, involving testing and evaluation through realistic situations in training scenarios. Thanks to this feedback, the overall satisfaction according to the evaluations of 18 FRs is 3.84 out of 5 for the Robust Vision Module and 3.99 out of 5 for the complete AR interface. These functionalities and the different display modes available for the FRs to adapt to each situation are detailed in this paper.

1. Introduction

Augmented reality (AR) is gradually being integrated into more and more critical human activities, as it can enrich reality with data that instantly provide valuable information in critical situations. Its usability and application can be beneficial for rescue teams, who are the first to arrive when an emergency occurs [1,2], especially in situations where low visibility is expected. First responders (FRs) often face very dangerous situations without prior information about what they will encounter, forcing them to make last-second decisions. The most experienced ones have likely been in situations where they lacked data during a mission such as a constant flow of information and directions from the control center or the knowledge of the location of their team members, the status of their tools, or their health status. These information inputs can save not only the lives of people in danger, but also the lives of the FRs themselves [3,4,5,6]. This is especially useful in conditions of restricted visibility, such as darkness or heavy smoke, where the amount of information that the end user can gather from the environment is limited.

Equipping FRs with AR displays endows them with extended visual capabilities, integrating information about themselves, their colleagues, and their surroundings in addition to their usual field of vision [1,2], which increases their overall Situational Awareness (SA) [7,8]. Existing proposals cover ARs that prioritize relevant information, such as the location of FRs and their health status [1]. However, since they need to navigate through dangerous environments, natural human senses such as sight can be highly limited under adverse conditions, becoming a higher-priority obstacle for operation. To overcome this challenge, FRs already use thermal cameras, which extend beyond the capabilities of human eyes and provide robustness in these situations [9,10]. Commercial thermal cameras are not usually integrated with the FRs’ setup, requiring them to be carried in hand and viewed directly on the device’s screen. The ergonomic implementation of these tools is necessary [11], as, otherwise, their integration can reduce the effectiveness of the FR against potential threats. Some research works incorporate the video stream of an integrated thermal camera in AR devices such as Hololens [12] together with localization information. In [13], an extended solution is proposed that also includes object detection (OD) algorithms for victim detection over thermal imaging. In that work, which was developed in 2020, FasterRCNN was the preferred detection architecture, but, today, a faster and lighter model such as YOLOv8, fine-tuned with thermal imaging detection datasets, can significantly improve detection capabilities [14].

Another important characteristic in ergonomic configuration is the adaptation of information in the AR interface [8]. In fact, information processing strongly depends on the cognitive load (CL) of FR within the rescue scenario [15]. CL is the level of mental energy required to process a certain amount of information [16,17]. The underlying principle behind this definition is that, the greater the quantity of information to be handled, the greater the amount of mental energy that is needed and, therefore, the greater CL for the FR. The objective is to avoid overloading the user, as this can have an impact on how the rescue mission is managed [18,19,20]. To do so, it is key to monitor the vital signs of the FRs in real-time, which are used to calculate the CL that the user is facing during a real scenario. The relationship between vital signs and CL has been thoroughly investigated in the literature, showing an important correlation [21,22,23,24].

In addition, the creation of user-adapted augmented reality tools requires the participation of end users in the design process. Recent articles highlight the importance of user-centered design in all stages of the development of augmented reality applications [25,26], including the validation procedure. The development of AR interfaces for rescue scenarios, such as the prototype presented in [13], can benefit from incorporating the opinions of professionals in the design process.

In this context, this paper describes the design and implementation of a configurable modular interface for transparent displays integrated into standard FR equipment. This interface has been rigorously tested and refined after each development period to adapt to the real needs of the FRs during different kinds of mission such as fires or mountain rescues. This was achieved by performing the test during some realistic simulations of rescue situations at specific rescue training centers throughout Europe. This interface supports FRs in adverse-visibility conditions and aligns with an architecture that provides rescuers with new tools and a functional data network in extreme situations. One of the tools provided, for example, is based on the machine learning module denoted as Robust Vision, which detects relevant objects in a rescue scenario, particularly victims, using the feed of a thermal camera. This feed can be displayed directly alongside the detected objects, helping FRs to avoid missing anything during the mission and preventing them from having their hands occupied by their usual thermal camera. Additionally, the exposition of information in the interface is organized according to the CL of FRs in real operations. The functionalities facilitating this interface modularity and distinct display modes are detailed in this paper. This flexibility facilitates the optimal data-flow in distinct rescue-field situations.

This paper aims to use the latest scientific advances and a direct line with FRs to provide the tools that they need in rescue missions under limited visibility circumstances. To analyze the present AR interface adapted for adverse-visibility scenarios Section 1, summarizes the interest in the technology presented. Next, Section 2 first details the architecture, highlighting the integration of tools such as the thermal camera and display, and then outlines the method used to integrate the feedback from the FRs into the development. Section 3 describes the resulting interface, while Section 4 expounds on the critical part of periodically testing and validating the interface developed within the European RESCUER project [27]. This part describes multiple iterations of the interface design during some realistic simulations of rescue situations. Several FRs tested the interface and provided feedback through questionnaires, which were used to incrementally adapt the Heads-Up Display design.

2. Materials and Method

This section describes the general architecture designed for the interface, from detailing the hardware, software, and communication components to describing how the displayed information screens were designed and implemented in the interface. Before exploring how to present the existing data in the most efficient way to the FR, the objective was to create an efficient environment for information transmission between everyone involved during a rescue scenario, regardless of its accessibility. In this section, the distinct roles and profiles (Section 2.1), along with the contextualization of technology in rescue events, are indicated. These include a graphical illustration of the system architecture considered and the necessary technical configuration for the main contribution of this study: the first responder augmented reality interface for adverse-visibility rescue missions. Next, the main components of this architecture are detailed: AR display technologies, the thermal camera-based Robust Vision Module (RVM), and the sensor kit and data-flow communication system, which are described in Section 2.2, Section 2.3 and Section 2.4.

2.1. Roles and Contextualization of the Architecture

The deployment is focused on two interconnected areas: the hardware integrated into the FR equipment and the software implemented on that hardware. Both faced several limitations due to the restrictions on FR equipment. Adding extra equipment to the FRs needed to be performed carefully to avoid interference with their regular tasks. Thus, the hardware had to be as light as possible, while still capable of running the created interface and its processing fluently. Consequently, making the software lightweight was a priority for the architecture. The scheme shown in Figure 1 summarizes the architecture considered to address all these frequent constraints in rescue scenarios, integrating the main contribution of this study: the AR FR interface. The present architecture considers two distinct endpoints according to distinguished user profiles:

• Command Center (C2): The personnel at the control station in charge of the C2 must receive all information from the mission members to monitor progress and make appropriate decisions for mission success. They also need the capability of sending messages to the FRs, as it is essential to update mission objectives as needed. Although communication with the command center is crucial in rescue scenarios, this study focuses on the AR interface deployed on FRs and their limited equipment. C2 information is displayed on regular screens and managed with fixed computer equipment, which allows greater processing power at the expense of added weight;
• First responder (FR): The professionals addressing the requirements of the emergency field who need access to accurate information for each situation. They face several limitations regarding the equipment, as extra weight can hinder movement or endanger the FR during operations. Therefore, the presented architecture considers hardware that is as integrated as possible with the equipment that the FR already carries, while being capable of running the necessary software for new functionalities. In addition, the same way that PPE is evolving towards light heat-efficient materials [28], the lighter the hardware that the FR has to carry, the easier it would be for the FRs to move during the mission.

2.2. AR Display Technology

An essential part associated with the AR interface is where it will be displayed, as depicted in the architecture visualization in Figure 1. This scheme considers a single component connected to the processing unit where the Smart Modular Interface is running and, therefore, the exclusive point of access for the FR to the information. The interface has been designed to deploy transparent displays that overlay elements onto the real world. It does not require interaction with the depicted elements, as it is a passive interface. As part of the RESCUER project [27], the interface has been configured to be compatible and tested on the Smart Helmet system, represented in Figure 2 [7]. This technology contains a set of hardware modules that includes a non-interactive, one-eye AR interface for FRs. Although it shows promising potential, it is still under development. However, there are commercial alternatives with similar hardware where the interface has also been tested, with the Hololens being one of the most well-known examples. The Hololens is a complex device that transcends a simple visor since it has cameras that facilitate gesture recognition and collisions between the drawn objects and the user’s hands. This functionality not only allows data visualization, but also provides greater interactivity between the FR and what they are seeing. Nevertheless, its dimensions, fragility, and operating temperature range do not make it the most suitable option for the situations that a firefighter may encounter. One of several commercial options in which the interface has been tested to be oriented towards FR usage is the Vuzix M4000 option, a monocular AR viewer manufactured by Vuzix in Rochester (NY, USA) that has an Android operating system. It is less bulky and more adaptable to the head of the FR, making it a better alternative for rescue scenarios since it is more compatible with personal equipment.

2.3. Thermal Camera Device and Streaming Configuration

In addition to the display, since we are enhancing the sight and SA of the FRs with an RVM based on thermal images—one of the main features of the AR interface—the thermal camera is also a key hardware element. This device requires a Long-Wavelength Infra-Red (LWIR) sensor, sensitive to electromagnetic waves ranging between 8–15 $\mathsf{\mu }$m, that allows the detection of heat signatures, such as the heat that a human body emits, without external illumination. The physical constraints are mainly the dimensions and weight of the device: it should be as compact and light as possible to be integrated with the FR’s equipment. Additionally, this camera must be able to process the acquired image to deliver a human-comprehensible video that can be streamed to the processing unit of the architecture in real-time. The Smart Helmet system [7] has also been used to perform most of the tests in this case, as it includes a thermal camera attached to an operational helmet that complies with all listed requirements. In this case, the camera provides a processed gray-scale image with 640 × 480 resolution, 60 Hz frame rate, and a 38-degree diagonal FoV. The gray-scale range is automatically adjusted within the temperature variation captured and the image is enhanced to facilitate the human interpretation of the image. Nevertheless, commercial solutions could also be considered in this architecture, such as the FLIR C3 camera. As stated before, other thermal cameras could be compatible if they overcome the problem of their integration with the FRs’ equipment and the robustness against adverse conditions encountered in rescue operations.

There is also additional software on top of the RVM that enables the implementation and the compatibility between components. v4l2loopback v0.12.6 [29] is a kernel module that is compatible with Video4Linux version 2 (V4L2), which is the set of drivers and API responsible for creating V4L2 device nodes (/dev/videoX) and tracking them in Linux systems. As the thermal camera is directly connected to the FR processing unit as a unique video device, v4l2loopback v0.12.6 has been used here for creating a virtual V4L2 device that allows the video to act as an input source for different applications simultaneously. Additionally, FFmpeg 6.0 [30] is a widely-extended cross-platform multimedia framework for streaming, transcoding, and filtering, among other functionalities. It is also compatible with V4L2 and is used to stream the original video device to the virtual device. Both tools are designed to be efficient and lightweight processes, minimizing the impact on video stream performance. This implementation ensures that both the RVM and the AR can access the virtual device simultaneously without blocking its usage, as shown in the Video Streaming Virtualization in Figure 1.

2.4. Sensor Kit and Data-Flow Communication System

To enhance the Situation Awareness of the FRs in rescue missions, different sensors can provide biological (heart rate, breath rate) and environmental (temperature, humidity) information. Additional tools (gas or life detectors) also furnish essential information about the circumstances of the rescue mission. In this context, since different types of mission require various kinds of equipment, we refer to the combination of all the sensors carried as the sensor kit.

The information provided by this sensor kit, combined with the camera feed and the detections made through the RVM, needs to be distributed throughout the entire architecture so that it can be displayed on the interface of each FR dispersed in a mission. This flow of information has to be designed not only to require minimal processing in any device but also to be modular, enabling the inclusion of new information sources or removing existing ones, depending on the scenario faced by the FRs. In order to achieve this, two elements were combined: a network generated ad hoc in the field by any number of gateways, and a lightweight and efficient messaging protocol offering reliable, scalable, and resource-efficient communication called Message Queuing Telemetry Transport (MQTT) [31]. On the one hand, the local network is generated by devices carried by the FR, characterized by its ability to dynamically connect when they are within range, allowing the transmission of data between the members of the rescue team and the C2. On the other hand, MQTT uses a Push/Subscribe topology that uses two types of systems: brokers and clients. Brokers are in charge of distributing the data between clients. Some clients publish information on the broker associated with certain topics (sensors, for example) and, by subscribing to those topics, other clients, like the interface receive the messages as soon as a modification is complete.

In this architecture, the processing unit carried by the FR has its own MQTT broker. This ensures that the communication between the MQTT clients of the sensors carried by the FR and the client of the interface is never interrupted, ensuring that the FR always has, at least, its own information on the display. However, the exchange of information between FRs and between a FR and the C2, when they are within the range of the network generated by the gateways, is carried out by a separate piece of software: the Data Share Orchestrator (DSO) [32]. The DSO is in charge of two tasks: propagating messages between all the nodes of the network within reach of the data-generating node and evaluating which data should be displayed, depending on the CL of the FR and the importance of the message received. This second task is fundamental to adjust the information visualized to the FR to maintain appropriate CL values. This CL value is estimated by considering the information gathered by different biosignal sensors. With this CL value, the DSO can filter information with lower-priority information to avoid overloading the user, as high CL values can have an impact on the performance of the rescue mission.

2.5. Validation in Rescue Training Scenarios

The user-centered design of the presented AR interface includes the participation of the FRs throughout the process. The H2020 European RESCUER project [27] provided realistic rescue training scenarios to test and refine this design based on feedback and suggestions from rescue professionals. The testing included two rounds of three pilots each, conducted at the end of each development period. In these pilots, FRs evaluated the developed technology in three use-case scenarios: earthquake scenarios (Weeze, Germany; Thessaloniki, Greece), mountain rescue scenarios (Navacerrada, Spain), and tunnel scenarios (Modane, France). Each place had a specific simulated environment to train European FRs for these particular scenarios.

These tests included an initial phase during which FRs were trained to interpret the display. This was followed by a simulated emergency field phase, where they performed rescue tasks such as dragging victims, detecting hazardous materials, and operating in smoke and darkness. Finally, the physical testing of the prototype was complemented by filling anonymized opinion surveys and open suggestion collections. Questions related to the presented AR interface are detailed in Section 4.

3. Results

The main focus of this section is to detail the basis and objectives of the interface design and implementation, explaining the selected characteristics to create a light and flexible tool, based on the FRs’ expressed needs, and also taking into consideration how it could be extended in future implementations.

3.1. Basis of the Design of a Modular Interface and Development Frameworks

The flexible architecture facilitates any replacement of the data source, allowing the integration of new tools. This adaptation requires two steps: creating new MQTT messages that follow the existing message structures and data frequency limitations, adding the new topic to the DSO so that it is distributed throughout the architecture; and adding the new component to the interface, including the selection of appropriate icons and text styling.

The modularity of the interface is defined by its ability to switch the screens, as well as the adaptability of the icon system and the boxes extended from previous work published in [8]. The icon system and corresponding definitions are subsequently detailed in Section 3.2. This subsection describes the different screens to which an FR has access during a mission, denoted as Views. Switching between Views can facilitate the change of information according to objective variations. The focus of the FR can be on sensor information, victim location in the darkness, or other target positioning. The transition between Views is made in carousel format using a single-button remote control, which could be replaced by implementing a voice control module for the FR. Sometimes, a view needs small adaptations to the user’s preferences while maintaining the overall concept of the display. These variations will be referred to as alternative modes, or simply modes, from now on.

The final programming language selected to develop environments and interfaces has been Angular v16.1. As a Javascript framework, it provides the ability to run in web browsers like Google Chrome or Firefox, which are present in virtually any operating system, including those running on laptops, Arduinos, or native software devices such as Hololens. It is not the lightest framework; there are ways to create lighter web pages using Python with libraries like Dash, for example. However, Angular offers two advantages in return: a variety of aesthetic libraries that allow for polishing the final style of the interface, and a library that enables running an MQTT client directly on the front-end, which is an important aspect of the architecture.

Within each of the Views, a key aspect is to modularly position elements on the screen based on the amount of information. This is where another advantage of ‘advanced’ JavaScript development environments, such as Angular, comes into play: their ability to adaptively redistribute components within a web page. A concept known as the Flexible Box Module, commonly called Flexbox, is designed as a one-dimensional layout model to help distribute space between items in an interface and improve alignment capabilities. It is a widely used technique, not only in Angular but also in other frameworks and development languages.

In this design, the tool used to add flexibility to the interface is PrimeFlex v3.3.1, which is part of the PrimeNG library v16.0. This library divides the screen into equally sized sections that can be grouped or further divided, allowing for the dynamic addition or removal of elements on the display without affecting the overall aesthetics.

The final design features three Views, with two of them offering two alternative modes each. The details of these modes are described in the following sections.

3.2. AR Sensor Data View

The AR Sensor Data View of the interface is designed to fully leverage the AR capabilities of the display. This View presents information from the multiple tools and sensors that the FR carries without obstructing the user’s view of reality. The tool and sensor representative icons are detailed in

Table 1. Icon description on the default view of the AR interface, as detailed in the recent article [8].

Tool Icon	Description of Corresponding Sensors
	Biosignals: Description of the biological signals of the FR. The color of the icons will change from green to red depending on if the values received are good or bad. It includes the information on the heart rate and the breath rate of the user.
	Black box: The black box is a device that has some environmental sensors inside: temperature, humidity, and number of people around it.
	Ad hoc network: Information of the wireless network generated by the gateways that each FR carries during an operation. It provides information about the battery level of the gateway and the status of communications between the C2 and other FRs.
	Augmented olfaction: It provides the concentration values of up to 5 types of gases, turning from green to yellow to red, depending on if the gas levels are dangerous or not.
	Signs of life: This tool provides information if it detects life (through walls, for example); it provides an estimated value of the distance to the found individual. The icon turns from yellow to green when life is detected by the device.
	Radar: This device shows the number of objects that are approaching the FR. The interface blinks when an object is approaching. The icon blinks red when an object is approaching.
	Wireless finder: This device measures the distance to devices that produce wireless signals such as Bluetooth or Wi-Fi, which usually come from mobile phones from victims under the rubble. The interface also guides the FR in the initial calibration phase. The icon turns from green to yellow to red as it gets closer to the wireless device.
	Robust Vision: Displays the type and number of objects detected by the object-detection module. It can be switched to a full-camera view that will show the IR video feed with the red squares instead of the transparent view.

. However, another type of information shown in this view is especially critical for the FRs in rescue operations: the location of teammates and points of interest sent by the C2. To visualize geographical information, two alternative modes have been developed for this view: one displays the information of the targets and their distances from the user on the circle in the direction that the FR needs to look in order to reach them, as illustrated in Figure 3. This method uses a magnetometer and the actual orientation of the FR to position the targets relative to the direction that the FR is facing. Some of the FR found this compass confusing; this is why, in the second case, a mini-map of the region where the rescue is taking place is visualized. On top of this map, the user’s position and the location of all the points of interest are directly drawn.

To manage the quantity and prioritization of the information displayed in this mode, cognitive load is measured to change the information shown on the screen, helping the FR to discern which information is most relevant.

3.3. Robust Vision Module Integrated View

One of the main objectives of the AR interface is to increase the SA of the FRs, enhancing their senses, in this case, human vision, without interfering with their operation. Following this basis, the set-up proposed in [7] includes an LWIR-sensor camera attached to the lateral of the helmet, named as thermal camera in the diagram in Figure 1. These cameras are particularly useful for seeing the heat signature of the objects around them, even in the presence of occlusions such as heavy smoke and darkness. In this implementation, since the camera is positioned at the helmet, the Field of View (FoV) of the camera matches that of the FR. Therefore, in the RVM view, the transparent background of the AR interface is replaced with the actual feed of the camera in real time, overlapping with their sight, providing them with SA of their surroundings, and enabling them to continue their operation despite low-visibility conditions.

In addition to the feedback of the thermal video streaming view, there is an auxiliary module, called RVM in Figure 1, that analyzes the streaming to provide a summarized alert of what the thermal camera is seeing, even when the FR has not selected this view. This module is an AI-powered algorithm based on the YOLOv8 architecture [33] that has been trained on an annotated thermal image dataset [34] to detect the possible objects of interest—in this case, people and cars. The training process is explained in detail in [7]. The input of the module is the video feed of the thermal camera. Every k number of frames, where k is selected according to the system or user needs, one frame is analyzed in order to detect any possible person or car on the image. The output of this model is a list of items detected with the following information:

• The label of each item indicates the type of object that has been detected; in this case, only two possibilities have been considered: person and car;
• The confidence of each detection ranges from 0 to 1, i.e., the confidence value could be 0.1 if the algorithm has low certainty that the item detected is correct, and it could be 0.9 if the probability of that detection being reliable is high;
• The coordinates of a bounding box that contains the detection. These coordinates are represented as the relative positions of the image analyzed, given by four numbers: the normalized x and y coordinates of the center point of the object bounding box, setting the origin at the top left corner, and the normalized length and width of the bounding box.

This algorithm is running in the processing unit that the FR is carrying. As with any other module of the diagram detailed in Figure 1, the results of the model are correctly formatted and shared through the implemented MQTT protocol, so the detections performed over the thermal camera of any FR can reach their interface, their colleagues, and the C2. Following the instructions of the FRS, two different modes of display for this view are created: one in which the thermal feed and the RVM information are displayed, and one in which the same information is combined with the AR Sensor Data View information, so that all the icons and data are shown on top of the thermal image. If the AR Sensor Data View or the combined mode is selected, the icon of the eye presented in Table 1 will provide the summary of the detections, that is, the label and the number of items found with that label (for example, “person:3”). Furthermore, in this view, the bounding boxes are above the items detected, and each detection includes its label and confidence value written in the top-right corner of the bounding box, as shown in Figure 4.

3.4. Live Situation Map View

The Live Situation Map View is designed to help the FR to visualize the location of the main events of the mission and locate the next objective, as illustrated in Figure 5. In this view, all elements apart from the map and the locations of companions and the points of interest are removed, allowing the map to occupy the entire screen. It is important to note that, since there is no guarantee of having internet access during the rescue scenario, the map visible in this view is preloaded before the mission. However, in future work, if a new map and coordinates are necessary, the command center could send the data using MQTT to update the FR’s view and trigger a screen reload.

4. Discussion

After continuous iteration through development, piloting, and evaluation, there is still work needed to address all requests from the FRs. However, the final version of the AR interface and RVM described in this work resulted from adjustments based on use-case scenarios and FR preferences. This ultimate version was tested in the final round of pilots in 2024, conducted in Navacerrada, Thessaloniki, and Modane in their facilities for FR training on mountain rescue, earthquakes, and tunnels, respectively. In these final tests, seven FRs participated in Navacerrada, three in Thessaloniki, and eight in Modane, evaluating the system and its components according to the availability of volunteers and the technical and time constraints faced during operations. The pilots aimed to refine the AR interface and RVM together with the rest of the SA-enhancing tools and sensors detailed in Table 1 to ensure that they were effective and user-friendly for different emergency scenarios.

It is important to note that extraordinary challenges were faced during the Thessaloniki pilot that compromised its overall performance. Location, planning, and resources were reallocated a few weeks prior to the pilot, resulting in less time for the overall organization. Additionally, unexpected technical complications, such as device overheating due to extreme weather conditions, extended the time required for technical setup. Consequently, the number of FRs testing the system was drastically reduced. On the other hand, the Modane pilot being the final one and benefiting from lessons learned in previous rounds, achieved top evaluation results. This success was partly due to improved organization, better time allocation, and effective technical integration, which accounted for expected limitations.

The results of the information provided by the AR interface for each tool are presented in

Table 2. Overall evaluation of the satisfaction with the information provided by the interface per tool in each pilot.

Tool	Navacerrada	Thessaloniki	Modane
Biosignals	4.14	4.00	4.38
Robust Vision	3.43	3.33	4.38
Black Box	not tested	3.33	4.25
Localisation	3.57	2.33	4.25
Wireless Finder	3.43	3.00	4.00
Signs of Life	3.00	3.00	3.75
Augmented Olfaction	not tested	not tested	3.67
Ad-hoc Network	3.86	2.67	3.63
Radar	not tested	3.00	3.13
Overall satisfaction 1	3.57	3.00	3.89

¹ Average of the individual tools satisfaction within that pilot.

. This table collects the average satisfaction level with the visualization of the information through the proposed interface, divided by tested tool and scenario. The satisfaction rating ranges from 1 to 5, with the following scale: Not satisfied (1), Not very satisfied (2), Neutral (3), Satisfied (4), and Very satisfied (5). The average results for each pilot range between Neutral and Satisfied. One of the minor changes made from the Thessaloniki pilot to the Modane pilot was the addition of the Live Situation Map View detailed in Section 3.4 to the AR Sensors view. This adjustment was implemented in response to feedback from some FRs, who reported difficulties in interpreting the circumference as a reference. Consequently, the evaluation of location information visualization has improved significantly across the pilots. Overall, the AR interface’s evaluation regarding the information provided by each tool satisfies FRs for most tools.

The questionnaires provided to the FRs for evaluating the system include questions about their perceptions of the interface and the modules, such as RVM, and how these might impact their operations.

Table 3. Summary of the evaluation results for the last round of pilots.

Question	Navacerrada	Thessaloniki	Modane	Overall 1
How would you rate the capacity of the user interface to improve your Situational Awareness?	4.29	3.67	4.75	4.40
How would you rate the user-friendliness/ease to control and operate this subsystem	3.86	3.33	4.25	3.95
Were you satisfied with the way the information is provided and displayed for the functionalities?	3.57	3.00	3.89	3.62
Overall satisfaction with the interface 2	3.91	3.33	4.30	3.99
How would you rate the relevance of the information provided by RVM?	4.14	3.67	4.33	4.15
How would you rate the visualisation of RV Module output?	3.93	3.33	4.33	4.01
How would you rate the capacity of RV Module to enhance your SA compare to existing situation?	3.93	3.67	4.33	4.11
The functionality could improve my efficiency in operations	4.07	3.67	4.67	4.27
The functionality could improve my safety in operations	4.14	4.00	4.44	4.25
The information provided disturbed me during the tests	2.93	4.00	2.44	2.89
I would use this functionality during a real mission	3.93	3.67	4.44	4.11
Overall satisfaction with RV module 2	3.74	3.29	4.13	3.84

¹ Weighted average of the satisfaction per question. ² Average of the satisfaction within each pilot.

details the evaluation of the most-representative questions about the interface and the RRVM for each pilot. Ratings range from 1 to 5, with the following scale for questions: Very bad (1), Bad (2), Neutral (3), Good (4), and Very Good (5). For the agreement with statements, the ratings also range from 1 to 5: Strongly disagree (1), Disagree (2), Neutral (3), Agree (4), Strongly agree (5). As mentioned previously, the average tendency found between pilots is lower rates for Greece and top evaluations for Modane. Considering the evaluation of the AR interface, we can conclude that the objective of improving the SA of FRs through this AR interface was successfully achieved, as it has been rated as Good and Very Good for most of the FRs, obtaining the highest score with a weighted average of 4.40. Similar ratings are obtained for its usability: as it is a passive interface, only one button is enough to control the complete interface and change the views; additionally, the icons have been designed to be intuitive and recognizable by FRs easily, as has the information provided to them.

Regarding RVM and its integration with the interface evaluation, we found that the incorporation of the thermal video feed into the interface was highly appreciated, with ratings for its relevance and visualization averaging around 4. This module is also considered to enhance SA and improve efficiency and safety during operations. However, one rating consistently remained slightly low in every round: the disturbance caused by the module. This was rated close to neutral in two cases, and strongly agreed upon in Greece, suggesting that the feed (and probably the detections) could overwhelm the FRs using the interface. To address this issue, the cognitive load module and DSO decided to filter out excessive detections from the RVM for Modane, thereby reducing the amount of information displayed on top of the thermal video feed. This adjustment led to significant improvements in Modane ratings, although there is still room for refining these algorithms and their integration into the AR system. Additionally, the thermal camera and Robust Vision capabilities are more beneficial for indoor rescue operations, such as in the tunnel of Modane, compared to open mountain landscapes.

Overall, the evaluation of the module demonstrates that it successfully meets the objective of enhancing SA and highlights the possible improvements of the tool for future operational use and integration into FRs’ operations.

5. Conclusions

One of the main conclusions of this work is that the synergy between the Robust Vision Module and augmented reality interface has significant potential for rescue operations. After testing and validating the proposed AR interface, which integrates AI-driven thermal image analysis to assist FRs in detecting victims under adverse-visibility conditions and adapts its design to different scenarios, the feedback from the FRs highlighted its promise for their work. However, once the system maturity is suitable for operational environments, evaluations should be performed for a larger number of users and objective indicators, such as the duration of the operation with and without the system.

The goal of developing an intuitive and non-intrusive interface, focusing exclusively on elements with which FRs are familiar, was successfully achieved. In life-threatening scenarios, technology must adapt to the circumstances, enabling FRs to concentrate on their objectives without distractions. The modularity of the interface facilitates the selection and presentation of relevant information according to the scenario and needs of the FRs, ensuring that the visualization remains unobtrusive. Additionally, this modularity allows for the future integration of new sensors or tools, as long as they adhere to the pre-defined messaging protocol. The synchronization and integration with different decision-making frameworks also enables it to adapt to alternative scenarios, such as earthquakes, tunnel rescues, or mountain rescues, as well as other low-visibility, non-rescue-related jobs.

Currently, the Technology Readiness Level (TRL) of the interface is 6, as it has been demonstrated in relevant environments. To achieve a market-ready tool, hardware and software will need to undergo strict verification in operational environments to ensure the safety of rescue operators. The ultimate goal for the future is to create a ready-to-be-used system that includes an extended version of this interface, which would facilitate rescue tasks while enhancing the safety and efficiency of FRs in diverse rescue scenarios.