Augmented Reality for Emergency Situations in Buildings with the Support of Indoor Localization ^†

Abstract

Augmented reality is showing a continuous evolution due to the increasing number of smart glasses that are being used for different applications (e.g. training, marketing, industry, risk avoidance, etc.). In this paper, we present an implementation that uses augmented reality (AR) for emergency situations in smart buildings by means of indoor localization through the use of sub-GHz beacons. This also includes the mapping of emergency elements in the three-dimensional building, together with some example cases.

1. Introduction

In the event of a fire in a building, there is the possibility that some people will be confined to a room and their only option for survival is to be rescued by emergency services. The first objective of emergency services is to save people’s lives without risking their own. In some cases, they know where the trapped people are but they do not know the best way to get to their location. This is often difficult because the typical conditions in the event of a fire are high temperature, poor visibility caused by smoke, and blocked routes caused by the collapsing of structural elements. Rescue teams must fight against these obstacles and try to save as many lives as possible while minimizing the risk to their own lives. Time is a crucial element for a successful campaign as situations usually worsen when working at high speed. Every second saved in the planning and execution of the search for people who are alive increases the probability of their survival.

We propose the creation of a system that combines augmented reality, indoor localization, and radio communications so that emergency services can plan and execute their emergency response operations in a much faster way, increasing the probabilities of success.

The paper is organized as follows: In Section 2, we present the state of the art. Section 3 presents the proposed system, followed by Section 4 that presents the implementation of the proposed system. In Section 5 we present the results, and finally we make our conclusions in Section 6.

2. State of the Art

Occasionally, emergency services face the challenge of rescuing people trapped inside complex buildings. In such cases, the lack of a planning and navigation system can make the task very difficult. An emergency support system that assists emergency services in locating victims and guiding emergency personnel towards them, taking into account all the possible difficulties such as smoke, high temperature, flooding, the presence of gas or other inflammable material, and structural collapse would be of great help in supporting the work of emergency professionals. Such requirements were already identified in 2013 by Gelenbe and Wu [1] in their study of future emergency management systems.

One of the important parts of such a system is the localization system, which would allow service personnel to be accurately positioned and guided towards the victims. Ferreira et al. (2017) [2] analyzed all the available localization systems for emergency responders.

In these scenarios, satellite-based positioning (such as GPS) is not always possible due to the lack of signal coverage inside buildings. Mobile networks typically have better coverage inside buildings, but are still not good enough for accurate positioning. In 2007, Varshavsky et al. [3] tried using GSM to locate the floor on which the person in need of rescue was located, but we should assume that, depending on the infrastructure of the building, GSM coverage will be bad.

In the case of emergencies such as fire, flood, collapsing buildings, earthquake, etc., a basic assumption is that power supply is not guaranteed, and some parts of the available localization systems could have been destroyed, so any emergency management systems should take this into consideration.

In 2013, Giuliano et al. [4] proposed the use of RFID tags to support localization. RFID tags are cheap and can be deployed easily along a building. However, they all need to be geo-localized and do not offer very good accuracy.

In 2014 and 2015, Li et al. [5,6] proposed deployment of the beacons during operations in order to construct a building model (BIM) to help in localization. The beacons would be deployed during rescue services, although this would slow the task of rescue. They would use Wi-Fi access points and send RSSI to triangulate the signal so that the coordinator would know the position of all agents inside the building.

As an alternative, in 2014, Simon et al. [7] proposed the pre-deployment of some radio landmarks (beacons) that would communicate with the mobile equipment of a firefighter with an inertial measurement unit (IMU) in order to improve localization. The Radio landmarks would use 868 MHz. The Sub-GHz band is a good option because it has a good coverage that does not require much infrastructure.

Because we cannot always rely on the standard networking building infrastructure, another option is to utilize people’s smart phones, such as in [8]. However, this approach has the drawback of requiring a large number of people, and we cannot always expect to find victims who are ready to assist the navigating system in the most dangerous areas.

In the last few years, many studies have appeared regarding indoor localization using different approaches to obtain high accuracy, utilizing different technologies like Wi-Fi and Bluetooth. Depending on the method used, the indoor localization system could be classified into different categories: time, angle, and received signal strength (RSS) [9,10].

The release of Bluetooth Low Energy (BLE) added a new possibility for indoor localization because of its low power consumption and the possibility of cheap implementation. There are two options when deploying an indoor localization system; using beacons on the infrastructure or using beacons on the person.

The cheapest option is to deploy beacons around the building, and the person who is going to navigate the building then needs a device that can receive messages from each beacon in order to calculate their position.

On the other hand, using beacons on each person means that the system requires deployment of antennas in the building that can receive the data from each beacon and send it to a server to combine the information and return it to the person.

The most common approach to indoor localization using BLE is the received signal strength indicator (RSSI), but there are other approaches, such as angle of arrival (AoA), which uses an array of antennas to measure the difference of arrival of the message to each antenna to estimate the location, but requires at least two antenna array devices to locate a person, and time of flight (ToF), which uses signal propagation to calculate the distance between the transmitter and the receiver [10,11].

Multipath is a problem on indoor localization due to the reflection of the signal within the structure of the building. This issue is present in several of the previously commented methods. With RSSI, the signal degradation impacts the signal power received and with ToF the distance traveled by the signal is much larger than the primary path and thus not optimal. The issue with AoA is the reflected signal being received at a different angle than the original [12].

The structure within the building also contributes to the degradation of the signal in terms of power reducing the range of the signal and contributing uncertainly on the localization of the device [13].

Besides localization, other important factors of the rescue system are the tools given to the rescue personnel. Many previously mentioned works use smart phone-based interfaces to navigate through the affected areas. This is not realistic in a harsh environment, as personnel need both hands to work on the rescue tasks. One of the options could be based in spatial augmented reality systems [14], but the technology is still not widely available in the market. An alternative is to use a video see-through augmented reality system embedded in the equipment of the workers. In 2011, Ahn et al. [15] already presented some ideas on using augmented reality but still using smart phones as the interfacing device. Augmented reality systems would require 3D models of the buildings. This would be a burden for the proposal if all buildings had to be modeled. However, we only foresee the modeling of complex public buildings such as hospitals, train stations, airports, etc.

To overcome the limitations of the systems found in the literature review, we propose an augmented reality system based on optical see-through augmented displays, preexisting 3D maps of important buildings, and a positioning system based on landmark nodes.

3. Proposed System

There are some aspects that we must take into consideration when we design the system. Firstly, when a fire is declared the first thing to fail would be the power supply, but in case it is still working, the emergency teams will cut off the power. With that in mind, our landmark nodes need to be able to work without power supply. The solution would be the use of batteries. In addition, we need to send messages to the outside so a good workout would be the use of a sub-GHz band.

Secondly, fire could destroy landmark nodes placed in the infrastructure building. To solve this inconvenience, the rescue teams would have to provide its own landmark to be able to instantly deploy new nodes to the system.

Next thing to consider is the coverage of landmark nodes, given the building does not have its own, in which case the rescue teams would have to deploy on known positions to cover all the building.

Finally, to mitigate the lack of visibility, we propose the use of augmented reality to help rescue members to move around the indoor spaces.

A technology that helps with localization is Bluetooth 5.0 using devices as beacons. Every person can carry a small device with a unique identifier and antennas deployed around an area would capture the transmitted package and send it to a server to calculate the position.

To facilitate the navigation inside the building and rescue the people trapped, they would use the building blueprints. By means of these maps, the rescue teams can look for the quickest way to reach trapped people and to be able to rescue them. Another solution would be to offer the rescue team a tool to be able to navigate inside the building together with a calculated route according to the inconveniences that can be found. This could be done using a wayfinding system.

This study is focused in presenting a solution to rescue teams which will help them to navigate inside a building using Bluetooth 5 and a 3D reconstruction of the building projected on augmented reality glasses.

With the previous consideration proposed at the beginning of this section, our system needs to offer a localization solution that can be installed on the building or deployed by rescue members on site. We need to assume that there will not be power supply, thus the landmark nodes need batteries to operate.

All the data sent by the landmark nodes must be collected by a gateway, which can be on the infrastructure of the building, but a redundant system must stay with the rescue team. In case of malfunction or destruction of the gateway in site, the data will be collected directly by the rescue team’s gateway.

Our augmented reality solution should show only the basic infrastructure of the building as we cannot assume where moveable elements will be placed at every moment. To be able to receive data we need to provide a wireless connection to our AR device. The rescue control truck could provide this connectivity.

With these assumptions, our architecture system will consist of three components: an indoor localization system, smart glasses device to provide augmented reality to each member of the rescue team and an operations management system. In Figure 1 is a representation of the design requirements.

To be able to track members of the rescue team inside a building they will have to carry a BLE beacon and the building must be prepared with antennas or a system of portable antennas needs to be deployed to receive the data.

3.1. Indoor Localization Component

The location system is composed of three elements. In first place, beacons equipped with a temperature and humidity sensor are carried by the rescue members transmitting a signal using BLE. In second place, the landmark nodes that will receive the mentioned BLE data will broadcast it using long-range protocol at 868 MHz. Lastly, a 868 MHz concentrator (gateway) to receive all the data sent by the landmark nodes and send it to the position computation system or could be sent to the Internet for external monitoring operations.

Besides these elements, we also need to take into consideration the position computation service which fuses all data from the beacons, calculates the new position of an agent, injects the positions to the system and informs the firefighters about the path and new positions. The firefighters take advantage of this system establishing an Internet connection through a wireless network.

Some buildings may have its own concentrator, although it is not mandatory. Rescue services will always carry its own system assuming that, in case there is a concentrator in the building, there will be no power supply.

3.2. Augmented Reality System for Rescue Personal

To be able to provide augmented reality and indoor navigation to a firefighter, we need to add some see-through system displays to the helmet in order to offer a 3D recreation of the building and guidance. Furthermore, a radio beacon is needed to send ticks to the landmark nodes. This beacon will also provide temperature and humidity to the agent and to the coordinator of the operation. The system needs a connection to receive position coordinates. A proposed helmet can be seen in Figure 2.

The system has been proposed to function on a Smart Glasses due to the limitation on modifying a helmet of a firefighter. This element implements three important functions. First, BLE communication with the beacon to get the temperature and humidity. Second, three-dimensional recreation of the building and the positioning in the scene. Finally, audio and video streaming server to share what the agent sees.

To provide data of different buildings, a three-dimensional modeler is needed. This component should have a large database of buildings, especially public buildings such as hospitals, universities, etc. Localization is more necessary in this type of buildings since smaller constructions are much less complex. Our system will connect with that database and request all data to generate the three-dimensional recreation.

In case of radio signal loss, the smart helmet device would use inertial odometry to compute the position.

3.3. Operations Management System

A control center is needed to plan the operations carried out inside the building. This component will have a three-dimensional model viewer with the inserted localization data to control in real-time the position of all agents inside the building. This control center could be placed in a vehicle close to the building and have a gateway to receive localization messages. Therefore, it would receive the landmark nodes readings in case the building concentrator fails.

Otherwise, the system can also forward the information to a cloud server, thus other personnel can monitor the operations.

4. Implementation

4.1. 3D Model

The real-time three-dimensional reconstruction module starts from the foundation plans described on a given model and builds the complete three-dimensional model of the building. The input data used for the modeling can be presented as a map in pdf, jpg or another image format. The user geo-references and vectorizes this element using a GIS tool. Those vectors are the inputs for the process that in a quick way produces a multi-layered model embedded in a virtual world. The model is stored in a database with the other elements in the scene, as extinguishers, fire hose or emergency exits.

The virtual model is immersing, meaning that all elements in a scene are positioned in a global reference system using the World Geodetic System 84 projection (also known as WGS 1984 or EPSG:4326). On the other hand, the indoor positioning system uses a local reference system. In order to solve the problem between the two reference systems, a known point using GPS coordinates needs to be determined to convert between coordinate systems.

The module sends to the core server all the data extracted from the foundation plans. Each element is stored in the database using its coordinates. A communication BUS receives data from every element connected to the system and a server reads this information and decides what to do with it as in storing, processing or sending it directly to the clients. Figure 3 shows a representation of the reconstructed building.

4.2. Localization

The indoor system is composed by a beacon that will be carried by every agent during the emergency, a variable number of landmark nodes distributed inside the building and a gateway connected to the localization server to receive data from landmark nodes.

The beacon is a small PCB with a BLE module and a temperature and humidity sensor. Each one has a unique ID. The main functionality is to provide localization of the agent in real time. This device is also connected to the smart glasses to send data of temperature and humidity, which will be used to calculate a possible thermal shock of the agent.

Landmark nodes need to be in a known position in the building. Using the coordinates of each device that receives the beacon package, we can triangulate the signal to locate the agent inside the building. If the building has its own landmark nodes, they will be on the blueprints. Otherwise, the rescue team will have to deploy its own landmark nodes system on designated coordinates to equip the building with a temporarily localization system. Landmark nodes are able to get BLE packages and send them to the 868 MHz channel to communicate with the gateway.

The gateway is the main element of the localization system and, without it, it is not possible to get the data from the landmark nodes. The rescue team must have its own gateway on a truck, just to ensure the optimal coverage of the building or to prevent a possible malfunction of the gateway in the building infrastructure. The gateway receives data at 868 MHz and sends it back to the localization server.

The localization server could be on the cloud or in the emergency central building and its main function is to fuse data from the landmark nodes. RSSIs and IDs received are stored on a database and every 5 s a process evaluates the new RSSI for each ID and triangulates the new position in the coordinate system. This new position is stored on another table in the database and published on an MQTT queue to communicate with other parts of the system.

4.3. Control Management

This component is a web interface that communicates with the Core server. The main function is to provide information to the control center which could be located in a truck close to the building or in the emergency central building. All the data generated by the agents inside the building is visualized in real time on the web interface. Figure 4 shows the web interface. There are multiple layers that could be activated to show more data of each element in the building. From this interface the coordinator could communicate with the agents using an audio channel or by activating the video of each agent to watch the actual situation inside the building.

4.4. Augmented Reality

In the absence of a helmet of a firefighter, the implementation and tests have been carried out on the intelligent glasses from Epson, the model BT-300 (Figure 5).

An application developed for Android smart glasses was used. First, the application makes a request to the core server through an HTTP data of the walls and static elements of the building. A JSON with coordinates of wall vertex is received and recreates the virtual model of the building and places different objects in the scene like extinguishers or emergency exits.

The communication between the smart glasses and the server positioning system is done using Wi-Fi. This connection allows us to use the video and audio services of the system. Although a more reliable system working at sub-GHz bands would allow that in case of lack of Wi-Fi coverage, the positioning system continues working, even if other systems like video and audio stop operating.

The smart glasses connect with the beacon carried by the agent and get the unique ID and data from the temperature and humidity sensor. If the agent wears a BLE heart rate sensor, it is also connected to the glasses.

Using the screen, the agent will be able to see through the environment. The system will provide information about the risk of thermal shock (computed from the temperature and humidity information), the position of another agent and the structure of the building (and location-identified personnel) directly to the screen of the agent. Figure 6 shows a diagram of how data flows through the system.

4.4.1. Communication with Wearable Sensors

Every agent will carry the wearable electronics module (developed in the project), which will provide activity, fall report, temperature and humidity information. This data will be transmitted to the control center to provide a real-time action in case of health risk for the agent. In addition, the data will be available on the AR glass screen, so the agent will be able to take action depending on the situation.

The communication between the AR glasses and the wearable platform will use BLE. Every few seconds the wearable platform will send data to the glasses which will be processed to refresh its information.

4.4.2. Video Stream and Audio Stream

Every glass integrates a front-camera providing a real-time video stream on-demand to the control center. There, the coordinator can select the video source (at a specific video quality according to the available QoS) given the agent’s ID. This request will be transferred to the corresponding glass and will be processed by the communication module. When the request made by the ID of the agent reaches its glasses, the process uses a new video socket to send the recording to the core server.

The video stream will be transmitted through the existing Wi-Fi network. The streaming process will compress the video according to the criteria received from the control center. In order to reach the required QoS, the process will select the corresponding compression standards and parameters to reach the required data rate and video quality to provide the best stream according to the network status.

The audio stream process is the same as that of the video. The communication module waits for a communication request between the control center and any agent. A new communication channel is opened when a new request arrives. In this case, it is a bidirectional channel, where the agent or the control center can send audio feed to be received by the other.

4.4.3. AR Data

The main process that will generate Augmented Reality data is the core module of the AR environment which handles, under the supervision of the emergency coordinator, the transmission of the selected information from the three-dimensional multi-layer visualization database to the agent.

This process begins from a connection with the database server to get new data objects depending on its position. The selected data will be shown on the screen of each agent at a given depth, thanks to its three-dimensional side-by-side display capabilities. In addition, the smart glasses will show local information collected by the wearable platform and transmitted through the bluetooth interface to show thermal shock risk, temperature and humidity data. This data is also sent to the core platform using the video stream module.

5. Results

To test the system, we have done it in a modular way, testing each of the elements involved to check the correct behavior.

First, we have tested how the AR recreates a building using data from the three-dimensional modeler. The results were satisfactory. We were able to recreate the structures of the building and position different statically elements on the scene (Figure 7a). The next test was to assess the see-through environment and to check that the elements on the scene concurred with the elements on the real environment (Figure 7b).

We have tested the positioning system on an indoor environment. The test was carried out on Mirakonta’s office. Using RSSI as the localization method and deploying 20 landmark nodes in two floors, we were able to get a localization with an accuracy of 1.5 m and to position the agents in the virtual scenario. With these results, we are working on implementing some tests on angle of arrival to try to improve the results.

Finally, we want to test the system within an emergency drill. Two different teams without knowing where the victim is located, one with our system and the other one with the methods they apply on their emergency rescues. We plan to time both methods to determine if our system is helpful in such situations.

On

Table 1. Emergency Rescue Systems.

Aspect	Giuliano [4]	Li [6]	Simon [7]	Ahn [15]	Our Proposal
PositioningchensunanTechnology1	RFID	802.11b/g/n (SSID, RSSI)	868 MHz RSSI	Pedometry and AR tags	Bluetooth 5.0 (RSSI, AoA)
Positioning Architecture	RFID reader (p)	Mobile phone (p)chensunanFixed Wi-Fi access points	Moving Beacons (p)chensunanFixed landmarks	Mobile phone camera to read AR tags and room numbers	Moving Beacons (p)chensunanFixed chensunanLandmarks
Positioning Determination	RFID known position	Triangulation from an external central node	Triangulation from handheld device	Image-based positioningchensunanPedometric localization	Triangulation from an external central node
Communication2	GPRS/3G/LTE	GPRS/3G/LTE	No	3G/4G	Sub-GHz
Installation1	Require RFID tags	Deployed by first responders	landmark nodes position	AR tags	geolocalized nodes.
Navigation1	No	No	Display on device	Yes	Yes
Building model	No	No	No	Yes (3D)	Yes (3D)
Navigation3	No	No	No	Mobile phone	AR Helmet

(p): (personnel); ¹ Indoor; ² External; ³ Device

, we compare our proposal with other ones with similar implementations as our system is more complete in terms of navigation compared with the other systems.

Taking in comparison implementations of other studies, we can see differences. From the four implementations analyzed, the only one with augmented reality is the solution proposed by Ahn et al. [15]. The other three solutions lack a navigation system. Only Simon et al. [7], has a solution that gives feedback about the indoor position to the person using the system. In terms of deployment, all solutions except Li et al. [6] need a previous installation. On the contrary, our system can be previously installed or can also be deployed by first responders in case the fire destroys the landmark nodes installed or the infrastructure building does not have the system installed. Comparing the system used to transfer data to the outside, all the systems use mobile phone connections, although the coverage on big buildings is not always optimal. That is the reason why we decided to use a sub-GHz band to have long range coverage.

6. Conclusions

This system could help rescue teams on situations where there is no visibility. On large buildings like hospitals, it is relatively easy to get lost as you do not know what the structure of the building looks like. In order to do this, our system can be of great help to emergency services that need to move quickly to rescue trapped people by generating a three-dimensional recreation of the building and facilitating navigation through it.

We are currently considering the performance of a system test in a controlled environment. To achieve this, we will present the proposal to the fire service to carry it out. These tests will allow us to collect opinions from experts in order to introduce improvements on the system.