Design of Multimodal Sensor Module for Outdoor Robot Surveillance System

Abstract

Recent studies on surveillance systems have employed various sensors to recognize and understand outdoor environments. In a complex outdoor environment, useful sensor data obtained under all weather conditions, during the night and day, can be utilized for application to robots in a real environment. Autonomous surveillance systems require a sensor system that can acquire various types of sensor data and can be easily mounted on fixed and mobile agents. In this study, we propose a method for modularizing multiple vision and sound sensors into one system, extracting data synchronized with 3D LiDAR sensors, and matching them to obtain data from various outdoor environments. The proposed multimodal sensor module can acquire six types of images: RGB, thermal, night vision, depth, fast RGB, and IR. Using the proposed module with a 3D LiDAR sensor, multimodal sensor data were obtained from fixed and mobile agents and tested for more than four years. To further prove its usefulness, this module was used as a monitoring system for six months to monitor anomalies occurring at a given site. In the future, we expect that the data obtained from multimodal sensor systems can be used for various applications in outdoor environments.

1. Introduction

Studies using various sensors have mainly been conducted in confined environments for industrial monitoring and manipulation. Recently, they have attracted attention as a solution for wide-area security, owing to the development of networking and improvement in sensor performance. The data obtained from various sensors are used to recognize the environment and understand abnormal situations for security over a wide area. To build an autonomous surveillance system in an outdoor environment, a sensor system that is robust against weather conditions such as snow, rain, and wind that can be used through the day and night time is necessary. Most outdoor surveillance systems employ fixed vision sensors to perform monitoring by acquiring color and infrared images during the day and night [1]. Such vision-based unmanned surveillance systems can perform continuous monitoring, but the surveillance area is extremely limited. Surveillance systems other than those used for local monitoring should provide autonomous monitoring by using both fixed and mobile agents. Therefore, a method that is robust against vibration and has excellent durability is required for mounting fixed and mobile agents. A vision sensor system that can synchronize and collect sensor data for agents to detect and autonomously drive complex environments in outdoor surveillance systems is required.

Research using a variety of sensors in the field of surveillance has mostly dealt with limited environments such as industrial monitoring or manipulation. Many researchers have used vision sensors for decades in various fields. These sensors are used for vision-related applications, such as object recognition, tracking, and situational awareness [2]. However, because most technologies learn using sensor data collected in stable environments during the day, data from different environments are required. Many studies have been conducted to generate data using various sensors. The most representative method is to collect data by installing equipment on a vehicle that is driven around, such as in a mobile mapping system [3,4]. These methods are being used in various ways, such as data for confirming the conditions of roads [5], obtaining data for autonomous driving [6], and learning data for recognition [4,7]. However, it is difficult to apply the data collected using this method to the fixed and mobile agents of an autonomous monitoring system simultaneously. In other words, it should be possible to acquire and apply various types of data that can overcome changes due to weather and time in an outdoor environment.

There have been various approaches for collecting various types of data for the purpose of sensor fusion. First, monitoring is possible with a monitoring sensor module for healthcare in a residential environment, but it includes a sensor that can be used in forms of indoor services only [8,9]. Robotic multimodal sensors are used for manipulation [10]. They focus on grasping objects and can be table mounted. For agent motion, multimodal sensors for mobile robots use encoder force and distance sensors for detection, rather than vision sensors for accurate postures [11]. Multimodal sensors for mobile agents only focus on indoor localization [12]. Studies to overcome the challenges of the outdoor environment, such as snow, help autonomous driving by learning with color and infrared images; however, it seems insufficient to recognize abnormal situations in outdoor monitoring areas [13]. Multimodal sensors for object detection in marine environments use color-based learning and can only be used during the daytime [14]. Sensor fusion for autonomous vehicle navigation is a method that uses only the active depth information from the RGB and LiDAR cameras [15,16]. The method using video and sound can only respond to intruders in a room [17]. Because the outdoor autonomous monitoring system must be able to operate for 24 h and under all weather conditions, the sensor system needs to be equipped with multiple vision sensors and sound sensors.

A method using a pan and tilt camera or a multi-modal sensor to secure a monitoring area has been proposed for the monitoring system only [18,19]. A video-based security system using a mobile robot [20] has been proposed, though it has certain limitations in the surveillance area: however, only a fixed or mobile agent is of little use. Studies employing heterogeneous agents are not suitable for outdoor spaces covering a large area because they operate indoors [21,22,23]. Research on the connection between mobile robots and surveillance cameras is limited to indoor environments, and scenarios for surveillance and response missions have been specified [24]. Although a method that can monitor a wide area based on wireless communication has been introduced [25], this method refers to video streaming over a wide area. Therefore, it is necessary to employ both fixed and mobile agents to cover wide monitoring areas [26,27]. In conclusion, agents with the same sensor system can be designed for high-level monitoring systems because they can simultaneously monitor and respond to abnormal situations. In addition, images captured in hazy or foggy weather conditions are seriously degraded by the scattering of atmospheric particles, which directly influences the performance of outdoor computer vision systems [28]. The monitoring systems can improve data acquisition through effective filtering and compression for real-world use [29,30,31].

In this study, we propose a multimodal sensor module for multiple fixed and mobile agents in outdoor environments. The proposed module provides six types of vision sensor data and four-channel sound data that are synchronized and integrates them using 3D LiDAR and a calibration method. Moreover, this module enables integrated data collection for 24 h monitoring of the surveillance area. For long-term outdoor operation, heat resistance and durability were considered, and a vibration-damping device and cover were included. We verified its usefulness by performing long-term operations and monitoring scenarios on a real site.

The remainder of this paper is organized as follows. In Section 2, we introduce a multimodal sensor module designed and manufactured for outdoor agents, and in Section 3, we describe a method of acquiring data by mounting a module on agents and a calibration method. Section 4 shows the application of the multimodal sensor module. Finally, in Section 5, we conclude the study.

2. Design of a Multi-Modal Sensor System

2.1. Multi-Modal Sensor System Configuration

To employ multiple sensors in one module, it is important to unify the interfaces for power and data. Figure 1 shows the configuration of a multimodal sensor system for mounting on an agent. The multimodal sensor system includes multiple vision sensors and 3D LiDAR and employs a single power (48V, DC) supply and three interfaces (analog, USB 3.0, and GigE). It is enclosed in one case, which includes a seal, cooler, and sunshade, and is connected to a damper for low-frequency vibrations. The sensor data generated from this module were synchronized with the 3D LiDAR data and calibrated.

2.2. Overall Structure and Arrangement of Multi-Modal Sensor Module

The multimodal sensor module contains sensors and devices, including multiple cameras. Figure 2 shows the overall structure of the multimodal sensor module. First, a reference image was captured using an RGB image for the entire field of view. In addition, night vision images provide information even in a dark environment. The depth image can measure the distance to the object in a three-dimensional field of view. Furthermore, a thermal image sensor detects the thermal point and temperature range during an emergency, such as a fire. A microphone was installed on each of the four sides of the multi-modal sensor to detect abnormal sounds. Finally, considering the heat and vibration, a sunshade and damper, were installed on the module.

When multiple cameras and sensors are combined as a single module, the arrangement is important in order to calibrate the acquired data more easily. Therefore, as shown in Figure 3, cameras and sensors were placed at regular intervals along the horizontal and vertical directions. The RGB and Depth camera was installed horizontally. Two cameras were in-built into the module (Night Vision and RGB and Depth cameras), arranged at equal intervals (a). With one of these cameras as a reference, the night vision camera was directed in the opposite directions at the same distance. A global shutter camera was installed at a distance (b) vertically below the right reference camera. The thermal camera was located at a distance (a) horizontally from the global shutter camera and (b) vertically below the left reference camera. Placing the camera in this way reduces the factors that need to be considered when integrating information data and can benefit the computational process.

Table 1. Field of view for multi-modal sensors.

Sensors	Field of View
RGB and Depth Camera	IR(2EA) 85.2° (H) × 58° (V), RGB 69.4° (H) × 42.5° (V)
Thermal Camera	90° (H) × 69° (V)
Night Vision Camera	116.8° (H) × 101.3° (V)
3D LiDAR	360° (H) × 30° (V)

lists the field of view of each sensor installed in the module.

2.3. Waterproof and Cooling Design

Considering its use in an outdoor environment, as shown in Figure 4, the module can offer protection during rainy weather. The waterproof seal fits around the lens of the camera exposed to the outside environment. Moreover, the microphone sensors are included waterproofing. A cooling fan and hole for cable entry are located at the bottom of the multi-modal sensor. Therefore, the design includes a rain gutter for rainwater along the edge at the bottom to prevent rainwater from flowing down and into the interior of the module.

Because the inside of the module is dense, with cameras and sensors, which causes high heat, the design provides for heat dissipation, as shown in Figure 5. A cooling fan is located at the bottom of the multi-modal sensor to allow outside air, which is cooler than the inside of the module, to flow inside and the hot air to escape through the vents on both sides of the case. The cover of the vent has a certain depth (c) for preventing rainwater from flowing through the vent during rainy weather. The air heated by the camera and other devices inside the module collects in the upper space. Therefore, the case is made of aluminum, which has a high thermal conductivity and permits heat to be radiated owing to heat conduction. In addition, to increase the radiation of heat by increasing the area in contact with air, the outer upper region of the case is designed as a heat sink.

2.4. Sunshade and Damper Design

For outdoor use, a sunshade was designed because the device may be damaged by exposure to sunlight for a long time, as shown in Figure 6. The cover of the sunshade is provided with a mesh of black fabric to absorb heat from sunlight and provide good ventilation between the inside and outside of the cover. The guide frame material of the sunshade is plastic so that the heat of the shade does not easily reach the multimodal sensor.

A damper was designed to absorb the vibration of a moving mobile robot and obtain clear information from the multimodal sensor, as shown in Figure 7. The upper and lower frames of this damper were assembled using wire damper parts that were inclined at 45°, and this wire damper uses a total of six such wires at 60° intervals.

3. Multi-Modal Sensor Calibration Method

3.1. Multi-Modal Sensor Synchronization

For the synchronization of multimodal sensor data, the data acquisition system is implemented as a data acquisition process (Grabber process) and a data storage process (Saver process) with a common shared space in the memory, as shown in Figure 8. The Grabber process acquires information such as camera images and LiDAR sensor data for uploading data to the Memcached server, and the Saver process reads the data from the Memcached server [32] at regular intervals and saves it in a file format. Therefore, sensor data acquisition and storage processes can operate at different cycles within the system to transmit and receive data asynchronously.

3.2. Multi-Modal Sensor Calibration Method

A calibration method for the multiple types of image sensors mounted on a multimodal sensor module can be employed using a well-known method detailed in [33]. However, in the case of calibration between 3D LiDAR and color or thermal images, it is difficult to find a point on the 2D image coordinate system and a corresponding point on the 3D coordinate system because of the low resolution of the point cloud of the LiDAR. Figure 9 shows that it is difficult to accurately detect the vertex of a rectangular calibration board on a 3D point cloud. To calibrate these heterogeneous sensors, black and white patterns and vertices of the pattern board can be detected in the image using a Harris corner detection algorithm, and the coordinates of the vertices of the board are designated as the corresponding points used for the calibration. Next, the distance from the LiDAR sensor to the board is specified in advance to extract only the data that the point cloud forms on the pattern board from the 3D LiDAR, and filtering is performed using the value of the distance, as shown in Figure 10. Subsequently, only the point cloud information corresponding to the plane is found using the plane model segmentation algorithm, which predicts and classifies the plane based on the random sample consensus (RANSAC) algorithm.

Next, a three-dimensional plane equation is derived using the filtered information of the 3D coordinates of the point cloud. At this time, the RANSAC algorithm-based plane fitting algorithm was used for plane detection, and the parameters of the 3D plane equation with high satisfaction were obtained, as shown in Figure 11.

To obtain the coordinates of the point cloud formed on the edge of the calibration board, the first and last data for each channel of the LiDAR were obtained and then projected on a 3D plane. Then, using these data as inputs, we used the RANSAC-based line fitting algorithm to find four directional vectors that were orthogonal to each other through a relational analysis of the data and derived the line equation at the corner of the calibration board that the closely matched the data, as shown in Figure 12.

Figure 13 shows that extrinsic calibration can calculate the final calibration parameters from the corresponding points obtained through the previous procedure.

Finally, as shown in Figure 14, the calibration method is applied using a board including a heating wire in the grid so that the same method can be employed for color and thermal images. Figure 15 shows that the proposed method can increase the accuracy compared to finding the existing click-based correspondences. For the outdoor environment, the synchronized image, including the calibration result, can be obtained, as shown in Figure 16.

4. Application of Multi-Modal Sensor Calibration Method

4.1. Computing for Multi-Modal Sensor System

To apply the proposed multimodal sensor module to a surveillance system, a computing device capable of processing the acquired sensor data and transmitting it to the control system is required. For active security, modules that synchronize and acquire sensor data, recognize a moving object based on the acquired data, and recognize an abnormal situation are installed to communicate with the control room. The computing system includes a module, mounted on the fixed and mobile agents, together with a multimodal sensor module, as shown in Figure 17.

4.2. Multi-Modal Sensor System Install and Test

The proposed multimodal sensor module is installed in the Safety Robot Demonstration Center, Pohang and Nano Industrial Complex, Gwangju, Korea. The Pohang Safety Robot Demonstration Center is an area where external access is controlled, and the Gwangju Nano Industrial Complex is an area that includes public roads, as shown in Figure 18. After installation, it has operated continuously for more than 4 years, and long-term operation and testing based on day/night security scenarios have been performed, as shown in Figure 19. In contrast, multimodal sensor data obtained from fixed and mobile agents recognize abnormal situations using multilayer probability maps, as shown in Figure 20. Finally, Figure 21 shows human and car detection results for the proposed sensor module performance. The multi-modal sensor data can robustly detect objects of interest during not only the day- and nighttime, but also in most weather conditions such as fog and rain. They were able to recognize and respond to abnormal situations, such as fire, illegal parking, noise, and intruders [34].

5. Conclusions and Future Works

In this study, we introduced the design and implementation of a multimodal sensor module for a surveillance system. This module contained various types of cameras and was designed to be waterproof and to dissipate heat effectively. A device that provides a simple interface and power supply and one that attenuates vibration during movement was also included, and a variety of cameras were employed. The proposed sensor module provides corrected data by calibrating the synchronized sensor data (See Supplement). This module can be employed for both fixed and mobile agents and has been in operation for more than four years.

Moreover, it was installed at the Safety Robot Demonstration Center, Pohang and Nano Industrial Complex, Gwangju, and operated as a surveillance system for more than six months to complete the demonstration test consistently. Therefore, it may be possible to recognize and respond to abnormal situations by using a multimodal sensor module during the daytime and nighttime. In other words, multimodal sensor modules have proven to be a useful tool for all-time outdoor security.

In the future, we plan to upgrade the module to be applicable in harsher environments (e.g., polar regions) and evaluate its usefulness. In addition, we plan to apply this to agent systems in other fields, such as medical institutions, smart factories, and Arctic exploration.