Application of AIoT Image Sensor for Lifting Operation Safety Monitoring of Mobile Crane ^†

Abstract

The emerging advances in deep learning and computer vision have enabled traditional cloud-based decision-making through edge computing with artificial intelligent internet of things (AIoT) image sensors (AIoT-IS). As a result, the timeliness and security of image recognition can be obtained. This study aims to develop an AIoT-IS-based smart safety control device for construction sites (“SmartCon” hereafter) for the operations of mobile cranes. The research is carried out to include the definition of hazardous control areas and the identification of unsafe scenarios for crane operations, the development of an intelligent prototype system for the safety control of lifting operations, system application demonstrations, and the evaluation of effectiveness. It is intended to assist labor inspection agencies and industry practitioners with a tool for monitoring lifting operations. The results of two empirical field tests show that the proposed SmartCon improves the safety monitoring of the operation of the machine through the real-time response to the identified potential risks on construction sites.

1. Introduction

The construction industry is considered a key driver for economic development by the governments of most countries in the world. According to the latest statistics from the Industry and Service Census conducted by the Directorate General of Budget, Accounting and Statistics (DGBAS) of the Executive Yuan in Taiwan (R.O.C), the construction industry employs more than 913,000 workers, which accounts for 7.49% of the total employed population [1]. More than 2.49 million family members are supported by construction employees. About 3.77% of Taiwan’s gross national product (GNP) is contributed by the construction industry. Dividing the industry’s output value by the number of employees shows that the per-capita GNP of the construction industry is significantly lower than the national average. On the other hand, the income of 913,000 construction workers is the main source of income for their families. When occupational accidents occur, the families of wounded workers often face a significant economic impact. This, in turn, affects the whole society.

Construction accidents have long taken up the highest percentage of industrial and serious occupational accidents around the world. There are significant harmful impacts on industrial production and workers’ lives. The main reason is that construction sites are open to risks.

According to the latest statistics from the DGBAS of the Executive Yuan in Taiwan in 2019 [2], there were 313 significant occupational casualties in Taiwan’s entire industry, of which the construction industry accounted for 46.32% (a total of 145 casualties). Statistics from 2012 to 2020 showed the total number of fatal occupational accidents in the industry was 2879, and the construction industry accounted for 47.65% (a total of 1372 casualties), which was about half of that of all industries. These statistics highlight the severity of safety issues for construction workers.

Referring to Article 22 of “Taiwan’s Enforcement Rules of Occupational Safety and Health Act”, dangerous machinery includes fixed and mobile cranes. According to statistics collected by earlier research [3,4], there have been 12 to 19 major occupational accidents involving mobile cranes in Taiwan annually since 2002. Between that year to 2012, 104 people died of these accidents. Three people suffered from minor injuries, whereas ten people suffered from serious injuries. In the analysis of the causes of accidents, “management factors” constituted the most significant portion (66%), followed by “human factors” (22%), “equipment factors” (8.5%), and “environmental factors” (3%). Materials falling, cranes tipping over, being caught, being hit, and electric shock are the leading causes of accidents. Although the relevant safety regulations have been issued by the competent authorities [5], the primary approaches to safety management and disaster prevention still mainly rely on manual inspection by occupational safety management personnel. There are still problems in practice. Even with detailed laws and regulations governing occupational safety, reducing site risks and implementing immediate control and prevention remain challenging. A promising solution for construction site safety management is to use information and computer technology (ICT) technology to help site safety managers improve the deficiencies of manual safety management.

Deep neural networks (DNN) have advanced quickly during the past decade as an emerging artificial intelligence (AI) technology [6]. Machine learning (ML) can judge and evaluate a worker’s safety on site in real time. Based on the above discussion, this research is carried out to apply AI technology based on DNN to the technology for construction site visual recognition and analysis to detect labor safety hazards of workers automatically during crane operations. The goal is to provide early warning of labor hazards to reduce or avoid the occurrence of construction accidents on site and achieve a safer working environment for construction workers.

The rest of this article is organized as follows. Section 2 sorts out some related works about construction safety. The methodology and framework of the proposed method are described in Section 3. The testing results are shown in Section 4. Lastly, Section 5 concludes this study.

2. Related Works

In recent years, various advanced technologies have been developed to assist in the safety control of construction sites and reduce the risks caused by lifting operations. For example, the Taipei City government in Taiwan continues to promote smart city-related plans, one of which is to use Bluetooth low-energy (BLE) technology to assist in the safety management of construction sites [7]. This technology installs Bluetooth transmitters (iBeacon) on the worker’s helmet or personal protection equipment (PPE). When the Bluetooth receiver receives the signal from the transmitter, it accurately calculates the position of the personnel or equipment, thereby notifying the construction workers to avoid entering the safety control area. In addition, CTCI Inc. developed technology to calculate the construction site by recognizing the triangular cone of lifting operations. It sends a warning to notify the workers when they work under lifting objects or pass through the range of a construction area [8]. Japan’s Taisei Corporation also uses the GPS positioning of construction machinery and construction workers to remind the workers not to enter the lifting area of construction machinery [9]. If workers are close to construction machinery, alarms are sent to smart watches.

In addition to general site safety control technology, the Swedish company Gigasense has developed a set of weight sensors installed on lifting equipment to monitor equipment in real time to ensure the balance state of lifting operations [10]. China Xiamen Baima Technology has also developed a remote monitoring system to monitor the lifting operations in a construction site [11]. The various sensors installed on the lifting equipment can obtain the data of the operation in real time and carry out calculations to identify the limitations of lifting operations such as anti-collision calculation, weight limit, torque limit, height limit, amplitude limit, angle limit, wind speed alarm, and main track anti-tilt.

Nath et al. proposed a construction site safety monitoring method in 2020 [12], which applied the convolutional neural network (CNN) to identify and monitor the protective equipment of workers. The method achieved 72.3% mAP (mean average precision) at an 11-FPS (frames per second) detection speed. However, the proposed method can only identify the correct wearing of labor personal protective equipment and no other automatic hazard identification functions.

Golcarenarenji et al. proposed a method called CraneNet in 2021 to detect people under crane lifting equipment [13]. In this method, a camera is hung on a hanging hook, and an image is transmitted to a small computer with NVIDIA Jetson Xavier through a wireless network. The ML technology based on CNN is applied (YOLOv4) to detect the position of personnel under crane lifting operations. On construction sites with complex environments that require real-time detection, this method achieves an accuracy of 92.59% at a detection speed of 19 FPS, which is close to the accuracy of human judgment. Although the accuracy is improved, the disadvantage of the CraneNet method is similar to the limitation of the method of Chen and Fang et al. [14,15]. Misrecognition was found due to the dynamic change of the background or the similarity of the object’s color to the background. Therefore, there are still difficulties in application in construction site practice.

On the other hand, Liu et al. proposed a method in 2021 to detect whether people are dozing off to prevent accidents caused by operational errors [16]. They used a camera to detect a driver’s face through DL technologies using CNN and LSTM and then extract the facial features to classify the driver’s behavior such as talking, sleepy eyes, yawning, and napping. If any behavior that affects safety is detected, it warns the driver to avoid accidents.

The above summarizes the advanced technology and latest research literature on the application of image recognition and sensing technology to the safety control of construction lifting operations. This technology’s technical viability, maturity, mobility, and economic viability must be considered to make the technology applicable to the actual construction site in this study.

3. Proposed Method

In this section, we describe the design of the proposed AIoT device, the smart construction site safety control device (SmartCon), for the lifting operations of mobile cranes on construction sites.

3.1. System Architecture

The SmartCon is an all-in-one and single independent device that can operate independently. The schematic design of the proposed SmartCon is shown in Figure 1.

To meet the practical needs of the construction site, multiple control devices can be moved and set up quickly at different locations according to the on-site environmental conditions to monitor multiple cranes and multiple lifting operations. The SmartCon uses a customized extendable stand to set up a waterproofed IP camera, network device, AI development board, alarm, battery, and other devices. All the devices and a battery must be set up in the case (except the IP camera) to make sure that the SmartCon is waterproof for all weather.

The following SmartCon features, including portability, heat dissipation, and water resistance, are considered in designing the SmartCon.

• Portability: the extendable stand includes all components, including the battery, controller, computing devices, and IP camera. Furthermore, the Jetson Nano computing device utilized in the design has a compact size, is lightweight, and has low power consumption, which increases the mobility of the SmartCon.
• Heat dissipation: considering the SmartCon’s outdoor operation requirement, the SmartCon may need to operate under the hot sun for a long time. If it is overheated, it damages the hardware. In this regard, we must consider isolating the external temperature and strengthening the internal heat dissipation. The internal heat dissipation is related to the device’s portability. The smaller the size, the higher the portability. By contrast, the interior heat dissipation space of the device is smaller. Therefore, it is necessary to install some fans to ensure that the heat in the space can be dissipated.
• Water resistance: considering the uncertainty of the weather at the construction site, the SmartCon may be used on rainy days. Thus, water resistance must be implemented to ensure that the device circuit will not be damaged.

The system architecture of the SmartCon is shown in Figure 2. The blue block is the hardware module in the device, including a 4G Wi-Fi router, Nvidia AI development board, RF receiver, alarm, and IP cam. The green blocks are the operator’s smartphone device and RF remote control. Furthermore, the device is connected to Line Notify for alert notifications.

The detailed specifications of the above components are described as follows.

• 4G Wi-Fi router: the SmartCon uses a 4G Wi-Fi Router to establish local networks for connecting devices. The IP camera uses PoE (power over ethernet) for the power supply and network connection. The router also connects the Jetson^® development board with ethernet. Externally, the 4G network is used for external communication, e.g., Line notification.
• Jetson development board and core: the SmartCon uses the Jetson Nano^®, an embedded system developed by Nvidia^®. The development board contains a high-performance computer. The hardware design is optimized for AI computing, and the product has the features of a small size and low power consumption. The small size reduces the weight and volume of the SmartCon, and the low power consumption reduces the battery capacity requirement. Since the battery capacity is proportional to the weight and volume of the battery, the physical volume and weight can be reduced by using the selected development board.

The proposed core includes a network module, video streaming module, AI recognition module, GPIO control module, Line API, and electronic fence module. The video stream of the IP camera is communicated through the RTSP streaming protocol and then detected by the AI recognition module. In this study, YOLOv4 is used for object detection. YOLO is a neural network for object detection characterized by its light weight and high efficiency. YOLOv4 is implemented using the darknet architecture. After identifying the object, it determines whether it conforms to the set rules of the electronic fence. If the preset conditions of the alarm are met, the recording and alarm notification are performed, including Line notification and the activation of the alarm.

• IP camera: the SmartCon uses a network camera with full HD (FHD) resolution, a PoE power supply, and a real-time streaming protocol (RTSP) as the video source. The characteristics of the IP camera in this study are as follows.
  • FHD resolution: the image resolution is proportional to the recognition rate, but inversely proportional to the recognition speed. Such cameras transmit lower-resolution images that need to be adjusted according to the requirements to achieve balance with speed.
  • PoE power supply: although this system provides Wi-Fi as the function of device connection, the wired network is still more stable than the wireless network. Users can connect to the internet without a network line through Wi-Fi, but it still requires a power line for the power supply. Through the PoE function, it only needs a single network line to power on, and connects to the internet simultaneously.
  • Real-time streaming protocol (RTSP): RTSP is an application layer communication protocol for multimedia streaming. It is used in multimedia transmission to establish a connection between a multimedia server and a client to monitor multimedia.
• Smartphone: the web interface can be used to set and monitor the SmartCon. Users can use any smartphone with a web browser and connect to the 4G Wi-Fi Router. The web interface functions include video streaming monitoring, electronic fence settings, enabling and disabling controls, and more parameter settings.
• Remote control: the SmartCon system provides a radio frequency (RF) remote controller, which is convenient for users to perform wireless control operations quickly. This RF communication refers to radio waves with frequencies ranging from 3 kHz to 300 GHz to transmit over long distances.
• Alarm: to create a more significant warning impact in the construction site environment, the SmartCon employs a higher-power alarm as a warning device.
• Line notification: when the danger alarm is triggered, the system transmits the time, place, and photo through the Line API to a Line account owned by the relevant personnel managing the construction site. This is convenient to understand the situation on the construction site quickly.

3.2. Deep Learning Module

As described in the previous system architecture, the SmartCon applied YOLO object recognition techniques for AI recognition module development, and the model training dataset uses the MS COCO dataset [17]. The MS COCO dataset is a massive object database with many images; more than 200,000 objects are labeled in the images. These labels are divided into 81 categories, with detailed information, including object location and image context.

3.3. Development Board

In order to reduce the cost of system implementation, we use the “Jetson Nano” for system development. “Jetson Nano” is the basic entry-level model of the Jetson embedded systems. Although the hardware capability is the lowest, it also has the lowest power consumption and weight, which reduces the battery capacity requirement and the weight of the host. Moreover, it optimizes the system environment and code as much as possible. The maximum hardware capacity is used to reduce hardware costs. As the functions increase and the system operation burden increases, higher-end hardware can be adopted in the future.

When image data are transmitted through the network, the image data are usually compressed through image coding to reduce the bandwidth required for transmission. Therefore, the receiving end must decode the information to obtain the original multimedia information. The CPU usually processes this decoding work, but the Jetson Nano^® is equipped with a multimedia decoding chip, which effectively improves the speed of the system reading camera images and reduces the CPU load.

4. Implementation and Testing

This section presents the implementation of the SmartCon and the experimental results in real-world construction sites.

4.1. System Implementation

The hardware architecture of the SmartCon is shown in Figure 3. The device is a stainless steel waterproof case covering the battery and electronics. On one side of the stand, the tube can be derived to give the camera a better angle for monitoring. The size of the battery case is 44 cm × 30 cm × 24 cm (width × height × depth). The size of the second case is 44 cm × 30 cm × 24 cm (width × height × depth). The extendable stand can be extended to 148 cm. The total weight without a battery is about 10 kg, whereas it is about 12.8 kg with a battery.

The operation web interface of the proposed system is shown in Figure 4, which includes the following functions: user login, real-time screen, control area setting, fine-tuning button, undo setting, clear area, status, enabling/disabling controls, showing alarm status, modifying the location name, modifying the equipment name, setting Line notifications, system log, and shutdown.

Users need to log in with a password to access the web interface. The user can monitor the real-time IP camera image in the interface and set the range of the control area. After enabling the control state via a one-click button, the workers are not allowed to enter the preset control area. If any workers enter the control area, the alarm is set, and the Line notification is sent immediately.

4.2. AI Model Selection

The AI recognition module refers to the preset usage situation. We set the camera resolution to 1920 × 1080, and set the camera at a height of 10 m to overlook the ground for actual testing. During the test, a video is recorded in this environment. The video has 495 image frames recording one worker passing through the control area. Figure 5 is a screenshot of the two experimental site recognition results. The recognized object (person) is displayed on a frame line with a confidence value (the larger the value, the more similar the object, a worker, is).

In this test, various YOLO models were used for the object recognition test, and the confidence value was set to be greater than 0.7 before the recognition result was passed. The recognition rate of the model for this environment was calculated through the inference results of each model.

The actual test data results in

Table 1. Evaluation of Recognition Precision and FPS.

Model Name: Version-Pixel Size	Recognition Precision	Frame Per Second (FPS)
yolov3-tiny-288	11/495 = 0.022	26.54
yolov3-tiny-416	41/495 = 0.082	20.40
yolov3-288	157/495 = 0.317	7.56
yolov3-416	266/495 = 0.537	4.70
yolov3-608	290/495 = 0.585	2.40
yolov3-spp-288	181/495 = 0.365	7.80
yolov3-spp-416	113/495 = 0.228	4.68
yolov3-spp-608	294/495 = 0.593	2.45
yolov4-tiny-288	28/495 = 0.050	26.72
yolov4-tiny-416	81/495 = 0.163	20.40
yolov4-288	246/495 = 0.496	7.56
yolov4-416	298/495 = 0.602	4.46
yolov4-608	321/495 = 0.648	2.30
yolov4-csp-256	42/495 = 0.084	11.80
yolov4-csp-512	269/495 = 0.543	4.00
yolov4x-mish-320	176/495 = 0.355	4.60
yolov4x-mish-640	291/495 = 0.587	1.44

show that the yolov4-608 (FP16) model obtained the highest recognition rate, and the subsequent system deployment is based on yolov4-608 (FP16).

4.3. Power Consumption

The developed equipment is easy to carry and is not troubled by the power supply. Thus, a large-capacity battery of 96,000 mAh is set up to power the system’s operation. After the system starts to detect, the maximum wattage detected per hour is 23.5 W. We estimated that the battery could run the system for about 13 h. The battery can run the system for 13.06 h in theory. In the field test, the SmartCon could be used for about 10 h.

4.4. System Evaluation

Table 2. Evaluation of response time and alarm trigger accuracy.

Experimental Site	Threshold	Average Response Time (s)	Alarm Trigger Accuracy
Site 1	0.7	0.53	100.00%
Site 1	0.6	0.53	100.00%
Site 1	0.5	0.53	100.00%
Site 2	0.7	0.53	98.80%
Site 2	0.6	0.53	99.60%
Site 2	0.5	0.53	100.00%

shows the evaluation results of the two sites. In this experiment, we calculate the accuracy with only the alarm triggered by the SmartCon when workers enter the control area. For the alarm trigger accuracy, it reaches over 98% in the threshold of confidence between 0.5 to 0.7. In this test, 0.5 is better for the alarm trigger accuracy. The Jetson Nano^® development board has been optimized as systematically as possible to maximize its benefits. The average response time is 0.53 s in Jetson Nano^®. Without cost limitations, it could recognize larger or more real-time images if upgraded to higher-end hardware (for example, Jetson AGX Xavier). The price will undoubtedly rise along with the upgrade.

4.5. Comparison

We also developed a previous device version with comparable features to the one developed in this study. To examine the differences between the previous and current versions, we compared the outcomes of this research and the previous systems. The comparisons are shown in

Table 3. Comparison of Previous and New-version Devices.

Items	Previous Version	Current Version
System architecture	Server and client (compute in server)	AIoT (compute in edge, Jetson)
Hardware architecture	Tripod stands with a case, but needs to set up the server host	All-in-one
Ease of installation and storage	Hard	Easy
Detection speed	Normal	Slow
Connection response	Slow	Fast
Water resistance	Not Available	Yes
Heat dissipation	Not Available	Yes
Battery capacity	53,600 mAh	96,000 mAh

. Figure 6 shows the previous edge device without computing. All of the image recognition must be recognized in the remote server (with GPU). The previous version device was developed with a server and client system (cloud-based) architecture, so the device is hard to install and store. Even if the detection speed is better than the current version, the connection response becomes slow when the internet connection is unstable.

By contrast, the current version has advantages because it is developed with edge computing. The AI algorithm can be processed at the edge, so the overall device is designed as all-in-one, which is convenient for installation and storage. Table 3 shows that the new design is more portable and convenient with water resistance and heat dissipation, making the equipment more suitable for construction management environments.

5. Conclusions

The proposed SmartCon offers an effective tool for improving the safety monitoring of construction site operations by a real-time response to detecting potential risks, according to the findings of two empirical field tests.

This experiment was conducted on a real-world construction site, with Earthpower Construction providing a site for us to finish the experiment. The cost of hardware parts was about USD 1600 excluding assembly, software system installation, maintenance, and repair costs. In the future, the price of hardware will become lower. If the user uses a chip with better performance to achieve a more accurate or faster recognition effect, the cost of the hardware will increase to upgrade the level of the Jetson development board.

We hope to deploy the SmartCon to more construction sites to obtain more test results in future research. Furthermore, we also plan to complete the detection of helmets and vests to better ensure the safety of workers in the monitoring area to make the system perfect.