Drone-Aided Path Planning for Unmanned Ground Vehicle Rapid Traversing Obstacle Area

Abstract

Even with visual contact equipment such as cameras, an unmanned ground vehicle (UGV) alone usually takes a lot of time to navigate an unfamiliar area with obstacles. Therefore, this study proposes a fast drone-aided path planning approach to help UGVs traverse an unfamiliar area with obstacles. In this scenario called UAV/UGV mobile collaboration (abbreviated UAGVMC), a UGV initially invokes an unmanned aerial vehicle (UAV) at the scene to take a ground image and send it back to the cloud to proceed with object detection, image recognition, and path planning (abbreviated odirpp). The cloud then sends the UGV a well-planned path map to help traverse an unfamiliar area. This approach uses the one-stage object detection and image recognition algorithm YOLOv4-CSP to quickly and accurately identify obstacles and the New Bidirectional A* (NBA*) algorithm to plan an optimal route avoiding ground objects. Experiments show that the execution time of path planning for each scene is less than 10 s on average. It does not affect the image quality of the path map. It ensures that the user can correctly interpret the path map and remotely drive the UGV rapidly, passing through that unfamiliar area with obstacles. As a result, the selected model can outperform the other alternatives significantly by average performance ratio up to 3.87 times on average.

1. Introduction

The Internet of Things, system-on-a-chip, dynamic random access memory, flash memory, and wafer process have made significant progress in recent years. Likewise, wireless networks have also developed Wi-Fi 6 and 5G. Traditional industries combined with emerging high-tech have greatly improved production efficiency, speed, automation, etc. Industries even entered the stage of intelligent production, and they can identify errors and issue warnings in advance or eliminate abnormal conditions by themselves. Advances in these industries have also led to unmanned vehicles such as unmanned aerial and ground vehicles (UAV/UGV). The open-source control chip [1], open-source mission planning [2], and dedicated transmission protocol [3] enable UAVs to execute the planned flight path and load the robot operating system (ROS) [4] into the development board for more complex manipulation and control. UAV is no longer mainly based on fuel-engine remote control fixed-wing but has integrated a multi-axis gyroscope, barometer, and satellite positioning control chip to develop a multicopter with a stable attitude. At present, applications can even control the cluster of thousands of UAVs through Intel technology [5]. These advances have also allowed UAVs to expand further. In addition to the multi-cluster control of UAVs, the batteries, motors, self-stabilizing gimbals, servo motors, cameras, and racks have all made significant advancements in technology, allowing UAVs to carry corresponding payloads as per different tasks [6]. These advanced technologies promote the expansion of the use of unmanned vehicles. In contrast, unmanned ground vehicles (UGVs) equipped with the same control system can also obtain greater flexibility in executing tasks.

The progress of the semiconductor manufacturing process has dramatically improved the computing power of chips, especially the visual processing computing unit [7] for single-purpose professional use. Advanced equipment allows deep learning or machine learning to be used widely in various industries [8], rapid object detection, and image recognition development. It improves the speed of object detection and maintains or even improves image recognition accuracy, which significantly impacts the operational detail of self-driving cars and the visual processing capabilities of unmanned vehicles. A UAV in the air assists in capturing images in a ground scene with spreading obstacles and then sends that image back to the back-end server for path planning through powerful visual computing and processing. In such a way, the server results in a map with a planned path and will promptly send it to the UGV on the ground. According to the guideline of this map, a UGV can traverse the scene smoothly without any collision with obstacles. It is not like a sweeping robot that slowly explores any route to avoid hitting obstacles and builds a map of the safety movement [9]. In short, the mobile collaborative operation of unmanned vehicles can significantly improve the time consumed by helping UGV traverse an unfamiliar area with obstacles very rapidly.

2. Related Work

The object detection methods have successfully developed their specific applications for years. In addition to high-end computing platforms, they can also operate on systems with low computing resources. Tran et al. studied an optimization solution to the hyperparameter tuning of an adaptive learning system [10] to improve object recognition accuracy. According to collected data from movement with the advanced driver assistance system (ADAS) to evaluate the previous convolutional neural network (CNN) models, the proposed method is to develop a framework for searching a set of learning hyperparameters. The trained model can be more intelligent than the previous model and show more diversified recognition results. Updating the model is repeated continuously throughout the ADAS life cycle. Mao et al. proposed a pipelined object detection method applied to embedded platforms [11]. They comprehensively analyzed the existing object detection methods and chose a fast region convolutional neural network (Fast R-CNN) as a possible solution. Furthermore, they modified the Fast R-CNN object detection method to balance speed and accuracy when operating on an embedded platform. This approach applied a multi-layer pipeline parallel operation method to an embedded platform with CPU and GPU, which can completely use the limited computing resources.

Both unmanned aerial vehicles (UAV) and unmanned ground vehicles (UGV) have recently developed rapidly, extending more application directions. Li et al. used UAVs to retain a ground image in the air and then constructed a ground map through image denoising, correction, and obstacle identification. They proposed a hybrid path planning algorithm [12] to optimize the planned path. They used a genetic algorithm to search the overall path and continuously used partial rolling to optimize the genetic algorithm’s results. Qin et al. proposed a new integrated vehicle system that uses UAV and UGV to work together [13] for autonomous exploration, mapping, and navigation in a 3D unknown environment where unmanned vehicles cannot use GPS. Experiments have proved its ability to realize heterogeneous UAV and UGV collaborative exploration and environmental structure reconstruction through active simultaneous localization and mapping (SLAM) and provided the optimal perception for navigation tasks. An et al. applied a self-organized aerial and ground group detection to achieve re-entry aircraft landing detection [14], for example, a search and rescue aircraft. The detection group consisted of multiple UAVs and UGVs. UAVs can rapidly detect objects and achieve high mobility, while UGVs can do more comprehensive detection due to the wide variety of equipment carried. These individuals make control decisions independently from others based on the self-organization strategies. After analyzing the overall requirements of group detection, the self-organization strategy to control group detection is appropriate and expandable.

To sum up, all UAV/UGV collaborations mentioned above will take time to complete the mission. It is necessary to develop a novel approach to resolve the time-consuming problem in ground-air vehicles collaboration. In particular, a rapid UAV-assisted path planning approach to help UGVs pass an unfamiliar area with obstacles is faster than the one without UAV-assisted path planning.

3. Method

3.1. Unmanned Aerial/Ground Vehicle Cooperative Operation

This part will introduce unmanned aerial/ground vehicle cooperative operation, including system architecture, unmanned ground vehicle (UGV), and unmanned aerial vehicle (UAV).

3.1.1. System Architecture

This system composes an integrated control interface through UAV, UGV, web server, cloud database, and Android apps [15]. According to the applications of the Internet of Things, unmanned aerial/ground vehicles synchronize the mobile phone, and the app in the mobile phone directly displays the streaming images taken by the camera or the thermal images captured by the infrared thermography on the screen through the wireless network, as shown by arrows 3 and 5 in Figure 1. This system uses the open-source software PhpMyadmin to build cloud computing and then uses Anaconda to build object detection, image recognition, and path planning services. The user directly controls the UAV/UGV through the app on the mobile phone. The embedded platform on the vehicle will continuously read the data from the cloud database and perform corresponding actions, as shown by arrow 1 in Figure 1. For the vehicle, the app in the mobile phone will control it to upload the videos or images captured by a camera to the cloud for storage in real-time, without occupying the storage space in the embedded platform. The vehicle’s monitoring system has four gas sensors, which can detect up to 12 gases such as CO, CO₂, PM2.5, liquefied petroleum gas, etc. The detected gas information will be sent back to the database for storage in real-time, as shown by arrows 2 and 4 in Figure 1. Therefore, the user can read the air quality situation around the vehicle in real-time through the app on the mobile phone. As for the mobile collaboration between the UGV and UAV, the UAV first captures the ground image and sends it back to the server. Next, the server recognizes obstacles in the captured image and then performs path planning in the same image to turn out a planned-path map. Finally, the server transmits this map to the app on the mobile phone, and according to the planned path on this map, the user using the app on the mobile phone controls the UGV remotely, passing the scene smoothly, as shown by arrows 6 and 7 in Figure 1.

3.1.2. Unmanned Ground Vehicle (UGV)

Several modules build an unmanned ground vehicle, as listed in

Table 1. UGV modules.

Number	Module
1	Arduino UNO R3
2	MPU6050 Gyroscope
3	TB6612FNG Motor Driver Board
4	GB37 Deceleration Motor
5	ESP8266 WiFi Module

. Figure 2 shows an unmanned ground vehicle with a multipurpose monitoring system (iMonitor) mounted to the UGV for surveillance [16]. Three lenses are on the front panel of the iMonitor; the one on the left side is a thermal camera. After quantifying the captured data from the lens, a thermal camera can obtain the surface temperature distribution around the object in the field of lens view and generate a thermal image. The lens in the middle charges the video streaming and face recognition. Once it has detected a face successfully, the app in the mobile phone will automatically mark it with a red box and synchronize the video streaming through the wireless network. The last one on the far right is responsible for recording videos and taking pictures. Due to the performance limitation of Raspberry Pi, UGV realizes the functions of recording videos and taking pictures by installing another camera. The iMonitor has also installed four gas sensors on the side panel of the box.

Table 2. iMonitor modules.

Number	Module
1	Raspberry Pi 3B
2	Raspberry Pi Zero
3	Arduino UNO R3
4	Intel Movidius MA2450 Vision Bonnet
5	Raspberry Pi Camera V2.1 Camera
6	MLX90640 Thermal Camera
7	SHARP GP2Y1051AU0F PM2.5 Sensor
8	DS-CO2-20 Sensor
9	MQ-4 CH4 Sensor
10	Grove Multichannel Gas Sensor

lists the modules used in the iMonitor. The app in the mobile phone can display the gas information for the user in real-time, which visualizes the air quality of the surrounding environment around the UGV.

3.1.3. Unmanned Aerial Vehicle (UAV)

Figure 3 shows a self-assembled quadcopter ZD550 equipped with a multipurpose monitoring system (iMonitor), the iMonitor is shown in Figure 4. The several modules that build an unmanned aerial vehicle (UAV), are listed in

Table 3. UAV modules.

Number	Module
1	Pixhawk 2.4.8 Flight Controller
2	M8N GPS Module
3	AT9S Pro 12 Channels Remote Controller
4	R9DS 9/10 Channels Receiver Module
5	433 MHz 100 mW Radio Telemetry Module
6	VX30 5.8 GHz 800 mW Video Transmitter
7	5.8 GHz USB Video Receiver
8	TAROT TL68A00 2-Axis Gimbal
9	Sony IMX477R with 6 mm Lens
10	ZD550 55 cm Quadcopter Frame
11	TAROT 6S 4108-380KV Motor
12	14 inch Carbon Fiber Propeller
13	40A Brushless ESC

. The purpose of a UAV equipped with an iMonitor is first to collect environmental hazardous gas information during the flight, especially in high-risk and ground-obstructed environments such as disaster sites, chemical plants, and so on [17]. After sensing the hazardous gas information, the iMonitor transmits it to the cloud server for storage immediately, and users can browse it on mobile phones in real-time. Next, the iMonitor can carry out surveillance from the in-flight camera and send the real-time captured images back to the in-cloud server. The recognition system uses these images to recognize faces, whether a human, a logged-in face, or a stranger with intruding intention. Finally, the iMonitor uses a thermal sensor lens to detect abnormal heat sources during UAV patrolling in sensitive areas. For example, it can monitor a fevered subject with a higher body temperature on the spot.

3.2. Android Client

The user uses the Android client app to perform operations such as streaming, shooting, setting, execution, and reading required for path planning. In the UAV operation interface, there will be a display streaming button (face indicated CAM) and a path planning button (face indicated PATH), as shown in Figure 5a. After pressing the CAM button, a streaming screen and a shot button (face indicated SHOT) will appear, as shown in Figure 5b. After pressing the SHOT button, the app takes the current image and uploads it to the server for storage. After returning to the initial interface, the user can press the PATH button to enter the path planning interface. At this time, the app receives the image taken by the UAV through the server, and three buttons (face indicated START, GOAL, and PATHPLAN) will appear for path planning, as shown in Figure 6. After pressing the Start or Goal button, the start or goal icon will appear and locate on the image. Users can use the vertical or horizontal slide bar to adjust the position of the start or goal icon on the image, as shown in Figure 7a,b. After that, the user clicks the start or goal icon, and the app will upload the coordinate of that icon to the database on the server. Once the starting point and goal point are available, the user can press the PATHPLAN button to detect the obstacles and execute the path planning in the image. The app in the mobile phone will change the PATH button to the NEW button when the path planning is performed in the server to indicate that users can browse the new result of path planning, as shown in Figure 8a. After pressing NEW, the app will load the path planning result and show the ADD button at once. Then the app also changes the NEW button back to the PATH button, as shown in Figure 8b. The user can thus operate the UGV to reach the location of the first object in the path, as shown in Figure 8c. After pressing ADD, the pinpoint (face indicated P) icon pops up, as shown in Figure 8d. The user uses the vertical or horizontal slide bar to move the pinpoint icon into the path close to the first object, and the app will indicate #1, as shown in Figure 8e. In Figure 8e, the app marks pinpoint #1 on the path when the user has pressed pinpoint #1. The user continues to operate the UGV to reach the second object location in the path, as shown in Figure 8f. Similarly, the user presses ADD to pop up a pinpoint icon and moves it into the path closed to the second object, as shown in Figure 8g. In Figure 8h, the app will indicate it as #2. By marking the pinpoints, the user can understand that the UGV has moved from the starting point to a particular position or goal point in the path while controlling the UGV to move.

3.3. Object Detection and Image Recognition

The following will discuss how to train a deep learning model applied to object detection and image recognition.

3.3.1. Training Environment

The user has established the model of the object detection and image recognition in a computing node that works for path planning and its hardware specifications, as listed in

Table 4. UGV modules.

Hardware	Specification	Amount
Server	HP Z4 G4	1
CPU	Intel Xeon 4108 @1.8 GHz	2
Memory	16G DDR4-2666 MHz	4
GPU	NVIDIA Quadro GP100 16 GB	1
Disk	SATA SSD 512 GB	1

. The path planning node regularly checks the database to determine whether there is a need to trigger the execution of object detection, image recognition, and path planning. With the operating system Windows 10, the path planning node has Anaconda installed to build the Python programming environment, the Pytorch training framework for the object detection model, and the image annotation tool for creating a user-defined data set. Here, the user uses Visual C++ to build Darknet to calculate the anchors’ parameters needed for user-defined models.

3.3.2. Training Data Set

A user-defined model collects the training data from free internet material and self-shooting images. Adjusting the same object through different shooting angles, distances, sizes, directions, etc., helps improve the model’s accuracy after training. The user collects more than 500 images in the data set and uses the Labeling tool to label all images with the recognized objects, as shown in Figure 9. Each labeled image will generate a document marked with the coordinates of the objects. Here, the user adopts cardboards only as identification objects to simplify the design of the experiments. In other words, the identification category is one object only.

3.3.3. Training Model

Training a user-defined model divides data into the training data set and test data set by 80% and 20% of data and generates two corresponding path files. After generating the path file, the system creates a YAML file and names the file for training the model. The content of the YAML file is the access location of path files of the training data set and the test data set, and the number of categories of the identified objects. The content of the names file is the category name. In the training data set, the clustering analysis provided by Darknet generates three sets of anchors with different scales enclosed in red squared boxes, as shown in Figure 10. To train a user-defined model, the user must modify the parameter configuration (.CFG) file. The parameters that need to be modified are width, height, filters, and anchors. The object detection and image recognition models use 608 × 608 pixels resolution and one recognition category. Therefore, the user sets the width and height of 608 pixels, assigns the number of filters 18, and fills in the anchors computed from Darknet.

Through 300 epochs in the training phase, the program stops the training and outputs the record of the training process, as shown in Figure 11. After completing the model training, the program uses the test data set to test the accuracy of the trained model, as shown in Figure 12. The test result shows that the mean average precision (mAP) is 0.931 indicated in a red squared box. The closer the mAP is to 1, the higher the recognition accuracy.

3.4. Object Detection and Path Planning

The below will describe the respective algorithms of object detection and path planning used in the application of an unmanned ground vehicle rapid traversing obstacle area.

3.4.1. Object Detection Using SCALE-YOLOv4

Based on YOLOv3, YOLOv4, [18] combines YOLOv3 backbone Darknet53 [19] with Cross Stage Partial Network (CSPNet) [20] and adopts the Mish activation function to improve it to CSPDarknet53. Moreover, it joins the Path Aggregation Network (PAN) [21], strengthening instance segmentation into the neck. YOLOv4 improves YOLOv3’s average precision (AP) and frames per second (FPS) by 10% and 12% [18]. Based on YOLOv4, Scale-YOLOv4 [22] proposes a neural network model scaling method to turn out a modifier one. Scale-YOLOv4 not only modifies the depth, width, and resolution of the input image in the neural network, but it can also modify the structure of the neural network. It can be scaled down to a small neural network or expanded to an extensive one to adapt to different equipment levels while maintaining the best performance and accuracy. Scale-YOLOv4 includes YOLOv4-Tiny, YOLOv4-CSP, and YOLOv4-Large.

YOLOv4-CSP is based on YOLOv4 and changed the first CSP stage of the backbone’s CSPDarknet53 to Darknet’s Residual Block to reduce the computation amount of the backbone’s floating-point operations (FLOPs). Meanwhile, its neck combined with the CSPNet, and Mish activation function reduces the amount of FLOPs computation in the neck. Spatial Pyramid Pooling (SPP) [23] also adjusts its position accordingly in the neck structure.

Wang, Bochkovskiy, and Liao [22] use the COCO minval dataset to compare the number of parameters, floating-point operations (FLOPs), frames per second (FPS), and average precision (AP) of different models. The backbone explores the comparison between Darknet53 (D53) and improved CSPDarknet53 (CD53s). They divided the neck into FPNSPP (FPN combined with SPP), CFPNSPP (FPNSPP combined with CSP), PANSPP (PAN combined with SPP), and CPANSPP (PANSPP combined with CSP). The activation function uses LeakyReLU and Mish alternatively. It selects the model CD53s-CPANSPP-Mish with the highest AP and denotes YOLOv4-CSP.

3.4.2. Path Searching Algorithm

A* (A Star) [24] is a widely used algorithm on plane graphics, with multiple nodes on the path and finding the lowest passing cost. This algorithm combines the advantages of the Dijkstra [25] and best-first search (BFS) [26] algorithms. While conducting a heuristic search to improve the algorithm’s efficiency, it can find an optimal path. The difference between these two algorithms is that Dijkstra needs to traverse the nodes to find the shortest path, while BFS is fast in execution but unable to find the shortest path. The algorithm switches the search mode dynamically between Dijkstra and BFS through the heuristic function in Equation (1), where $n$ represents any node in the plane graph, $g\left(n\right)$ stands for the actual distance from the initial node to any node n, and $h\left(n\right)$ is the estimated distance from any node $n$ to the target node.

$$f\left(n\right)=g\left(n\right)+h\left(n\right)$$

(1)

The A* algorithm divides the movement mode into four-direction or eight-direction [26]. There are two coordinate points $P_1\left(x_1,y_1\right)$ and $P_2\left(x_2,y_2\right)$ on the two-dimensional grid map. The four-way movement uses the Manhattan distance to calculate the distance between two points in Equation (2). Eight-direction movement uses diagonal distance to calculate the distance between two points in Equation (3).

$$D_{Manhattan}\left(P_1,P_2\right)=\left|x_1-x_2|+|y_1-y_2\right|$$

(2)

$$D_{Diagonal}\left(P_1,P_2\right)=1.414\times \min \left(\Delta x,\Delta y\right)+\left|\Delta x-\Delta y\right| where \Delta x=\left|x_1-x_2\right| and \Delta y=\left|y_1-y_2\right|$$

(3)

The modified A* algorithm becomes the Bidirectional A* (BA*) algorithm [27]. BA* can initiate a two-way search from the starting and goal points and end after the two paths cross. The general BA* primarily uses the balanced heuristic. Nevertheless, this study uses another modified version of the BA* called New Bidirectional A* (NBA*) [27], which is no longer limited to the balanced heuristic. Its search speed is faster than the general BA*.

3.4.3. Path Planning Flow

This study used the NBA* search algorithm as a path planning method. This method mainly operates on a flat grid graph, as shown in Figure 13. First, the user converts the input binarized PNG image into an occupied grid map (OGM). In the OGM, the value 1 represents the current node occupied, and 0 is free to go. Converting PNG images to OGM can meet the conditions required for the NBA* search algorithm in path planning. Before path planning, the program will read the coordinates of the starting and goal points from the cloud database. The user will adjust the coordinates of starting and goal points appropriately according to the different resolutions of the input image. In other words, if the algorithm scales the input image resolution, it will also scale the coordinates to avoid causing errors in the coordinates of the starting and goal points. Once the user has completed the OGM and the coordinates of the starting point and goal point, the user can execute the NBA* search algorithm for path planning. In path planning, the way to search using the NBA* algorithm is eight-way movement, which is more smooth and complete when compared with four-way movement. After obtaining the planned path, the user can use matplotlib to draw the path data into an image.

3.5. Executive Approach

The next section will present the executive flow of the proposed algorithm to help an unmanned ground vehicle rapidly traverse obstacle areas.

3.5.1. UAV/UGV Mobile Collaboration in Object Detection and Path Planning (UAGVMC-Odirpp) Algorithm

This study proposes an approach to path planning making from UAV in the air as per the ground UGV’s request. Once a UAV has uploaded the image to the server, the NBA* algorithm will execute the image’s path planning operation, as shown in Algorithm 1. The algorithm adjusts the resolution of the input image and then reduces the color saturation. Object detection and image recognition identify the obstacles and place the detection frame on these. After the image binarization, the image’s white represents the detection frame, and the rest are all black. The algorithm will change the aspect ratio of the original binary image from 4:3 to 1:1 to perform path planning operations. After path planning, the server obtains the map with a planned path. Then, the algorithm will adjust the image’s resolution appropriately to speed up the subsequent images processing and ensure the quality of the path planning image. At this stage, the algorithm adjusts the image’s aspect ratio from 1:1 to 4:3, making the background transparent. The last stage adjusts the transparent image back to the resolution of the original input image and then combines them to obtain the final result.

Algorithm 1: UAGVMC-odirpp

Input: img1 (obtain an image from the server). Output: img_final (put a planned-path image to the server).  1: begin  2:   img2 ← Down img1’s resolution according to phase 1 parameter  3:   img3 ← Reduce img2’s saturation  4:   img4 ← Input img3 to object detection model and circle each detected obstacle  5:   img5 ← Binarize img4  6:   img6 ← Square borders to img5 and get its shape ratio to be 1:1  7:   img7 ← Input img6 to path planning task and output an image  8:   img8 ← Up img7’s resolution according to phase 2 parameter  9:   img9 ← Crop img8 and get its shape ratio to be 4:3  10:    img10 ← Remove img9′s background  11:    img11 ← Restore img10 to be the same resolution as img1  12:    img_final ← Merge img1 and img11  13: end

3.5.2. Execution Flow

The UAGVMC-odirpp algorithm implements image processing, object detection, and path planning, as shown in Figure 14. The execution process frequently checks whether there is a newly submitted job in the cloud database. If so, the system downloads images from the server and gets this job started. The resolution adjustment in the image process will affect the execution speed of the job and the quality and reliability of the output results. The UAGVMC-odirpp algorithm is applied to optimize the execution performance, thus improving the output quality. Based on the UAGVMC-odirpp algorithm, Figure 15a is an input image, and Figure 15b is the first image resolution adjustment. The adjustment of the image resolution here is to speed up the execution in the subsequent image processing. Figure 15c reduces the image saturation to prevent the image’s color out of the subsequent binarization processing from causing errors in the pattern judgment. Figure 15d is the result of identifying obstacles through object detection. Figure 15e separates the identification frame and the background through the HSV color mask to obtain a binary image. Figure 15f adds the upper and lower borders for changing the image’s aspect ratio from 4:3 to 1:1 to meet the conditions from an input image to the path planning. Figure 15g is the path planning result drawn by the Plot function where the red dot represents the starting point, the green dot stands for the goal point, and the yellow line is the planned path. Whiting the background in the color white is used as a pre-processing for background transparency. Figure 15h adjusts the image resolution while outputting the path planning results. Figure 15i changes the aspect ratio from 1:1 to 4:3 after the image cropping. Figure 15j makes the image transparent within the white background. Figure 15k restores the resolution of the current image to be the same as the original input image. Figure 15l combines the image of a planned path with the original input image to obtain the final output result and then uploads it to the server to let users browse a planned path on the Android client. The object detection model displays the obstacle’s position in the binarized image map through the bounding box, as shown in Figure 15e,f. The portion of that image with the distance between any two obstacles that exceeds 1.3 times the width of the vehicle is used as the image background to plan the candidate paths. Therefore, the planned path is wide enough to let UGV pass. The path planning algorithm in this study mainly draws the shortest-straight-line path between the starting point and the goal point, as shown in Figure 15g,l, and the user then can remotely control UGV to move forward through the obstacle area according to the path planning result.

3.5.3. Predetermined Modes

According to the UAGVMC-odirpp algorithm, two stages, Phase 1 and Phase 2, in the execution flow, can adjust parameters. Combining two parameters will affect the overall performance in the execution flow and the quality of the final output image. The Phase 1 parameter affects the operations in the first five stages: reduce saturation, circle object, binarized, square border, and draw a path. In addition, it also affects the accuracy of object detection and the pros and cons of path planning. On the other hand, the Phase 2 parameter affects the operations in the last four stages: crop, remove background, restore resolution, and merge in the execution flow. Two parameters affect image processing performance, and they also affect the quality of the final output image. Therefore, this study set the predetermined nine modes that represent different parameter combinations, as listed in

Table 5. Initial Parameter combination.

Mode	Phase 1 Parameter	Phase 2 Parameter	Tool
1	1280 × 960	1280 × 960	OpenCV
2	1280 × 960	1280 × 960	Pillow
3	640 × 480	1280 × 960	Pillow
4	320 × 240	1280 × 960	Pillow
5	256 × 192	1280 × 960	Pillow
6	128 × 96	1280 × 960	Pillow
7	128 × 96	512 × 384	Pillow
8	128 × 96	256 × 192	Pillow
9	80 × 60	256×192	Pillow

. Then, the user tests these modes and selects which modes can work typically and do not affect the output image quality.

Figure 16 shows the object detection results according to nine modes listed in Table 5. From Mode 1 to Mode 8, all can exactly recognize all obstacles, while it does not work in Mode 9. Thus, users can only reduce the Phase 1 parameter to the image resolution of 128 × 96. Since Mode 9 cannot identify all of the obstacles, from Mode 1 to Mode 8, all can perform the path planning overall and evaluate the quality of the final output image appropriately. After the user has checked the final output image quality in Mode 6, Mode 7, and Mode 8 with the identical Phase 1 parameter, the final output image quality in Mode 6 and Mode 7 is clear, but it is blurred in Mode 8, as shown in Figure 17. Therefore, the user can only reduce the Phase 2 parameter to the image resolution of 512 × 384. According to the test results mentioned above, this study chose Mode 1 to Mode 7 for the following experiments and performance evaluation.

4. Experiment Results and Discussion

4.1. Executive Approach

This study explores the practical application of mobile collaborative operations between unmanned vehicles, which optimizes object detection, image recognition, and path planning. It looks forward to achieving the best execution performance of the cooperative operation. In

Table 6. Parameter combination.

Mode	Phase 1 Parameter	Phase 2 Parameter	Tool	Time Complexity
1	1280 × 960	1280 × 960	OpenCV	O(bd)
2	1280 × 960	1280 × 960	Pillow	O(bd)
3	640 × 480	1280 × 960	Pillow	O(bd)
4	320 × 240	1280 × 960	Pillow	O(bd)
5	256 × 192	1280 × 960	Pillow	O(bd)
6	128 × 96	1280 × 960	Pillow	O(bd)
7	128 × 96	512 × 384	Pillow	O(bd)

, the user has set the different parameter combinations of Phase 1 and Phase 2 as seven testing modes in the experimenters. Their respective time complexity of computing is also shown in Table 6, where b represents the branching routes and d stands for the shortest distance from entry to exit. The resolution of all input images in the experiments is 1280 × 960 pixels. The Phase 1 parameter represents the adjusted resolution of the input image, and Phase 2 the output resolution of a drawn path image. Mode 1 uses OpenCV to perform image processing, and instead, the others from Mode 2 to Mode 7 use Pillow. In particular, from Mode 1 to Mode 7, all use OpenCV for fast image binarization. This study has presented nine patterns in the experiments and pre-assigned a pair of starting and goal points to each pattern as a corresponding experiment. Each experiment will carry out the drawn paths according to 16 scene layouts. In scene layouts, carton boxes represent obstacles. Each scene executes the planned route based on seven modes and records the execution time of each mode. Each experiment records its time to enter the start point and reach the goal.

Each experiment draws its path planning results of 16 scene layouts and plots the execution time of each mode, based on Equation (4), in each scene layout in a figure. In Equation (4), $t_{k_j{}_i}$ represents the execution time of each mode in an experiment with a designated scene, where the subscript $i$ is the number of scenes, $j$ the number of modes, and $k$ the number of experiments. As a result, ${\overline{t}}_{k_j}$ stands for the average execution time of each mode in the experiments in Equation (4). In Equation (5), $P_{r_{k_j}}$ represents the performance ratio of each operative mode to Mode 1 in the experiments. Finally, we obtain $P_{r_j}$ that stands for the average performance ratio of each Mode to Mode 1 in the experiments, as shown in Equation (6).

$${\overline{t}}_{k_j}=\frac{1}{l} {{\displaystyle \sum }}_{i=1}^l{t}_{k_j{}_i}, j=1,2,\dots ,m, k=1,,2,\dots ,n$$

(4)

$$P_{r_{k_j}}=\frac{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{${\overline{t}}_{k_1}$}\right.}{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{${\overline{t}}_{k_j}$}\right.}, j=1,2,\dots ,m, k=1,,2,\dots ,n$$

(5)

$$P_{r_j}=\frac{1}{n}{{\displaystyle \sum }}_{k=1}^n{P}_{r_{k_j}}, j=1,2,\dots ,m$$

(6)

This study uses a scenario and seven modes to compare the execution time of the three path search algorithms, as shown in

Table 7. Performance comparison of path searching algorithms (second).

Mode	A*	BA*	NBA*
1	21.2	13.4	11.5
2	21.2	13.4	11.5
3	6.8	3.5	2.9
4	1.54	0.8	0.6
5	1.13	0.5	0.4
6	0.2	0.11	0.09
7	0.2	0.11	0.09
Average	7.47	4.55	3.87

. In Table 7, the execution time range is from the beginning of the path searching to the end. As a result, we chose the best-resulting NBA* algorithm to apply path planning in the following experiments.

4.2. Data Set

The user has designed the nine sets of patterns indicating the starting point (denoted red dots) and goal point (denoted green dots) for nine experiments, as shown in Figure 18. Each pattern set will run in conjunction with 16 sets of scene layouts. In Figure 19, the yellow squared boxes in the figures represent the cartons (obstacles) they located.

4.3. Experiment Results

First, the user sets the designated coordinates of each experiment’s starting point and goal point of the scene layout as listed in

Table 8. The designated coordinates of starting and goal points for the experiments.

Experiment Number	Starting Point Coordinate	Goal Point Coordinate
I	(80, 760)	(1200, 760)
II	(80, 760)	(1200, 480)
III	(80, 760)	(1200, 200)
IV	(80, 480)	(1200, 760)
V	(80, 480)	(1200, 480)
VI	(80, 480)	(1200, 200)
VII	(80, 200)	(1200, 760)
VIII	(80, 200)	(1200, 480)
IX	(80, 200)	(1200, 200)

. Each starting point and goal point pair named the experiment number from I to IX. Next, the user starts the experiments to implement the path planning for each scene layout. Finally, the experiments resulted in the planned paths, as shown in Figure 20, Figure 21, Figure 22, Figure 23, Figure 24, Figure 25, Figure 26, Figure 27 and Figure 28. The only optimized path from the designated starting point to the goal point can display on each scene layout in every experiment.

For estimating the performance evaluation of path planning in various operative modes, the experiments recorded the execution time of each operative mode for running in all scene layouts, as shown in Figure 29. After that, the user has the average execution time of each operative mode running in all scene layouts in every experiment, as listed in

Table 9. Comparison of the performance ratio of each mode in the experiments.

Experiment	Mode	Average Execution Time (s)	Performance Ratio
I	1	28.55	1.00
	2	20.74	1.38
	3	12.67	2.25
	4	10.76	2.65
	5	10.52	2.71
	6	10.17	2.81
	7	9.32	3.06
II	1	37.00	1.00
	2	29.72	1.25
	3	14.77	2.51
	4	11.29	3.28
	5	10.79	3.43
	6	10.25	3.61
	7	9.55	3.88
III	1	43.73	1.00
	2	36.04	1.21
	3	16.24	2.69
	4	11.73	3.73
	5	11.07	3.95
	6	10.36	4.22
	7	9.53	4.59
IV	1	34.47	1.00
	2	26.97	1.28
	3	14.08	2.34
	4	11.10	3.11
	5	10.69	3.22
	6	10.23	3.37
	7	9.45	3.65
V	1	40.36	1.00
	2	32.50	1.24
	3	15.52	2.60
	4	11.52	3.50
	5	10.96	3.68
	6	10.35	3.90
	7	9.53	4.24
VI	1	35.34	1.00
	2	27.65	1.28
	3	14.17	2.49
	4	11.04	3.20
	5	10.72	3.30
	6	10.26	3.44
	7	9.43	3.75
VII	1	43.73	1.00
	2	36.34	1.20
	3	16.18	2.70
	4	11.57	3.78
	5	10.99	3.98
	6	10.34	4.23
	7	9.52	4.59
VIII	1	37.51	1.00
	2	29.60	1.27
	3	14.74	2.54
	4	11.22	3.34
	5	10.81	3.47
	6	10.33	3.63
	7	9.47	3.96
IX	1	29.07	1.00
	2	21.26	1.37
	3	12.89	2.25
	4	10.81	2.69
	5	10.52	2.76
	6	10.16	2.86
	7	9.40	3.09

. As a result, based on the performance ratio, the performance comparison of path planning of seven modes in nine experiments is indicated in Table 9.

4.4. Performance Evaluation

Mode 2 uses Pillow instead of OpenCV to perform image processing, and its average performance ratio (i.e., the execution speed) is 1.28 times compared with Mode 1. Therefore, the user sets Mode 2~Mode 7 to use Pillow to perform the image processing in the UAGVMC-odirpp algorithm. According to the nine sets of starting and goal points in the experiments from session 4.3.1 to session 4.3.9, the user can obtain the average performance ratio of each operative mode compared with Mode 1, as shown in

Table 10. Performance evaluation.

Mode	Average Performance Ratio
1	1.00
2	1.28
3	2.49
4	3.25
5	3.39
6	3.56
7	3.87

. Mode 3 reduces the Phase 1 parameter to 640 × 480 pixels, and its average performance ratio is 2.49 times that compared with Mode 1. Likewise, Phase1 parameters of Mode 4, Mode 5, Mode 6, and Mode 7 are 320 × 240, 256 × 192, 128 × 96, and 128 × 96 pixels, respectively. Accordingly, their average performance ratio is 3.25, 3.39, 3.56, and 3.87 times compared with Mode 1.

4.5. Discussion

Compared with the related literature, the proposed approach can integrate a variety of vehicles more efficiently. The proposed approach does not construct a map in advance but only uses live images to identify obstacles at the local scene and plan the current walking path immediately. The method has adopted the Scaled-YOLOv4 object detection model to detect obstacles. According to the experimental results, the object detection model has achieved better performance and is more suitable to apply to these cases. The NBA* path planning algorithm is an improvement of the two-way version BA* algorithm derived from the classic and reliable A* algorithm. Its characteristics are also suitable for the multi-obstacle layout requirements of this example.

Regarding limitations, because obstacle detection uses bounding boxes to mark detected obstacles, space utilization is poor compared with edge detection or instance segmentation algorithms. Path planning is also affected by the size of bounding boxes. It is not so intuitive and smooth. Nevertheless, it can achieve a more rapid performance than the others due to the point of view of time. In terms of the path planning algorithm, this study does not consider the volume and wheelbase of the ground vehicle, so it may happen that the ground vehicle with a large volume and wheelbase cannot walk successfully according to the planned path.

Moreover, the planned path may encounter particular terrain and features, for instance, the bridges (the UGV could go beneath) or a path with holes and uniformities. How to deal with these problems? Finally, it is possible to consider changing the user’s remote control of the ground vehicle to self-driving so that the ground vehicle can travel more smoothly.

5. Conclusions

Generally speaking, a single UGV with camera equipment would spend a long time finding a way out of an unfamiliar area with obstacles. Therefore, this study proposes a fast drone-aided image-sensing path planning approach to supply the UGV with an image map of the well-planned path for traversing that unfamiliar area rapidly. The proposed approach, the UAGVMC-odirpp algorithm, has employed one-stage rapid object detection, and an image recognition method to detect obstacles in a live image and quickly makes a planned path based on the object detection results. As a result, according to the well-planned path map, the user can then remotely control UGV to move forward through the obstacle area. We have to devote ourselves to object detection and path planning improvements in future work. In terms of object detection, how to devise a new approach based on an instance segmentation algorithm with the same performance as Scaled-YOLOv4 to increase the space utilization of path planning. For path planning, we should consider that parameters such as the volume and wheelbase of the ground vehicle should register to the in-cloud database. Thus, we can plan a smoother path according to these parameters. Hopefully, object detection and path planning improvements can conquer the limitations mentioned in the discussion section. Finally, another future objective is to develop autonomous ground vehicles instead of remote-controlled UGVs.