1. Introduction
A teleoperation system is a technique that enables an operator to control the system from a remote location. Teleoperation is obtained from the Greek words Tele and Operation, which mean distance and task completion, respectively. The length could be physical, such as a human operator controlling a robot in a remote location, or a change in scale, such as a surgeon using teleoperation to perform surgery at the micro-scale level [
1]. Nowadays, teleoperation technology is applied for various applications, including entertainment systems, industrial machinery, remotely operated vehicles (ROVs), remote surgery, unmanned aerial vehicles (UAV), etc. [
2]. Mobile robots have been developed for search and rescue [
3]. Mobile robots have difficulty running in an unknown environment, such as a disaster area. Moreover, mobile rescue robots cannot efficiently perform because they are expensive due to the initial cost of equipment, peripherals, installation cost, programming, and training [
4]. This robot needs regular maintenance and a power source to drive the actuators. It cannot respond to emergencies and has limited sensors, vision systems, and real-time responses [
5]. For these reasons, researchers are interested in biobots for search and rescue operations because this type of robot overcomes the limitation of mobile rescue robots [
6]. Biobots reduce the equipment, installation, and training cost. The capabilities of sensors, vision systems, and real-time responses are higher than those of traditional mobile rescue robots due to their instinct. For the cyborg hybrid robot, one advantage is the power supply. It can be recharged or replaced with electrical power devices for a long time [
7].
However, controlling biobots is a difficult task. Li et al. developed a brain–computer interface (BCI) to control the cockroach biobot [
8]. They proposed a functional data transfer pathway from the human brain to the cockroach brain. Nguyen et al. developed sideways walking control of a cyborg beetle [
9]. They developed a cyborg beetle control system that users can commend through infrared remote control. Cao and Sato developed remote radio-controlled insect–computer hybrid robots for search and rescue operations [
10]. These cyborg controls are based on a wireless communication system for a limited distance range.
Teleoperation technology has been widely used to control various kinds of robotic applications. During pandemics, researchers have understood the necessity of a teleoperation system [
11]. Tabaza et al. developed a robotics-assisted surgery robot using teleoperation technology [
12]. Wang et al. developed a humanoid robot’s intuitive and versatile full-body teleoperation [
13]. Diolaiti and Melchiorri researched the teleoperation of a mobile robot over haptic feedback [
14]. Yuji et al. developed a teleoperation mobile robot that utilized the robot sensor network [
15]. Ruan et al. created a motion control self-balance robot using vision-teleoperation [
16]. By using a head-mounted display, Martine and Ventura built an immersive 3D teleoperation of a search and rescue robot [
17]. Okabe et al. developed the teleoperation system for rescue robots by recording the target position of the end-effector [
18]. Hong et al. proposed practical hardware design tactics and control methods for a rescue robot to save patients in disaster-stricken environments. The team created a powerful dual arm mechanism and a hybrid tracked-legged mobile platform. The motion was synthesized using dynamics-based optimization and a controlled hierarchical control scheme [
19]. Teleoperation is one of the advanced technologies which is very popular for various applications in the modern world.
User interface (UI) is needed to be incorporated into the teleoperation system for efficient remote telecontrol of mobile robots [
1,
20,
21]. The UI manages teleoperation processed, and displays measured data, a live-camera view to the operator. Communication channels such as the internet, radio frequency, Bluetooth, and infrared are commonly used for teleoperation. In previous research studies, Bluetooth and infrared are limited to the communication link range between the operator and remote robot [
1,
22]. Qian K et al. developed a small teleoperated robot with a control distance reaching 5000 m in an open area using radio wave communication [
23]. Bluetooth, infrared, and radio wave are relatively low-cost. Internet communication allows a remote robot can be accessed and controlled from any part of the world [
1,
24]. Due to their size, various sensors, actuators, and cameras can be attached to these teleoperated robots
Previous biobot research studies are based on local communication and limited communication distance between computers and biobots [
9,
25,
26,
27]. Wireless technologies such as infrared [
9,
28,
29], ZigBee [
30,
31,
32], Bluetooth [
8,
33,
34,
35,
36,
37], WiFi [
38,
39], and radio frequency (RF) [
25,
27,
40,
41,
42,
43] were used to control the motion of biobots and wireless data transmission. These wireless devices cannot reach long-distance communication. The biobots cannot be controlled from a remote distance, such as city to city or country to country. Therefore, controlling the biobot using teleoperation technology from one country to another is quite a new and hot research topic. It will be useful during a pandemic when most people cannot travel from one country to another due to a travel ban. We propose a teleoperation system to monitor and control biobots from Bangladesh to Japan. This research will assist in the emergency or pandemic situation when biobot operators cannot physically come to the onsite area where the biobots are placed at a remote distance. These proposed methods will enable the operator to monitor and control the biobot from a remote location.
In this study, we proposed a biobot system that can be monitored and controlled from a remote distance through a developed teleoperation UI (remote and local UI). We employed two communication channels for the teleoperation UI. Virtual network computing (VNC) was selected and implemented for teleoperation communication between Japan and Bangladesh. The VNC was applied to provide live-stream video and send remote stimulation commands through remote UI. A radio frequency (RF) device was implemented for local communication between a computer and the biobot in Japan. The RF device sent remote commands from the computer to the backpack and received inertial measuring unit (IMU) data from an electronic backpack to the computer. The first camera was used as an online motion-tracking system on the local UI, while the second one was utilized for the remote live streaming video on the remote UI. The cockroach states were displayed on the local UI. A multithreading algorithm was applied to run multi-rate computational processes concurrently on local and remote UI. A remote operator in Bangladesh was tasked to steer the cockroach in Japan using the developed teleoperation UI.
3. Teleoperated Experiment
This study aimed to develop a biobot control method using the proposed teleoperation method between two remote countries. As shown in
Figure 8, the experimental testbed was located in Morishima lab, Osaka University, Japan, while the operator is located in Chittagong, Bangladesh. Raspberry Pi 4 was connected to the wired LAN router for a faster internet connection. A semicircle path with an average width of 16.5 cm was drawn for the path following the mission, as shown in
Figure 9a. For the finish area, a wireless cage with a dimension of 29 cm in length, 19 cm in width, and 7 cm in height was built using a micro servo motor augmented with the wireless receiver, as revealed in
Figure 9b. The cage gate can be opened or closed using the teleoperated command.
Five biobots were used in this experiment for two schemes of teleoperated experiments: path with obstacle and path without obstacle. An obstacle was placed in the middle of the track with a diameter of 10 cm and a height of 5.5 cm. The operator was tasked to direct the cockroach remotely from Bangladesh to follow the path and avoid the obstacle to reach the finish area. Before the cockroach reached the finish area, the operator opened the gate and then directed the cockroach to enter the cage. After the cockroach entered the cage, it was closed by teleoperation command.
An operator from a remote place can send commands to the biobot and the cage by pressing predetermined keys on the keyboard. The keyboard keys for controlling the biobot from a remote laptop are summarized in
Table 1. Firstly, the biobot was placed at the testing area start point, as shown in
Figure 9b. The operator monitored biobot movement on the web camera displayed on the remote UI, which was set in the experimental testbed in Japan. The operator controlled the biobot to follow the semicircle path and placed the cockroach in the wireless cage (finish area). After the biobot was in the wireless cage, the operator closed the gate by the teleoperation command. The experiment succeeded if the operator successfully operated the biobot to follow the semicircle path and placed the cockroach in the wireless cage/finish area. After that, IMU, commands, and computed position data were collected to analyze further.
Table 2 shows the sampling rate of local UI’s to run parallel computation simultaneously using a multithreading algorithm. Finally, the experiment was performed with and without obstacles. This study needs to investigate whether the operator can overcome the obstacle using the teleoperated command from Bangladesh.
The teleoperation UI applied two wireless communication channels to enable a teleoperation system between Japan and Bangladesh. Internet communication was applied for remote communication between Japan and Bangladesh. The RF device was implemented for local wireless communication between a desktop computer and the biobot in the local area (Japan). The transmission control protocol (TCP) and user datagram protocol (UDP) are widely used in internet communication protocols for the teleoperation of mobile robots [
1] and industrial robots [
44]. Due to security reasons, these methods could not be implemented in the experiments. Therefore, VNC and the developed teleoperation UI were applied to enable the teleoperation system between Bangladesh and Japan. VNC was selected because it uses the remote frame buffer (RFB) protocol. This protocol is suitable for the proposed teleoperation UI operation under the Python program. Remote UI with VNC provided a live-stream video from a cockroach experiment in Japan and sent remote commands from Bangladesh. Local UI received the IMU data from the biobot and remote commands from Raspberry Pi. The communication configuration implemented in the teleoperated system is presented in
Figure 10.
4. Result and Discussion
4.1. Path following Tasks
In this section, biobots were steered remotely from Bangladesh to follow a predetermined path. Two schemes of teleoperated path following missions were tested: path following without and with the obstacle. Ten cockroaches were implanted with the electrode on both antennae and cerci in the initial preparation. After electrode implantation, the cockroaches were put back in the container for 24 h to recover and rest. The antennae with implanted electrodes were put back and glued to the cockroach pronotum to prevent the cockroach from breaking the electrode. Unfortunately, during recovering and resting time for 24 h, two cockroaches broke one of the antennae. Therefore, we could not use these two cockroaches for the experiments, although the implanted antennae were glued on the cockroach pronotum. Eight implanted cockroaches were tested using a developed electronic backpack with electrical stimulation. Only one cockroach could not respond to the given stimulus to one of the antennae of eight cockroaches. However, all cockroaches were successfully stimulated on the left cercus, right cercus, or cerci. From the ten implanted cockroaches, seven cockroaches were ready to use in the initial experiments.
The teleoperated experiments were divided into two sections, i.e., with an obstacle and without an obstacle placed in the middle of the predetermined path. We did not determine the sequence of the two sections/schemes of the teleoperation. Each section/scheme was repeated five times for each cockroach. Between the two sections/schemes, the cockroaches were put back on the container to rest after performing five trials on one of the experimental teleoperated schemes for about 15 min. Based on the teleoperated experiments, three cockroaches (first, second, and fourth) were successfully directed to follow the predetermined trajectory for a path with and without an obstacle. Two cockroaches broke their antenna during the initial teleoperated trial on a path with an obstacle. Therefore, we could not continue to use these two cockroaches for the experiments because one of the antennae from these two cockroaches was broken and could not be fixed. The third cockroach was controlled remotely from Bangladesh on a path with an obstacle, but the cockroach was able to break the antenna during the experiment on a path without an obstacle. The fifth cockroach could be controlled remotely on a path without an obstacle. However, the cockroach broke one of the antennae during the initial experiment on a path with an obstacle. Hence, we could not resume the teleoperation task using the fifth cockroach with an obstacle and the third cockroach without an obstacle.
The local UI wirelessly received the estimated Euler angles and stimulation commands from the electronic backpack attached to the cockroach. It was placed and glued on the first segment of the cockroach thorax. The IMU measures the cockroach thorax Euler angles and sends them to the desktop PC through a wireless transceiver. The IMU outputs zero value for the yaw angle measurement when the cockroach faces the north. All obtained data during one successful mission on a path with an obstacle are presented in
Figure 11 and
Figure 12.
Figure 11 reveals the attitude angle responses concerning the stimulation inputs. Stimulation on the right antenna caused the cockroach to turn left, while stimulation on the left antennae caused the cockroach to turn left. Sometimes the cockroach did not move forward after stimulating the left or right antennae. Stimulation on the left or right cercus or both cerci triggered the cockroach to move forward with a slight turn to the right or left. By implementing this stimulation strategy, a remote operator from Bangladesh can direct the cockroach to follow a predetermined trajectory and avoid an obstacle in front of the cockroach. The position of the cockroach in pixels acquired from the image processing technique is shown in
Figure 12.
The processed image of the cockroach trajectory was displayed on the local UI in real-time. The image was processed at a sampling rate of 25 fps. The displayed image of the cockroach trajectory commanded from Bangladesh is depicted in
Figure 13. The cockroach is the fourth cockroach in the fifth trial. The photos show that a remote operator successfully directed the cockroach to follow a predetermined trajectory from starting point to the finish area. The cockroach could be controlled to avoid the obstacle in front of it. The overall teleoperated operation for the biobots is presented and discussed in
Section 4.1.1 and
Section 4.1.2.
We implemented five cockroaches to investigate the repeatability of our proposed teleoperated system. In this experiment, cockroaches were controlled remotely from Bangladesh for two teleoperation schemes. The first scheme was to follow the path with an obstacle and the second one was to direct the cockroach to follow the path and avoid an obstacle until the cockroaches entered the wireless cage. A remote operator opened the wireless cage when the cockroach was in the middle of the path and then closed the cage when the cockroaches entered the cage in the finish area. The teleoperated command was be considered successful if the cockroaches could be directed to follow the path and enter the cage.
4.1.1. Path without Obstacle
A remote operator was tasked to direct four cockroaches (first, second, fourth, and fifth) to follow a predetermined path without an obstacle. The operator monitored the cockroach’s current position on the remote UI. The remote stimulation and open/close the cage commands could be sent by pressing the predetermined keys on a remote laptop, as summarized in
Table 1. The teleoperated trial was conducted five times for each cockroach. The teleoperated results for each cockroach are presented in
Figure 14.
In the teleoperated experiment without an obstacle, first, second, and fifth cockroaches could be easily directed by a remote operator to follow a predetermined path. These cockroaches were successfully steered without touching both line paths on the right or left side, except for the third trial on the fifth cockroach, which slightly walked outside the left path. Two trials on the second cockroach (second and fifth trials) walked outside the left path. The cockroach ignored the stimulation command on the left antenna to turn right. After that, the cockroach was directed to follow the path and enter the cage. For all trials on four cockroaches, a remote operator opened the wireless cage when the cockroach reached the middle of the path and then closed the cage after the cockroach reached the finish area in the cage.
The second cockroach ignored a few stimulation commands on the right or left antenna. However, it still responded to the stimulation commands on the cerci. After carefully investigating the four cockroaches, we strongly believe that electrode implantation degradation caused the second cockroach to ignore some stimulation commands on the antennae. The cockroach utilized for the teleoperated mission was surgically implanted two weeks before. In two weeks, the cockroach tried to release or break the implanted electrode on the antenna, although certain antennae parts were glued on the pronotum. The second cockroach could be controlled to follow the predetermined trajectory, although few stimulation commands were ignored due to electrode implantation degradation. This degradation was not found on the first, fourth, and fifth cockroaches. We performed teleoperation experiments with these cockroaches one to two days after implanting electrodes. The cockroach almost responded to every stimulation to the left or right antenna by turning left or right. For the electrode implanted on the cerci, the cockroach has difficulty reaching the implanted electrode by using its legs.
Some results show that a remote operator effectively remotely controlled the cockroaches to follow the predetermined path without providing stimulation on the cerci. This motion could happen because the operator applied cockroach free walking motion to direct the cockroach on the path. However, for other trials, cockroaches tended not to move forward with antennae stimulation. Therefore, the operator stimulated the cerci to trigger the cockroach to move forward. After the cockroach moved forward with a slight turn, the operator directed the cockroach to turn left or right by providing stimulation to the antennae. Implementing this stimulation strategy allows the cockroaches to be steered remotely to follow the path and enter the cage.
4.1.2. Path with Obstacle
A cylindrical obstacle with a diameter of 10 cm and a height of 5.5 cm was put in the middle of the path. A remote operator was asked to steer the cockroach remotely from the starting position to the finish area in the wireless cage. The cockroach must avoid and pass the obstacle. After passing the obstacle, the operator opened the cage remotely and closed it after the cockroach had entered the finish area in the cage. This path with an obstacle is a relatively more complicated task than the previous one without an obstacle. Four cockroaches were utilized and repeated five times for each cockroach. The results for all the cockroaches are presented in
Figure 15.
It revealed that the cockroaches could be steered to follow the path, avoid the obstacle, and enter the cage, although in some trials, the cockroach passed to the left or right line of the outside path. The first cockroach at the second trial could be directed to avoid the obstacle. However, after passing the obstacle, the cockroach ignored the stimulation on the right antenna to turn right. After passing the outside line of the right path, the cockroach did not ignore the stimulation on the right antenna to turn left and enter the cage. Due to the electrode implantation degradation on the antennae for the second cockroach, the cockroach could pass the outside left line path at the initial teleoperated command. The cockroach did not respond to the right stimulation commands, but the cockroach could be commanded to enter the path and avoid the obstacle. Then the cockroach could be directed to enter the cage. Overall, the cockroaches could be steered remotely to follow the path, avoid the obstacle, and enter the finish area in the cage.
We found that when the cockroach reached near the obstacle, the cockroach tended to circle the obstacle or did not move. In order to prevent this free motion, the operator stimulated the left or right cercus to trigger the cockroaches to move forward with a slight turn to the left or right. After the cockroach was triggered to a forward walking motion, the cockroach was steered to follow the path and enter the finish area in the cage. Based on the teleoperated experiment, the cockroaches did not respond to the stimulation signals on a few occasions, especially on the antennae. Previous research studies also reported that the cockroach did not always respond to the stimulation on the right or left antenna [
27,
38,
45]. An action camera was put on the experimental testbed to record the cockroach’s response. The cockroach was steered to follow a path with an obstacle, as depicted in
Figure 16. The figure demonstrates that the cockroach was successfully directed to follow the predetermined path, avoid the obstacle, and enter the cage. By using multithreading, we could run multiple computational processes simultaneously on the developed user interface as performed by previous research. Multithreading can run the embedded machine learning system in real-time [
46]. The videos of the teleoperated locomotion for path following without (
Video S1) and with obstacle (
Video S2) can be seen in
Supplementary Materials.
The time required to remotely steered the biobots from the starting position to the final position was measured for path following with and without obstacles. The average time required for each biobot is presented in
Figure 17. Based on the results, the time required to steer the biobots is higher than that of biobots without obstacles (
p-value = 0.08). This longer time happens because biobots tend to cycle the obstacle. The remote operator needs to control the biobots to avoid obstacles and follow the path until reaching the final position.
4.2. Accuracy for the Input Command
In the previous experiment for path following tasks, almost all cockroaches responded to stimulation given to the left and right antennae. The input commands were sent from Bangladesh, while the cockroach responses were recorded in Japan. On a few occasions, the second cockroach did not respond to the given stimuli on the antennae due to electrode implantation degradation, but all cockroaches responded to the stimuli provided to the cerci. Therefore, the second cockroach was excluded from the repeatability test command. The previous teleoperation commands were utilized to calculate the repeatability command. We recorded and collected command accuracy tests from the previous experiment session. The total number for this test is 120 samples. The test samples were collected from the recorded video of the experiment session for turning right, turning left, and moving forward commands. The stimulation commands were given randomly to the right antenna, left antenna, or cerci. Each command test result has 40 samples, as summarized in
Table 3. Equation (3) is applied to calculate the command accuracy ratio.
The average accuracy for turning left, right, and moving forward commanded remotely from Bangladesh is 87%, 85%, and 97%, respectively. The biobots accepted all commands transmitted from Bangladesh to Japan. Delays occurred during data transmission from input commands in Bangladesh to the biobots in Japan. The delay was found in remote transmission using internet communication from Bangladesh to Japan and local transmission using RF devices from a computer to the insect. The computed delay in internet and RF communications is presented in
Section 4.3. There is also an acceptable low delay in the live-video stream between Bangladesh and Japan. A remote operator in Bangladesh still successfully steered and monitored the insects to follow a predetermined path with and without obstacles.
4.3. End-To-End Time Delay
A proposed teleoperation system using internet communication can be accessed from Bangladesh for a long-range operation. However, internet communication is also susceptible to delay, data loss, jitter, and communication blackout, as presented by a previous study [
47]. A remote operator controlled the biobots from Bangladesh to Japan in pure teleoperation mode. Time delay is one of the most significant factors affecting the performance of remote operations and manipulation in teleoperation systems [
48,
49]. Delays that occurred on internet and RF communications are presented in this section. The desired commands started and ended in Bangladesh. There are four kinds of delays for the end-to-end time delay: transmission delay, propagation delay, queuing delay, and processing delay. Transmission delay means the time needed to put the information/data bits to the transmission link. Mathematically, transmission delay is directly proportional to the length/size of data packets. Propagation delay is a time delay in transferring the information to the destination when the information is in the transmission link. In this study, queuing and processing delays were not measured for the end-to-end time delay for the teleoperation system. The measured propagation delay of Bangladesh to Japan and Japan to Bangladesh is shown in
Figure 18. A command prompt was used to ping Japan to Bangladesh and Bangladesh to Japan with unique internet protocol (IP) addresses.
Transmission delay also occurred in the experimental field (local communication) in Japan during wireless transmission commands from the Raspberry Pi transmitter to the electronic backpack. The transmitter sent stimulation commands to the backpack with a sampling rate of 10 Hz. In order to measure this transmission delay, a step input signal, as shown in
Figure 17, was implemented as the transmitted signal. The signal received by the electronic backpack was measured using a serial monitor on Arduino IDE. Based on the obtained transmitted and received signals, the delay from the Raspberry Pi transmitter to the electronic backpack was 0.19 s, as depicted in
Figure 19.
The obtained average end-to-end time delay to control the biobot from Bangladesh to Japan is summarized in
Table 4. The table shows that transmission and propagation delay Bangladesh to Japan and Japan to Bangladesh. The calculated average end-to-end time delay of this system was 275 ms. However, the time delay will be different in different circumstances depending on the internet connection speed. The obtained low time delay is still acceptable because the motion of the biobots is relatively slow. Therefore, a remote operator in Bangladesh could steer the cockroach on the path-following mission, as presented in
Section 4.1.1 and
Section 4.1.2.
4.4. Battery for Electronic Backpack
A Lithium Polymer (LiPO) battery with a capacity of 50 mAh was applied to power the electronic backpack. A digital multimeter was used to measure the battery’s voltage drop. In this test, the Raspberry Pi transmitter sent stimulation commands (50 Hz, 50% duty cycle of 3.3 V PWM signals) to the backpack for 30 s and then sent no stimulation signals for 30 s. The sequence of stimulation signals was repeated for more than 70 min. The backpack was powered with the LiPO battery and sent stimulation commands and IMU data to the wireless transceiver connected to the desktop PC. The developed local UI was utilized to observe the received stimulation commands and IMU data on the desktop PC. The backpack was not attached to the cockroach because too long, and many stimulation commands could damage the cockroach. The voltage drop of the battery was measured every two seconds. Three 50mAh LiPO batteries were used in this test. The results of the measured three batteries are revealed in
Figure 20. The fully charged battery was 4.15 V. From the tested three batteries, the batteries could not supply the backpack if the voltage was below 2.50 V. Based on the test result, the batteries can provide power to the backpack from 52 min to 62 min.