Robot Scheduling for Assistance and Guidance in Hospitals

Abstract

At present, the global COVID-19 epidemic has not slowed down. To reduce the contact between people during the epidemic and prevent the epidemic from expanding, we have developed a robot to assist medical staff in patient guidance and communication services. The robot can provide an emergency contact so that users can immediately contact the counter for help. The user does not have face-face contact with the medical staff. When the robot encounters obstacles in the path of travel, the detected event and the time of occurrence are sent back to the back-end system. It also provides security personnel with real-time images and robot control rights to understand the situation and deal with it in real-time.

1. Introduction

With increasing global labor costs and the problem of global population aging [1], the world will face population and human resources shortages [2] in the future. Therefore, the replacement of human resources by robots will be a global trend in the future. According to International Federation of Robotics (IFR) research [3], as shown in Figure 1, in 2016, the sales volume of service robots was much lower than the sales volume of industrial robots. Still, as the growth rate of industrial robots gradually became saturated, the demand for service robots also relatively increased.

In addition, because the new coronavirus epidemic is continuing around the world [4], countries worldwide are actively implementing epidemic prevention measures. During the epidemic prevention period, it is necessary to avoid human-human contact as much as possible, and service robots can assist people in completing work tasks while avoiding human-human contact. In addition, robots can alleviate the problem of insufficient medical resources that are caused by the epidemic and significantly reduce human resources with the assistance of robots.

2. Related Work

In the medical industry, the use of robots is increasing year by year [5]. Mobile robots [6] specialize in assisting nurses in transporting medical supplies and moving patients to other areas. Chat-type robots help patients measure body temperature and detect whether they wear masks.

At the University of California, San Francisco Medical Center [7], they introduced 25 intelligent transport robots named TUGs, as shown in Figure 2a. Assist medical personnel in transporting medicines, goods, laboratory specimens, and food, automatically navigate through the hospital floor plan that is stored in the robot, avoid obstacles at any time, and use wireless network communication to control the opening and closing of elevator doors.

In India, a Mitra robot [8] is responsible for allowing patients that are diagnosed with new coronary pneumonia that are in isolation to keep in touch with their relatives and friends, as shown in Figure 2b. The face recognition technology that is built into the robot’s eyes can memorize human faces. The robot can actively greet the patient when it sees them. It allows the patient to make video calls with relatives and friends through the touch screen that is built into the robot’s chest. In addition, the Mitra robot can assist nurses in disinfection work, answer patient questions, allow doctors and patients to conduct remote consultations, and remind patients to take medication.

3. Methods

3.1. System Structure

The program that we run on the robot can be divided into two parts, the guidance service and the patrol service, as shown in Figure 3. The guidance service guides the user to the correct place through the navigation function and provides the translation and emergency contact functions. The patrol function detects suspicious objects on the patrol route through the navigation and detection functions. After the suspicious objects are detected, the detected information is uploaded to the FTP server through the report function. The security personnel can download it from the robot at any time. The control center understands the robot patrol’s current route and movement information. If the object is indeed dangerous, the security personnel can learn the robot’s current location from the system control center; the security personnel then will go to resolve it.

3.2. Pre-Processing Work

Before the system starts to operate, it is necessary to confirm the scope of service of the robot. The robot can pre-load a map that has been created before or build a new map. Assuming that the system must build a new map, we set the scanning map system on the robot system. We use the charging pile of the robot as the generation and judgment of the map. Therefore, the robot needs to start from the charging pile and then build the fixed points of the area in order before returning to the charging pile. In this way, the map building can be completed and the newly made map can be uploaded to the cloud system for storage.

3.3. Emergency Contact and Translation Service

We have developed an emergency contact function through Software Development Kit technology. The address book can be downloaded through the mobile phone application to make a video-call with the robot. When the user clicks the emergency contact button, the robot can directly make a video-call to the service staff who installed the mobile application and ask for help immediately.

Firebase provides translations in 59 languages [9]. The translation results are displayed on the robot screen through a third-party SDK where the user enters the text to be translated in the input box and selects the desired text after the input is completed. The translated result will be displayed in the text box below. In addition, users can have simple conversations with other people through the robot. We use the text-to-speech (TTS) function [10] to convey the text input by the user and the result of the translation to help the user communicate with other people. The translation process is shown in Figure 4.

3.4. Return to the Initial Position

When the robot completes the task, the robot has left its original position. However, as there may be the following user waiting by the side, the setting on our system will not return to the initial work after the user is finished using it. Therefore, we set that if the robot is not used within 10 min, the robot will automatically return to the initial position (charging pile). In addition, when the battery power of the robot is less than 15%, the system will prompt the user whether to return the robot to the initial position. Suppose that the user chooses to continue using it when the robot’s power is less than 10%. A prompt message will pop up again to remind the user to return the robot to the initial position (charging pile) immediately. The flow of returning to the initial position is shown in Figure 5.

3.5. Obstacle Detection Report System

During the robot’s mission, the robot will detect and report at the same time. To enable the robot to detect obstacles on the walking path, we use the camera on the robot with TensorFlow lite (tflite) [11]. We let the robot turn on the camera simultaneously while it is moving and transfer the picture that is taken by the camera to the tflite model to determine whether there are obstacles in the picture, and then report back to the back-end system.

The process of the reporting system is to write a patrol report on the time and type of the obstacle that is detected and send it back to the back-end server. The system will notify the security staff via video-call. After the security personnel connects to the video-call, the security personnel can remotely control the robot to view the nearby situation. If an incident does occur, the security personnel can learn the robot’s current location from the robot control center [12] and go to solve it. After the security personnel handles the obstacles, the robot will continue to move.

3.6. Face Recognition and Object Detection

When the robot is traveling in the hospital, the robot will use a facial recognition system and an object detection system. We chose to use the robot’s camera combined with object detection to allow the robot to simultaneously check whether there are suspicious persons and objects on the road while it is traveling; this function was mainly aimed at when the robot is traveling. When the robot encounters a human that is accidentally entering a specific area or encounters an unknown item, the robot actively reports to the system so that the security personnel can assist in processing.

We use MobileFacenet [13] as the facial recognition model. When the system detects a face, it will capture a face image. We scale the size of the face to 112 × 112 as the input of MobileFacenet. We use the Kaggle [14] and CoCo [15] dataset as our AI face object recognition system dataset, use the tensorflow object detection api to select ssd_mobilenet for training, and then convert the trained model through tensorflow lite converter as a tflite model.

4. Experimental Result

4.1. Map Building

The robot must begin the mapping function built from the charging pile. We used a 360-degree optical radar that is built into the robot, adjacent to the sensor, for map building. Through the movement offset and movement angle of the robot, the position detection of each area of the map is gradually completed. The map construction is completed after the robot returns to the charging station, as shown in Figure 6 where the green dot is the charging pile.

4.2. Location Guide

The analysis was divided into two situations. The first situation is that the robot sets obstacles in the path to the destination, but there is still room for the robot to pass. The experimental results showed that although the robot traveled slowly while avoiding obstacles, it could still avoid obstacles to reach the destination smoothly. In addition, when the obstacle is mobile, the robot will stop moving to avoid a collision with the obstacle. Figure 7 shows the result of preventing obstacles when the robot is traveling where the blue line is the traveling route of the robot.

The second situation is that the route of the robot to the destination is closed, represented here as a closed blue line path. At this time, the robot system shows that it has encountered an obstacle. As shown in Figure 8, the robot will issue a voice message warning. If the robot still cannot pass, the robot will start to recalculate a new route, instead choosing the red line path to move.

4.3. Recognition Result

We tested the accuracy of the converted face and object recognition system model. We set the model confidence level to 80% and tested 213 photos with an accuracy of 93.8%, as shown in Figure 9.

In the process of system identification, we found three situations where most of the wrong identifications are made. The first situation is when two objects overlap, sometimes only the larger object is detected. For example, when a person is holding a cat, the model sometimes fails to detect the cat.

The second situation is that the probability of a cat and a dog being judged by the model will be very similar at a certain angle and light, causing the system to have judgement errors. The third situation is that the object occupies too small a proportion in the image. If the object image is too small, the model may not be detected.

The first and second cases have little impact on our system because the robot will report back as long as it detects a suspicious object during its travel. At this time, security personnel can learn about the situation on the spot through remote video.

The third is more impactful because when the robot does not detect suspicious objects, the robot will continue to move. When the robot approaches the suspicious object, the proportion of the object in the image will increase and the robot will detect a suspicious object.

4.4. Human-Computer Interaction

Our system helps users understand the surrounding environment, and we designed the robot to have three functions in traveling. The first is to detect the types of objects around the user, the second is the distance to the user’s obstacles, and the third is to tell the user when an event occurs. We used the Android open-source library to complete these three functions. First, we used the Tensorflow object detection API model to determine 80 object categories in terms of detecting objects. We prioritized these object categories according to the risk to the user. The objects with high risk will be reported to the user and these objects will be avoided to prevent injury to users.

We used the triangle similarity law that was provided by OpenCV to calculate the distance between the robot and the object. We divided the screen device of the robot into three areas the left, middle, and right. When the detected object appears on the screen, the robot will determine which area the detected object appears in to inform the user of the direction of the obstacle. Finally, after the information that was obtained by the above methods is integrated, the user is notified in time through the speaker of the robot device.

5. Discussion and Conclusions

We propose a robot assistance system. The robot can guide the user to the correct place through voice without human assistance and the robot also considers the translation language function. The emergency contact method is provided through the robot. When an emergency occurs, the security personnel can be directly contacted for assistance and the security personnel can use the remote system to control the robot to assist the user. On the walking route of the robot, we conducted an anti-collision analysis for obstacles. The results showed that the robot could effectively avoid obstacles and avoid danger when walking.