Teleoperated Driving with Virtual Twin Technology: A Simulator-Based Approach

Abstract

This study introduces an innovative Teleoperated Driving (ToD) system integrated with virtual twin technology using the MORAI simulator. The system minimizes the need for extensive video data transmission by utilizing text-based vehicle information, significantly reducing the communication load. Key technical advancements include the use of high-precision GNSS devices for accurate vehicle location tracking, robust data communication via the MQTT protocol, and the implementation of the Ego Ghost mode in the MORAI simulator for precise vehicle simulation. The integration of these technologies enables efficient data transmission and enhanced system reliability, effectively mitigating issues such as communication blackouts and delays. Our findings demonstrate that this approach ensures stable and efficient operation, optimizing communication resource management and enhancing operational stability, which is crucial for scenarios requiring high video quality and real-time response. This research represents a significant advancement in ToD technology, establishing a precedent for integrating virtual twin systems to create more resource-efficient and reliable autonomous driving backup solutions. The virtual twin-based ToD system provides a robust platform for remote vehicle operation, ensuring safety and reliability in various driving conditions.

1. Introduction

As the interest in autonomous driving features has markedly increased, technological advancements in this field have rapidly accelerated [1,2]. Recent research on Autonomous Vehicles (AVs) employs a combination of sensors, cameras, radar, and artificial intelligence to perceive their environment and make driving decisions based on HD precision maps [3,4,5]. Furthermore, the addition of various learning methods related to artificial intelligence has further fueled research in autonomous driving [6,7]. AVs are classified into levels from 0 to 5 by the Society of Automotive Engineers (SAE), with Level 0 representing no automation and Level 5 representing full automation without any need for human intervention. In higher levels of autonomous driving, especially in systems at Level 4 and above, the driver is completely disengaged from vehicle operation [8,9]. This disengagement necessitates reliable backup systems to ensure safety and continuity in vehicle operation. In these contexts, the importance and demand for Teleoperated Driving (ToD) as a backup system for driverless autonomous vehicles have risen significantly in response to potential system failures [10,11,12]. ToD plays a crucial role in the autonomous driving system as a solution for transitioning to a Minimal Risk Condition (MRC) during system failures or in traffic situations that the system cannot handle independently [13,14,15]. With the application of V2N (Vehicle to Network) in C-V2X (Cellular Vehicle to Everything Communication) and with 5G communication, the reliability and speed of communication have greatly improved, enabling faster and more stable teleoperated driving [12,16]. Furthermore, there has been a significant amount of recent research related to 5G technology [17,18,19]. 5G technology provides high data transfer speeds and low latency, enabling real-time response during remote driving and ensuring smooth data communication between the vehicle and the remote operation center. Additionally, the security enhancements of 5G networks play a crucial role in protecting the system from Denial of Service (DoS) attacks and other cyber threats [20,21]. These security enhancements increase the reliability of teleoperated driving systems, ensuring that autonomous vehicles can operate safely in various driving conditions.

Meanwhile, virtual twin technology, replicating real roads in the digital world, has become integral to autonomous driving technology [22,23]. It is primarily used for simulating and evaluating complex, potentially hazardous traffic scenarios that are impractical to replicate in reality. This technology not only allows for detailed testing and refinement of driving algorithms but also enhances the safety and adaptability of autonomous systems by mirroring diverse real-world conditions and integrating continuous feedback for improvement [24,25,26].

As a solution to the challenges faced by traditional ToD systems, this study proposes the integration of ToD with a virtual twin system. By using a virtual twin system in ToD, instead of transmitting video, vehicle information like coordinates, direction, and steering angle can be sent in a message format, significantly reducing the data communication load. This approach effectively addresses issues such as communication interruptions, latency in real-time monitoring, and remote assistance, along with high costs and telecommunication hardware challenges [27]. This research aims to enhance the effectiveness of the ToD system by applying digital twin technology. Through these developments, we seek to improve operational efficiency, reduce latency, and ensure seamless remote control of autonomous vehicles in various driving conditions.

The rest of this paper is organized as follows: Section 2 describes the methods and system architecture used in developing the virtual twin ToD system. Section 3 presents the results and discusses the performance and effectiveness of the proposed system. Section 4 concludes the paper with a summary of findings and suggestions for future research directions.

2. Methods

2.1. Teleoperated Driving (ToD)

In this research, we built upon a previously developed ToD system to construct the virtual twin ToD system [28]. While adhering to the basic vehicle operation mechanisms of the original ToD system, we utilized advanced simulators for visualizing vehicle information and camera footage. At the Remote Operation Center (ROC), a high-performance PC based on Linux [Ubuntu 20.04] was installed for stable and fast processing, along with a Logitech G29 driving device. The remote control mechanism was divided into two primary axes: longitudinal control using a PID (Proportional–Integral–Differential) controller based on the input values of brake and throttle; and lateral control adjusting the steering angle values transmitted in real-time through a PI (Proportional–Integral) controller [29]. This control approach allowed for real-time reflection of the remote driver’s steering input, enabling synchronous operation with the vehicle.

In terms of vehicle system configuration, we equipped the vehicle with a PC [NVIDIA, Jetson Xavier AGX], image capture card [CANLAB, CLMU-200N], and Sekonix camera. Additionally, a high-precision GNSS [Novatel, Pwrpak7D-E2] device was installed for accurate vehicle location tracking, enhancing the precision of our positioning system. The experimental vehicle selected was the Hyundai Kona Hybrid, equipped with a 5G/LTE router [HUCOM wireless, HE-900] to facilitate high-speed data transmission. At the remote control station, we established a stable communication environment through a 100 Mbps Ethernet connection, focusing on enhancing the reliability of remote control operations.

2.2. Virtual Twin ToD System

We developed a virtual twin ToD system by integrating a ToD vehicle, Kona, with the MORAI simulator, and demonstrated it at the C-track, located in Cheongju City. C-track is an autonomous driving test space equipped with various test roads, including circular, urban, and application roads [4,30]. An aerial illustration of C-track is shown in Figure 1. Due to legal restrictions related to ToD demonstrations in Korea, the virtual twin ToD system was implemented and demonstrated within C-track, not on public roads.

Figure 2 presents a detailed diagram of the virtual twin ToD system. ROC sends control inputs such as steering, gear shifting, and acceleration for vehicle operation to the ToD vehicle through the MQTT network. The ToD vehicle transmits its position and rotation data to the simulator PC at the ROC. The simulator PC integrates this location information with the steering and velocity data received directly through the ROS network from the controller, then sends it to MORAI through the ROS bridge. The MORAI simulator processes these data, producing a precise and clear visual representation. Additionally, as an option to prepare for and prevent unexpected accidents or problems, a low-bandwidth AVM (Advanced View Monitor) video feed can be transmitted from the ToD vehicle to the operator [28].

2.3. Virtual Twin Implementation

Figure 3 illustrates the fundamental concepts and differences between the traditional ToD and virtual twin ToD systems. In the traditional approach, the ROC communicates directly with the vehicle, where the substantial communication load required for video streaming presents one of the biggest challenges. Even though video data are compressed through encoding before transmission, the inherently large size of video data inevitably leads to significant load. In contrast, the virtual twin ToD system incorporates a simulator for visualization, which substantially reduces and distributes data transmission size, thereby implying a relatively lower communication load and overall burden.

We utilized the latest version of MORAI’s simulator, MORAI 23.R.1.0, for the virtual twin. This simulator was operated on the ToD ROC PC. The MORAI simulator operates in an ROS environment, so we installed and used the ROS Noetic version suitable for the Ubuntu 20.04 environment. The advanced option of the simulator, the Ego Ghost mode, which replicates real vehicles in the simulation world, was primarily used [31]. This mode receives vehicle information (position, rotation, velocity, and steering angle) in message format. The message format for the Ego Ghost mode is standardized in the egoGhost.msg format included in MORAI’s ROS package, allowing for direct utilization. The position and rotation information is gathered from the high-precision GNSS device installed in the test vehicle, KONA, and transmitted from the vehicle to the ROC using the TCP/IP-based MQTT protocol.

The velocity and steering angle data are directly monitored and captured through the ROC operator equipment. Utilizing the same PC for both simulation and ToD operations enables efficient data transfer, achieved through ROS’s internal communication system. Through ROS topics, a robust message exchange between nodes occurs, allowing for real-time, accurate updates of vehicle dynamics in the simulation.

To convert the received position and rotation data into the MORAI simulator’s internal coordinate system and vehicle heading values, the GNSS coordinates were first transformed into UTM (Universal Transverse Mercator) coordinates using the Python UTM library. We then subtracted the easting and northing offset values from these transformed UTM coordinates to complete the conversion. These offset values are map-specific, provided for each distinct virtual environment available in the MORAI simulator when using its GPS sensor. Each map in the MORAI suite represents a different geographical area, with unique easting and northing coordinates. The rotation, consisting of x, y, and z axes, used only the z-axis value as the yaw value. Received in degrees, these rotation values were adjusted by subtracting a specific offset to obtain the Heading value. During the fine-tuning process of the easting, northing, and rotation offsets, we conducted hands-on driving tests on a designated test track (C-track) with the vehicle; thus precisely setting the vehicle’s position and heading to reflect real-world conditions. Through this process, the egoGhost.msg is completed and input into the MORAI simulator via the ROS bridge. To verify the accurate implementation of the ToD vehicle in the simulation, a GPS sensor was also mounted in the simulation at the same location as the GNSS on the actual vehicle. When driving on the track, sensors from both the real world and the simulation were read to analyze the differences between them.

In measuring latency, we precisely calculated the delay in the MQTT communication process using the message ID and its corresponding timestamp. Additionally, we assessed the time it took for the vehicle’s data to appear on the simulator screen by tracking when the egoGhost.msg was processed and the output of ROS topic data. Combining these two aspects, we determined the total delay for the simulation’s visualization. For the traditional ToD system, we measured the complete latency from the vehicle’s camera to the ROC monitor’s display, including the time needed for video encoding and decoding. In contrast, with the virtual twin system, we measured the delay from the moment the vehicle’s GNSS device transmitted data until the vehicle was rendered on the simulator screen. Figure 4 visually represents these latency differences.

3. Results and Discussion

Figure 5 presents a demonstration environment and example of the virtual twin ToD. The ToD operator drives while observing the simulation environment and, if necessary, can optionally check the AVM (Advanced View Monitor) video (Figure 5a). Figure 5b,c show the views that the operator would see in traditional and virtual twin ToD systems, respectively. In Figure 5b, the view is fixed to the camera’s position, whereas in Figure 5c, the simulation view can be adjusted according to the user’s preference, including angle of view, direction, and distance.

3.1. Driving Test

The driving tests involved comparing GPS data from a real vehicle (KONA) and a simulated vehicle using the virtual twin ToD system. Tests were conducted in both expressway and urban areas. In urban areas, specific situations such as roundabouts, U-turns, and left turns were targeted for measurement. GPS data were acquired at a frequency of 50Hz from both sources, and the distance differences between the two sets of coordinates were calculated and recorded simultaneously.

Table 1. The coordinate differences of the virtual twin ToD and the real vehicle.

Coordinate Difference [m]	Average	Median	Standard Deviation
Expressway	0.05650	0.02742	0.08014
Roundabout	0.07420	0.06251	0.03379
U-Turn	0.05654	0.04742	0.02454
Left Turn	0.07361	0.06450	0.03066

and Figure 6 and Figure 7 display the results of operating a vehicle using the virtual twin ToD system. In Table 1, which presents the results of coordinate differences for each driving course, the average distance difference across all test routes was less than 0.08 m. This indicates a negligible difference between the real and simulation datasets, with the distance discrepancy being only a few centimeters. Figure 6 overlays the actual GPS data of the KONA vehicle (red line) on the satellite image of the C-track, along with the GPS data obtained from the simulator (blue line) for the expressway. The two paths (KONA GPS and simulator GPS) perfectly coincide in the image.

To facilitate further analysis, the line graph and histogram depicting the difference in GPS data obtained are displayed in Figure 7. The line graph on the top part of Figure 7 presents a direct comparison of distance discrepancies between real-world and simulated GPS readings. On this graph, coordinate differences are traced in blue and then smoothed with a moving average, represented by the red line. The orange and green lines mark the mean and standard deviation, respectively. The histogram on the bottom, using a logarithmic y-axis, reveals that the data points predominantly fall within a 0.2 m range, indicating negligible variance in the GPS readings.

The results from urban areas in Figure 8 were found to be nearly identical to those from the expressway. In Figure 8a–c, it was observed that the paths for roundabouts, U-turns, and left turns all align well, indicating accurate coordinate matching. Furthermore, in Figure 8d, it can be seen that for all three scenarios, the noise-mitigated red line remains below 0.1 m. Noise can be visually identified in each line graph above. The smoothed graph, obtained using the moving average method, compensates for this noise and stays below the 0.1 m range, aligning closely with the mean value. These results validate that the vehicle represented in the virtual twin and the actual vehicle move along nearly identical paths, proving the accuracy of the simulation.

In the graphs from Figure 7 and Figure 8, the noise shows unique characteristics that differ from typical sensor noise. Unlike common white noise with random magnitudes, the noise here includes specific error values, resulting in a band-like appearance. Additionally, there are no values below a certain level. To analyze this noise, we performed additional experiments for a similar graph while the vehicle was stationary, shown in Figure 9. Figure 9a’s blue line depicts the stationary state with the same y-axis scale as before, revealing almost no visible noise due to its small scale, aligning closely with the smoothed red line. For a detailed view, the y-axis is magnified in Figure 9b, and the data are also presented as a histogram in Figure 9c. The magnified graph in Figure 9b,c exhibits typical white noise, which also shows a Gaussian distribution shape in the histogram.

This random white noise is characteristic of GNSS hardware and sensor noise. While the simulator’s GPS values remain perfectly fixed when stationary, the real vehicle’s GPS signal fluctuates due to noise. The difference between the fixed values and this fluctuation appears as white noise in Figure 9, with very slight values. The fluctuation is relatively large initially and decreases over time, stabilizing after about the 4000th point, likely due to the system settling down after being powered on. This indicates that the noise observed in Figure 7 and Figure 8 is not typical white noise but rather originates from a different source or issue.

One such issue is the error from data sampling inconsistencies. Both the simulator and the real vehicle’s GNSS equipment are set to a 50 Hz sampling rate. Although both the simulator and the real vehicle’s GNSS equipment are set to same sampling rate, the simulator consistently maintains this rate, while the real equipment may fail to receive all signals at 50 Hz, especially during motion, potentially missing some data points. In the previous experiments (expressway and urban area), the vehicle moved at approximately 10 km/h (2.78 m/s), and, with a 50 Hz sampling period (0.02 s), the distance between sampling points should be about 5.56 cm. Figure 10 illustrates a segment of Figure 8d, highlighting quantized noise signals with specific magnitudes (red box). These peaks are approximately 16.8 cm, 11.2 cm, and 5.6 cm, corresponding to 3, 2, and 1 times the distance between sampling points, respectively. When plotted on a line graph, these consistent error values connect with the preceding and subsequent points, forming vertical lines and ultimately creating the band-like graph. Adjusting the simulator’s GPS timing to match the real vehicle’s data input timing could improve this discrepancy, as the real vehicle’s precise sampling timing is challenging to handle due to hardware issues.

3.2. Data Transmission

The traditional ToD system relied on driving through real-time video streaming from the camera (Figure 5b). The acquired images in this system were encoded using the standard video compression technology, H.264/AVC, and transmitted through UDP communication using the real-time end-to-end transmission protocol, RTP. This method operates at a resolution of 640 × 480 pixels (SD quality) and 30 FPS, with a data transmission rate of approximately 1 Mbps.

In contrast, the virtual twin ToD system primarily relies on data transmission in the form of ROS messages, mainly containing coordinate information. The total size of the data packet, including the transmitted message, is about 63 bytes, sent at a frequency of 50 Hz. Consequently, the data transmission rate is measured to be approximately 25 Kbps.

Table 2. The results of both traditional and virtual twin ToD measurements.

	Transmission Medium	Data Packet [Byte]	Transmission Rate [bps]	Control Latency [s]	Video/ Visualizing Latency [ms]	Frame Rate [FPS]
Traditional ToD	Video Streaming	Variable	1 M	0.0312	32.5	30.38
Virtual Twin ToD	Message	63	25 K	0.0312	4.5	50

shows a comparison between the traditional and virtual twin ToD systems. The most significant difference in network communication lies in the medium of transmission, specifically image versus message, leading to a substantial difference in data size and communication load. Although a direct comparison of data packet sizes is challenging due to the use of video compression technology in the traditional ToD system, a comparison of transmission rates is possible. The transmission rate of 25 Kbps for the virtual twin ToD and 1 Mbps for the traditional ToD shows a difference of about 40 times.

This substantial disparity in transmission rates can lead to significant challenges in network resource allocation, as higher data rates may require more bandwidth and potentially cause network congestion. In scenarios with limited network capacity, the high data throughput of traditional ToD could result in slower transmission speeds or even data loss, having an impact on the reliability of remote vehicle operation. Furthermore, such a large difference in transmission rates might necessitate the implementation of a more robust and costly network infrastructure to ensure consistent and reliable data flow, particularly for traditional ToD systems operating in dense urban areas or over extended distances.

Furthermore, traditional ToD uses UDP communication, while virtual twin ToD employs TCP/IP-based MQTT communication [32]. Unlike UDP, MQTT ensures reliability and integrity in the data transmission process. While MQTT may be slower than UDP in terms of speed, the measured delay time was only a few milliseconds. MQTT, suitable for transmitting small-sized messages, maintains stable communication even in network delays or congestion. These characteristics contribute to the stable operation of the virtual twin ToD system in even unstable network environments.

3.3. Latency

In remote vehicle control, latency is extremely important. Reducing latency allows the remote driver to control the vehicle more naturally and quickly recognize obstacles and surrounding conditions. Latency can be divided into control-related latency from the ROC to the vehicle and visualization latency from the vehicle to the ROC for images or simulation. The control-related latency at the ROC is the same as in the traditional ToD system, averaging 0.0312 s. However, there is a difference in visualization latency. Table 2 presents these results. The latency measurements indicated an average latency of 32.5 ms for the traditional ToD and 4.5 ms for the virtual twin ToD.

The reasons behind these latency differences can be found in the data transmission diagram illustrated in Figure 4. The most notable distinction between traditional ToD and virtual twin ToD is the presence or absence of video processing. With video streaming, the process of encoding for large video transmission is essential, and, subsequently, the video must be decoded into a playable format, adding additional processing time ($\mathsf{\Delta }{\mathrm{t}}_{enc}$, $\mathsf{\Delta }{\mathrm{t}}_{dec}$) and computational load, thereby extending the total latency ($\mathsf{\Delta }{\mathrm{t}}_{proc1}$, $\mathsf{\Delta }{\mathrm{t}}_{proc2}$) [33,34].

In contrast, the virtual twin ToD simplifies the data processing and transmission ($\mathsf{\Delta }{\mathrm{t}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$) and streamlines the data delivery to the simulator ($\mathsf{\Delta }{\mathrm{t}}_{\mathrm{r}\mathrm{e}\mathrm{c}}$), shortening the latency through these differences. Though the visualization latency of the virtual twin ToD may vary with simulator operation and the use of additional PC resources, the current findings show an approximate 86% reduction in latency, significantly enhancing the remote driver’s reaction time and perception of safe driving.

Such a reduction in latency enhances the accuracy and responsiveness of remote control, particularly beneficial in complex road environments or urgent situations. Furthermore, lower latency provides the remote driver with more accurate, real-time information about the vehicle and its surroundings, thus increasing the overall safety and efficiency of remote driving.

4. Conclusions

In this study, we introduced an innovative virtual twin ToD approach by integrating traditional ToD systems with virtual twin technology to enhance the efficiency and safety of remote vehicle control. The system significantly improved the speed and precision of remote driving. Path analysis comparing the virtual twin ToD system with an actual vehicle showed an average trajectory error of within 8 cm, demonstrating high accuracy and reliability. Additionally, the virtual twin system’s message-based data transmission approach increased efficiency, optimized network resource usage, and enhanced communication stability, significantly reducing latency and improving the reaction time and safety perception of the remote driver. These results underscore the virtual twin ToD system’s capacity to replicate real driving conditions closely and provides a robust platform for informed decision-making during remote operation.

To conclude, while our virtual twin ToD system reduces communication load and enhances efficiency, further study is needed to fully realize its potential in real-world applications. The current system has been validated only with the driving vehicle, highlighting its limitations in public road scenarios. Future research should focus on integrating the virtual twin system with real driving conditions, visualizing external object data through the simulator, and incorporating LiDAR sensor data to identify potential hazards around the vehicle. These enhancements will ensure a more accurate and robust virtual twin system, leading to safer and more efficient teleoperated driving. Overall, this study provides a solid foundation for developing advanced virtual twin ToD systems and sets the stage for future research to build on these findings, broadening the potential for applications in remote control technologies and various other fields.