CarTwin—Development of a Digital Twin for a Real-World In-Vehicle CAN Network
Round 1
Reviewer 1 Report
The authors propose an experimental prototype for a car Digital Twin based on the CAN communication bus. Although is an interesting matter, I suggest the authors revise the entire paper structure and deepen the experimental part so that the article can be considered scientifically valid.
The authors describe too much the CAN bus background and its topology. This section, although important, should take up fewer pages and be more concise.
The following section describes the software implementation of the Digital Twin and is detailed well enough, as well as the part on the hardware and software level deployment.
I consider the evaluation section the most troublesome. While well explaining the experimental conditions on local roads and highways both for the real car and for the digital twin, some details are lacking, such as an indication of how many trials were made for every condition. Moreover, I'd like to see a Results section, in which authors could report the numerical results of their experiments in a more concise way (for example splitting the results between local roads and highway conditions and using some tables to summarize them) as well as a Discussion section, where results are analyzed more extensively. The only "discussion" part of the paper is the sentence "By comparing the signals shown in Figure 13, we validate the real-world behavior of the models from the CarTwin setup", but no detail is given on how the comparison was carried out. I don't think that a visual comparison is sufficient to validate a model.
In the Abstract section, the authors claim that "This experimental prototype can be further used in numerous applications. One such specific use-case could be the validation of cyber-attacks and defenses on in-vehicle networks in a more realistic manner without exposing the actual car to such attacks, since the latter could cause safety concerns" while this topic was not discussed in the paper. How the model could be useful to study cyber-attack on cars while the authors did not include them in their experimentation?
Author Response
Reviewer#A
Reviewer #A, Concern #1: The authors propose an experimental prototype for a car Digital Twin based on the CAN communication bus. Although is an interesting matter, I suggest the authors revise the entire paper structure and deepen the experimental part so that the article can be considered scientifically valid.
Author response: We thank the reviewer for the comment. We now add more details on the experiments, the software associated with the experimental model and the interpretation of the experimental data. Modifications in the paper are highlighted in blue and explained in the response to each of the reviewer comments that follow.
Reviewer #A, Concern #2: The authors describe too much the CAN bus background and its topology. This section, although important, should take up fewer pages and be more concise.
Author response: We thank the reviewer for the comment. We shorten the CAN background section by removing some unnecessary text, e.g., the bit stuffing information related to CAN communication and several other details related to CAN voltage. From the topology section we removed some details related to ECUs. The topology description from Section III is needed for the experimental setup of CarTwin shown in Figure 12 (since we are using the exact same harnesses shown in Figure 3).
Reviewer #A, Concern #3: The following section describes the software implementation of the Digital Twin and is detailed well enough, as well as the part on the hardware and software level deployment.
Author response: We thank the reviewer for the positive appreciation of this section from our work.
Reviewer #A, Concern #4: I consider the evaluation section the most troublesome. While well explaining the experimental conditions on local roads and highways both for the real car and for the digital twin, some details are lacking, such as an indication of how many trials were made for every condition. Moreover, I'd like to see a Results section, in which authors could report the numerical results of their experiments in a more concise way (for example splitting the results between local roads and highway conditions and using some tables to summarize them) as well as a Discussion section, where results are analyzed more extensively. The only "discussion" part of the paper is the sentence "By comparing the signals shown in Figure 13, we validate the real-world behavior of the models from the CarTwin setup", but no detail is given on how the comparison was carried out. I don't think that a visual comparison is sufficient to validate a model.
Author response: We thank the reviewer for the comprehensive comment related to the evaluation section. We have added two subsections inside Section 6 (Evaluation of the Digital Twin Model in the experimental setup) after we described how the models were integrated in the CarTwin experimental setup. In the “Results” subsection we provided details related to the vehicle trace, analyzed the signal outputs from the traces (model output, real-world vehicle signals) on the whole trace and then, separately, for the local road driving conditions and highway conditions. We also add a new paragraph to show the correlation between the model outputs and the real-world vehicle signals. The following paragraphs were added:
“Local roads. In Figure 15 we show the signals while the vehicle is driven in the city and on local roads. The plots from the left side are the outputs from our simulation, i.e., Figure 15 (i), Figure 15 (iii), Figure 15 (v) while the plots from the right side are the signals collected from the real car, i.e., Figure 15 (ii), Figure 15 (iv) and Figure 15 (vi). For the vehicle speed signal, the model output varies between 0 km/h and 85 km/h while the vehicle speed signal collected from the vehicle varies between 0 km/h to 80 km/h (with the exception of one spike to 120 km/h during a car overtake in the real scenario). Engine speed varies between 800 rpm and 2500 rpm in the model output while in the vehicle trace the signal value is of 900 rpm to 2500 rpm (except for a few spikes at 4500 rpm). We have a different number of samples for the trip distance (between our simulation and the signal collected from the car) because our model runs at 20ms while the car CAN bus message that contains the odometer is transmitted every 1 second. However, the trip distance signal has a similar variation over time.
Highway. In Figure 16 we show the signals while the vehicle is driven on a highway. Again, the plots from the left side are the outputs from our simulation, i.e., Figure 16 (i), Figure 16 (iii), Figure 16 (v) while the plots from the right side are the signals collected from the real car, i.e., Figure 16 (ii), Figure 16 (iv) and Figure 16 (vi). For the vehicle speed signal, the model output varies between 90 km/h and 135 km/h while the vehicle speed signal collected from the vehicle varies between 125 km/h to 148 km/h (with the exception of a few spikes to 80 km/h at the beginning of the plot). Engine speed varies between 2600 rpm and 3700 rpm in the model output while in the vehicle trace the signal value is more stable between 2600 rpm and 3400 rpm (except for a few spikes at the beginning of the plot between 1000 rpm and 4600 rpm). The trip distance signal from the model and the trip distance signal from the car have a similar variation in time.
Statistical comparison. As an additional metric for the accuracy of the model outputs, we compute the mean values for the differences between the output signals of the model and those from the vehicle trace and the correlation coefficient between these signals. The values were computed for the entire trace, which includes both driving on the local road and the highway, and are shown in Table 3. Since the digital twin model is designed by us in MATLAB/Simulink, while the real-world vehicle is an actual physical system that is influenced by the environment, differences between the results are expected (both the vehicle and the environment are nearly impossible to model with absolute accuracy). The mean of the recorded differences is about 25 km/h for the vehicle speed and 610 rpm for the engine speed. We note that the range of the signals is computed according to the collected dataset and the only common input which links the synthetic model with the physical is the signal applied to the brakes. The correlation coefficients between the synthetic data and the real-world data are 0.85 and 0.71 respectively (for the vehicle and engine speed), which show a good to strong relation between the synthetic and the real-world result.”
Reviewer #A, Concern #5: In the Abstract section, the authors claim that "This experimental prototype can be further used in numerous applications. One such specific use-case could be the validation of cyber-attacks and defenses on in-vehicle networks in a more realistic manner without exposing the actual car to such attacks, since the latter could cause safety concerns" while this topic was not discussed in the paper. How the model could be useful to study cyber-attack on cars while the authors did not include them in their experimentation?
Author response: We thank the reviewer for the remark. We now add a new subsection dedicated to potential applications and explain how this model can be useful for analyzing cyber attacks on the CAN bus. To avoid misunderstandings on the main focus of our work, we remove the statement on security from the abstract. The following text was added.
“6.2 Possible applications
A possible application for the experimental model is in the evaluation of cyberattacks on in-vehicle networks. Indeed, many related works on intrusion detection for in-vehicle buses, use real-world traces collected inside the vehicle which are augmented with attacks in an off-line manner, e.g., [54], [55], [56]. One of the most common attacks in such works are fuzzing attacks in which frames containing random data are sent on the bus [57], [58], [59]. Clearly, exposing the actual car to such an attack may cause safety concerns since random packets may cause unexpected behavior for the car. A reason for which, the off-line attack procedure is a good choice. However, this off-line attack procedure overlooks the impact of one parameter on another. As we show in Figure 17 (i) and Figure 17 (iii) when the vehicle speed and engine speed are subject to an off-line attack augmentation, the attack values will show as spikes on the original signals. These spikes on the vehicle speed and engine speed are however poorly correlated which is not necessarily the case in a real vehicle. Obviously, in reality, there is good correlation between these two signals and thus the off-line attacks may be quite artificial. This is visible in Table 4, where the correlation between the two attacked signals is 0, which is expected as the attack values are random, while in the real-world data as well as in the CarTwin experimental model, the correlation between legitimate frames for the same signals is around 0.9. The lack of correlation between these two signals (vehicle speed and engine speed) will make such attacks easier to detect but also not very realistic for the real-world behavior of the car. In Figure 17 (ii) and Figure 17 (iv) we show how an attack on the vehicle speed will influence engine speed when the CarTwin model is employed. The correlation is significantly better for the attacks on the CarTwin model as can be seen in Table 4. Even in case of the attack frames on vehicle speed and their impact on engine speed, the correlation coefficient is still 0.49 (note that in the off-line generated trace the correlation is 0). There is a decreased correlation with an increase in the attack probability which is expected (as the attack becomes more frequent, the correlation lowers since the attack represents an anomaly). While it is out of scope for us to delve further into security related details, this suggests CarTwin to be useful in gaining insights into cyberattacks.
Another possible area of investigation for the model is safety and fault tolerance. Our model does contain safety-relevant signals, such as brake, buckle and airbag status signals, etc. Fault tolerance is indeed highly recommended or even mandatory in case of safety critical signals. A well-known and employed solution to assure fault-tolerance is redundancy, either by using different sources for a signal or deriving it by distinct computations. The case in which these signals are faulty, redundancy is a means to correct such faults. For example, vehicle speed is reported both by the ABS and PCM controllers, both of which are present in our model, etc. Such consistency checks can be also done based on the data from the model.”
[54] Zhou, A.; Li, Z.; Shen, Y. Anomaly detection of CAN bus messages using a deep neural network for autonomous vehicles. Applied Sciences 2019, 9, 3174.
[55] Hossain, M.D.; Inoue, H.; Ochiai, H.; Fall, D.; Kadobayashi, Y. LSTM-based intrusion detection system for in-vehicle can bus communications. IEEE Access 2020, 8, 185489–185502.
[56] Andreica, T.; Curiac, C.D.; Jichici, C.; Groza, B. Android Head Units vs. In-Vehicle ECUs: Performance Assessment for Deploying In-Vehicle Intrusion Detection Systems for the CAN Bus. IEEE Access 2022, 10, 95161–95178.
[57] Kim, H.; Jeong, Y.; Choi, W.; Lee, D.H.; Jo, H.J. Efficient ECU Analysis Technology Through Structure-Aware CAN Fuzzing. IEEE Access 2022, 10, 23259–23271.
[58] Aldhyani, T.H.; Alkahtani, H. Attacks to automatous vehicles: a deep learning algorithm for cybersecurity. Sensors 2022, 22, 360.
[59] Islam, R.; Devnath, M.K.; Samad, M.D.; Al Kadry, S.M.J. GGNB: Graph-based Gaussian naive Bayes intrusion detection system for CAN bus. Vehicular Communications 2022, 33, 100442.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The work shows the design and development of a digital twin for car communications using the CAN protocol.
The paper is well written, and has a good order that allows you to understand the details of development and experimentation even if you are not an expert in the automotive field.
For those of us who have worked in the automotive sector, we know how difficult it is to deal with the communications of a car, starting from the fact that it is expensive to have the ECUs physically, to the difficulty of making the communication harnesses. Therefore, I believe that the work contributes significantly to the state of the art.
I only have a few specific observations.
-You mention a C# application to connect with vector, does that application have a graphical interface? can you give more details?
-In the case of matlab, did you only use the simulink library? How do you go from simulink to C code?
-It is mentioned that the digital twin has applications to analyze security, could it be applied to analyze safety? For example, if a sensor on the car fails and indicates the wrong speed, how should the other ECUs behave? This is thinking of developing technology that is fault tolerant.
-I think that the conclusions can be better, you have enough data and results to write a better conclusion
Author Response
Reviewer#C
The work shows the design and development of a digital twin for car communications using the CAN protocol.
The paper is well written, and has a good order that allows you to understand the details of development and experimentation even if you are not an expert in the automotive field.
For those of us who have worked in the automotive sector, we know how difficult it is to deal with the communications of a car, starting from the fact that it is expensive to have the ECUs physically, to the difficulty of making the communication harnesses. Therefore, I believe that the work contributes significantly to the state of the art.
I only have a few specific observations.
Author response: We are grateful to the reviewer for the positive appreciation of our work.
Reviewer #C, Concern #1: -You mention a C# application to connect with vector, does that application have a graphical interface? can you give more details?
Author response: We thank the reviewer for this concern. Yes, the application has a graphical interface which is now depicted in Figure 12 of the manuscript. The application interface shows the various signals that are collected or sent while the deployment of this application with more details added in Section 5 (Hardware and Software Level Deployment of the Digital Twin).
Reviewer #C, Concern #2: -In the case of matlab, did you only use the simulink library? How do you go from simulink to C code?
Author response: We thank the reviewer for this concern. We have generated C code from Simulink models by using the Simulink Embedded Coder feature from MATLAB. We now explain this better in Section 5 (Hardware and Software Level Deployment of the Digital Twin) where we add a reference to the Mathworks website that details this feature. The generated model code is then integrated in the AURIX studio project that we configured for the embedded microcontroller Infineon AURIX TC275. The details on pages 13 and 14 on this matter have been improved (modifications are highlighted in blue).
Reviewer #C, Concern #3: -It is mentioned that the digital twin has applications to analyze security, could it be applied to analyze safety? For example, if a sensor on the car fails and indicates the wrong speed, how should the other ECUs behave? This is thinking of developing technology that is fault tolerant.
Author response: We thank the reviewer for the comment. Our model also deals with safety-relevant signals, such as brake, buckle and airbag status signals, etc. Fault tolerance is indeed highly recommended (if not mandatory) in case of safety critical signals. A well-known and employed solution to assure fault-tolerance is redundancy, either by using different sources for a signal or deriving it by distinct computations. The case in which these signals are faulty, redundancy is a means to correct such faults. For example, As the reviewer suggests, vehicle speed is reported both by the ABS and PCM controllers both of which are present in our model, etc. Currently, we did not perform redundancy checks, but these can be in principle added to the model. We added a discussion on this in subsection “6.2. Possible applications”.
Reviewer #C, Concern #4: -I think that the conclusions can be better, you have enough data and results to write a better conclusion
Author response: We thank the reviewer for the remark. We have improved the conclusion of the work.
Author Response File: Author Response.pdf
Reviewer 3 Report
This is a good study and making interest of a reader specially for me but I have few concerns that will increase the readibility of this work.
1. Write complete names of figures while citing them such as figure 13 (ii), and figure (iii).
2. There are many typos such as the word is starting from small letters in the start of a sentence.
3. The idea is good but authors should have to present in a significant way.
4. The comparative analysis of this work is missing. Authors added their results but they should compare it with some state of the art articles.
5. There are various equations provided but not cited and discussed in the article.
6. You discussed in the abstract "One such specific use-case could be the validation of cyber-attacks and defenses on in-vehicle networks in a more realistic manner without exposing the actual car to such attacks, since the latter could cause safety concerns", can you please link this sentence and provides the mapping of this work in the paper?
7. In my humble opinion, the abstract is not synced with conclusion and contributions.
8. I suggest to add some more recent articles. Further you have cited many articles related to your work but you have not provided comparison with them.
Author Response
Reviewer#B
Reviewer #B, Concern #1: This is a good study and making interest of a reader specially for me but I have few concerns that will increase the readibility of this work.
Author response: We thank the review for the comments, a point-to-point response follows. Modifications in the paper are highlighted in blue and explained in the response to each of the reviewer comments that follow.
Reviewer #B, Concern #2: Write complete names of figures while citing them such as figure 13 (ii), and figure (iii).
Author response: We thank the reviewer for this finding. We now reference the Figures with their complete names.
Reviewer #B, Concern #3: There are many typos such as the word is starting from small letters in the start of a sentence.
Author response: We thank the reviewer for this finding. We now fix all sentences to start with a capital letter.
Reviewer #B, Concern #4: The idea is good but authors should have to present in a significant way.
Author response:We thank the reviewer for the remark. We did several improvements in the presentation of our work (also in response to other reviewer’s comments) as follows: 1) improve the comparison with related approaches, 2) add more details on the application interface, 3) add more clarifications regarding the experiments, and 4) emphasize on intrusion detection systems as a potential application for the model.
Reviewer #B, Concern #5: The comparative analysis of this work is missing. Authors added their results but they should compare it with some state of the art articles.
Author response: We thank the reviewer for the remark. The design of digital twins for real-world vehicles is only a recently emerged topic. The closest work to ours is [9] which designs a digital twin for the mechanical behavior of the car. This work does not contain an actual car topology which is the main objective of our investigations. We now add a comparative analysis with recent works on car twins in Table 1 and discuss this in the “Introduction and motivation” section:
“The design of digital twins for cars is only a recently emerged topic and there is only a very limited number of related works which can be compared with the developments from our work. A research team from Toyota has designed PASTA (Portable Automotive Security Testbed with Adaptability) [13], an adaptable vehicle cybersecurity testbed as an evaluation environment for automotive attacks. Their testbed integrates development boards with models for various ECUs functionalities from real-world vehicles which communicate on two separate CAN networks connected through a Gateway unit. Authors from a more recent work, RAMN (Resistant Automotive Miniature Network) [14], designed a small and inexpensive automotive testbed that includes implementation of models for the gateway, powertrain, chassis and body ECUs connected to a single CAN bus. As already stated in the introduction section, an implementation of digital twins for vehicle dynamics using the steering system, braking system and powertrain is done by authors in [9]. Our contribution with respect to design of digital twins is the utilization of the same real-world vehicle bus topology besides the definition and implementation of ECU models. A comparison of our work with research papers that address vehicle level functions using digital twins is shown in Table 1.”
[9] Shetty, S.S. Development of a Digital Twin of a Toyota Prius Mk4. Master thesis, Eindhoven University of Technology, 2022.
[13] Toyama, T.; Yoshida, T.; Oguma, H.; Matsumoto, T. PASTA: portable automotive security testbed with adaptability. London, blackhat Europe 2018.
[14] Gay, C.; Toyama, T.; Oguma, H. Resistant Automotive Miniature Network. In Proceedings of the Chaos Computer Congress, 2020.
Reviewer #B, Concern #6: There are various equations provided but not cited and discussed in the article.
Author response: We thank the reviewer for this finding. We have now referenced each equation in the manuscript and provided more details related to the terms from the equations. Changes for this concern are highlighted in blue in the manuscript.
Reviewer #B, Concern #7: You discussed in the abstract "One such specific use-case could be the validation of cyber-attacks and defenses on in-vehicle networks in a more realistic manner without exposing the actual car to such attacks, since the latter could cause safety concerns", can you please link this sentence and provides the mapping of this work in the paper?
Author response: We thank the reviewer for the remark. In the abstract we just wanted to suggest an application area, not necessarily to pursue it as a case study since it was not within the objectives of the paper. But we agree that this could be better explained and dedicate subsection “6.2. Possible applications” to this purpose. To avoid confusions on the paper content, we remove the statement from the abstract.
Reviewer #B, Concern #8: In my humble opinion, the abstract is not synced with conclusion and contributions.
Author response: We thank the reviewer for the remark. We have revised the abstract and the conclusion of the work.
Reviewer #B, Concern #9: I suggest to add some more recent articles. Further you have cited many articles related to your work but you have not provided comparison with them.
Author response: We thank the reviewer for the suggestion. We added a few more recent articles, i.e., [14], [35], [54], [55], [56], [57], [58], [59]. Also, as already stated in the author response for Reviewer #B Concern #5, we have added a new paragraph with more details of recent articles in the same area and a Table containing a comparison of existing research papers that address vehicular digital twins.
[14] Gay, C.; Toyama, T.; Oguma, H. Resistant Automotive Miniature Network. In Proceedings of the Chaos Computer Congress, 2020.
[35] Conti, M.; Donadel, D.; Turrin, F. A survey on industrial control system testbeds and datasets for security research. IEEE Communications Surveys & Tutorials 2021, 23, 2248–2294.
[54] Zhou, A.; Li, Z.; Shen, Y. Anomaly detection of CAN bus messages using a deep neural network for autonomous vehicles. Applied Sciences 2019, 9, 3174.
[55] Hossain, M.D.; Inoue, H.; Ochiai, H.; Fall, D.; Kadobayashi, Y. LSTM-based intrusion detection system for in-vehicle can bus communications. IEEE Access 2020, 8, 185489–185502.
[56] Andreica, T.; Curiac, C.D.; Jichici, C.; Groza, B. Android Head Units vs. In-Vehicle ECUs: Performance Assessment for Deploying In-Vehicle Intrusion Detection Systems for the CAN Bus. IEEE Access 2022, 10, 95161–95178.
[57] Kim, H.; Jeong, Y.; Choi, W.; Lee, D.H.; Jo, H.J. Efficient ECU Analysis Technology Through Structure-Aware CAN Fuzzing. IEEE Access 2022, 10, 23259–23271.
[58] Aldhyani, T.H.; Alkahtani, H. Attacks to automatous vehicles: a deep learning algorithm for cybersecurity. Sensors 2022, 22, 360.
[59] Islam, R.; Devnath, M.K.; Samad, M.D.; Al Kadry, S.M.J. GGNB: Graph-based Gaussian naive Bayes intrusion detection system for CAN bus. Vehicular Communications 2022, 33, 100442.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
Thanks to the authors' modifications the article can now be accepted for publication
Author Response
Reviewer #A, Comment #1: Thanks to the authors' modifications the article can now be accepted for publication
Author response: We are grateful to the reviewer for the previous comments that helped us improve our work.
Reviewer 3 Report
I appreciate the efforts of the authors but I suggest them to improve the results and discussion section again. If it is the noval idea then you can compare with respect to various other aspects like features wise, results wise, etc.
Further the comparison portion should be in result section So, add the table 1 in results section and remove it from introduction.
Author Response
Reviewer#B
Reviewer #B, Concern #1: I appreciate the efforts of the authors but I suggest them to improve the results and discussion section again. If it is the noval idea then you can compare with respect to various other aspects like features wise, results wise, etc.
Author response: We thank the reviewer for the suggestion. We now provide and discuss the distributions of vehicle speed and engine speed from model output signals, vehicle trace signals and their difference as separate bins on the X axis with the percentage shown on the Y axis. The following text was added under the “Statistical comparison” part from the results and discussion section:
“To provide a comprehensive evaluation for the computed mean difference, we show several plots with the distribution of vehicle speed and engine speed signal values from the model output, vehicle trace and the difference between them. The distribution of the vehicle speed from the model is shown in Figure 17 (i) with more than 20% of values in each of the following intervals, i.e., [0–20 km/h], [100–120 km/h] and [120–140 km/h]. The distribution of the engine speed from the model is shown in Figure 17 (ii) with 30% of the values in the [3240–3780 rpm] range and more than 20% of the values in the [540–1080 rpm] and [2700–3240 rpm] ranges. The vehicle speed from the vehicle trace has more than 30% of the values in the [132 – 154 km/h] range, i.e., highway, and around 25% of values in the [44–66 km/h] range, i.e., while on local roads, as shown in Figure 17 (iii). The engine speed was in the [2800-3500 rpm] range for more than 40% of occurrences in the vehicle trace and around 25% within the [1400-2100 rpm] range as illustrated in Figure 17 (iv). The distribution for the vehicle speed difference is shown in in Figure 17 (v) with 46% of values in the [0–20 km/h] interval and 80% of values below in the [0–40 km/h] interval while the distribution for the engine speed difference is shown in Figure 17 (vi) with 60% of values in the [0–560 rpm] interval and 83% of values in the [0–1120 rpm] interval. Numerical data which contains the bin width and the bin percentages for each of the 7 bins from the distributions is presented in Table 2.”
Reviewer #B, Concern #2: Further the comparison portion should be in result section So, add the table 1 in results section and remove it from introduction.
Author response: We thank the reviewer for the suggestion. We moved the comparison table in the results section.
Author Response File: Author Response.pdf
