**4. Performance Evaluation**

To evaluate the proposed bit-level CAN bus reverse framework, this study implements it on an actual vehicle and obtains specific details of the vehicle CAN message data fields depicting the vehicle behavior for that vehicle. Using the reverse results, the accuracy of the algorithm is evaluated for practical applications based on the available DBC files [45]. In addition, this section evaluates the execution performance of the framework and compares the advantages of the algorithm over other reverse methods. Finally, the advantages of the algorithm in applications are discussed, and an example is given for reversing other vehicle messages when DBC files are not available.

### *4.1. Performance in Real Vehicle*

### 4.1.1. Device Description and Data Processing

For the evaluation a 2017 Japanese B-Class sedan was used, whose internal network implements the standard CAN protocol and whose functionality is representative. A DBC file for this model has been obtained, which is used as ground truth for the reverse framework evaluation. To better represent the vehicle behavior, sensors are placed on the body, steering wheel, brake pedal, gas pedal, gear knob, and wiper switch to collect the behavioral data of the vehicle components, which are structured as shown in Figure 9. The CAN data is collected through the OBD-II interface using the combination cable synchronously when collecting vehicle data. The collected CAN data is written to a log file using the upper computer program, containing the ID, type, length, data field, and timestamp of CAN messages. For accuracy evaluation, more than 3,661,000 consecutive CAN bus messages were collected, and more than 5,000,000 vehicle behavior sensor data were sequentially collected in the same period. The dataset (The dataset is partially open source and can be accessed at http://49.232.218.41:8000/data.zip accessed on 23 January 2022) is quantitatively described in Table 4, which describes the measurements and CAN data collected synchronously for each vehicle behavior.

**Figure 9.** Sensor Acquisition Setup: (**a**) Gear angle; (**b**) Steering wheel angle; (**c**) Brake pedal angle; (**d**) Gas pedal angle; (**e**) Wiper switch angle; (**f**) Vehicle speed.

**Table 4.** Number of vehicle behaviors and CAN messages.


By analyzing the collected CAN traces, the frequency distribution of the messages is shown in Figure 10. This result shows that the number of IDs collected from the test vehicle is 82, which means that there are 82 types of messages in the CAN network. For each type of CAN message, we analyze whether the data field of this CAN message changes and eliminate the messages with unchanged data fields. Based on the analysis and processing of CAN traces, the vehicle behavior data collected in Table 3 is resampled 82 times to obtain *Ysi*. A multiple linear regression model is built between *Ysi* and *Xi* according to the message filtering process.

**Figure 10.** CAN message frequency distribution.

### 4.1.2. Message Filter Results

The results of the multivariate linear regression of the collected continuous vehicle behavior with each type of message are shown in Figure 11. The *x*-axis is the determination coefficient *R*<sup>2</sup> of the multiple linear regression model, and the *y*-axis is the effective ID distribution.

For the linear regression results of vehicle speed and CAN trace, according to the threshold value 0.6, three types of messages can be filtered out that directly record vehicle speed information with IDs 0x202, 0x215, and 0x217, as shown in Figure 11a. In addition, in this result, there are some *R*<sup>2</sup> values close to the threshold, such as 0x130, 0x165, 0x167 and 0x200. This is because they may describe information such as RPM, throttle, etc., that correlate with the vehicle speed, which explains their larger *R*2. However, since these types of vehicle data cannot be collected by sensors, they cannot determine their exact meaning. As shown in Figure 11b, with a *R*<sup>2</sup> of 0.1 as the dividing line, the messages IDs strongly correlated with steering wheel angle are 0x086, 0x082, and 0x240. These messages may contain data describing steering wheel torque and steering rate in addition to the information directly representing steering angle. In the same way, messages related to the accelerator pedal are filtered out including messages with IDs 0x165, 0x167, 0x202, 0xFD, and 0x21F, with 0.2 as the divisor, as shown in Figure 11c. Messages with IDs 0x78, 0x202, and 0x165 are categorized as related to brake pedal angle with a threshold of 0.18 as shown in Figure 11d. The results of filtering information related to wiper switch and gear angle are shown in Figure 11e,f. With a threshold of 0.6, the message IDs related to the wiper are 0x9A, and the message IDs related to the gear are 0x165 and 0x228, respectively.

As can be seen from the results of the message filtering, the *R*<sup>2</sup> and threshold values for messages related to steering angle, acceleration, and brake pedal are generally small. This result is due to the slight variations in vehicle behavior when collecting these data. For example, the pedal is unlikely to be located at the lowest position when collecting the gas pedal angle while driving. In addition, the results for vehicle speed, gas pedal, and brake pedal show that a certain number of messages have an *R*<sup>2</sup> value that is below the threshold, but very close to it. Although these messages do not directly describe the state of the vehicle speed, gas pedal, and brake pedal, they do describe vehicle behavior correlated with the state. For example, the near-threshold telegrams in the throttle results describe the vehicle's speed, torque, and acceleration, among other things. However, since these messages do not directly describe the vehicle speed, they are classified as irrelevant messages by the threshold. Also, as shown in Figure 11e,f, the *R*<sup>2</sup> of the messages related to wiper and gears are clearly distinguished from others. Since the vehicle behavior (gear angle and wiper angle) data and the related CAN messages are all discrete, they can be clearly distinguished from the other messages when the linear regression modeling is performed.

**Figure 11.** *Cont*.

### 4.1.3. Bit-Level Reverse Results

By analyzing the linear regression result of the filtered messages, it is possible to reverse the portrayal of the vehicle behavior by the individual bits of the message.

The reverse result for the speed-related messages is shown in Figure 12. There is a two-fold relationship between the messages with IDs 0x202, 0x215, and 0x217 and the *β* of the vehicle speed. As shown in Figure 12a, bits 34 to 42 in the message with ID 0x202 indicate the vehicle's speed, arranged in the format of Motorola. For the message with ID 0x215, according to Figure 12b, bits 0 to 12, bits 16 to 28, bits 32 to 44, and bits 48 to 60 represent the vehicle speed information and the arrangemen<sup>t</sup> format is Motorola. The value for the *β* with ID 0x217 is shown in Figure 12c, and the bits describing the vehicle speed are 34 to 46, and the arrangemen<sup>t</sup> format is also Motorola.

The reverse results of the steering-related messages are shown in Figure 13. Bits 22 to 31 in the message with ID 0x82 describe the steering angle arranged in Motorola. In the corresponding message with 0x86, the steering angle is specified in bits 3 to 13 and 28 to 36, respectively. The message with ID 0x240 does not describe the steering angle directly, but because its *R*<sup>2</sup> is greater than the threshold, it is related to the change in steering.





**Figure 12.** Speed-related messages reverse result: (**a**) ID 0x202 reverse result; (**b**) ID 0x215 reverse result; (**c**) ID 0x217 reverse result.



**Figure 13.** Steer-related messages reverse result: (**a**) ID 0x082 reverse result; (**b**) ID 0x086 reverse result; (**c**) ID 0x240 reverse result.

The results of the throttle-related message are shown in Figure 14. There is an approximate relationship of 2 times in the *β* corresponding to 0xFD, 0x167, and 0x202 in the results, so based on the *β*, we find that bits 49 to 55 in the message with 0xFD describe the gas pedal angle. As shown in Figure 14b, in the message whose ID is 0x167, bits 0 to 7 portray the angle of the gas pedal. The angle of the gas pedal in 0x202 is represented in bits 39 to 47. For the messages 0x165 and 0x21F, there is no 2x relationship in *β*. But the bits 40 to 43 of 0x21F indicate the rate of change of the gas pedal angle as shown in Figure 14d. For 0x165, the gas pedal angle is converted to a discrete state using a threshold: accelerated or not. The result of the discrete value is shown in Figure 14e, from which it can be seen that bit 29, and bits 22 to 26 of ID 0x165 describe whether the gas pedal is activated or not.


**Figure 14.** Gas-related messages reverse result: (**a**) ID 0x0FD reverse result; (**b**) ID 0x167 reverse result; (**c**) ID 0x202 reverse result; (**d**) ID 0x21F reverse result with gas angle change rate; (**e**) ID 0x165 reverse result with discrete state.

The results of the bit reverse for the brakes are shown in Figure 15. Based on the *β* of 0x78, the bits representing the brake pedal are bits 32 to 37, arranged as Motorola. Since there are no significant features in the *β* of 0x202 and 0x165, the linear regression *β* of these two types of IDs with discrete states of the brake pedal (braked or not) was calculated using the same method. The results show that in 0x165, bits 0, 1, 3, 7, and 8 indicate whether the vehicle's state is accelerated or not. For the message with 0x202 as ID, the results show that it does not describe the braking behavior but only the vehicle behavior with respect to braking.





**Figure 15.** Brake-related messages reverse result: (**a**) ID 0x078 reverse result; (**b**) ID 0x165 reverse result;(**c**)ID0x202reverseresult.

The reverse results for the gears are shown in Figure 16. Since the gear behavior data is discrete, it is evident from the *β* that the message with 0x228 describes the gear information in bits 3, 5 to 7, 10 and 35 to 39, and 0x165 describes the gear in bits 51 to 54. The reverse result of the wipers is shown in Figure 17. The data describing the wiper speed in 0x9A are bits 37 to 38 and bit 50, And the specific reverse results are shown in Table 5.


**Figure 16.** Gear-related messages reverse result: (**a**) ID 0x228 reverse result; (**b**) ID 0x165 reverse result.


**Figure 17.** Wiper-related messages reverse result.

**Table 5.** Results for gears and wipers of bit-level reverse.

