3.2. Penetration Testing Process in ICVTest
Providing a concise summary of the vehicle penetration testing process tailored for ICVs holds significant importance in standardizing test procedures. Based on the NIST three-stage penetration testing specification [
31], ICVTest outlines six distinct steps for conducting VPT, as illustrated in
Figure 1.
- 1.
Scope Determination
The architecture of ICVs is intricate. Without a precise test scope, testers cannot accurately define the boundaries of the test, which may impact normal vehicle functions and cloud platform services during testing. Clarifying the test scope resolves the issue of unclear test boundaries during the testing process, making it easier for testers to comprehend and establish these boundaries.
- 2.
Information Collection
During the information collection stage, testers gather as much relevant information as possible about the TOT with proper authorization. This includes obtaining network addresses, chip details, communication protocols, interfaces, etc. Valuable public information can be sourced from official websites, promotional pages, and news reports. Testers can also disassemble the test target under authorized conditions to acquire crucial information such as hardware interfaces and chip models. The process of gathering information is iterative throughout penetration testing. As testing progresses further, more abundant and applicable data becomes available.
- 3.
Threat Analysis
The objective of penetration testing is to comprehensively evaluate the TOT’s cybersecurity from an attacker’s perspective. The primary challenge lies in determining what to test, which can be addressed through threat analysis. Threat analysis serves as a reference for developing a robust penetration testing scheme.
- 4.
Testing Scheme Development
Testers employ techniques such as attack trees to model potential threats faced by the TOT and identify its specific cybersecurity risks. The resulting penetration testing scheme meticulously covers these identified threats, ensuring that they are individually tested to prevent their evolution into actual vulnerabilities and mitigate serious cybersecurity risks.
- 5.
Testing Execution
Once the testing scheme is determined, the tester selects the appropriate testing technology to conduct penetration testing based on the established scheme.
- 6.
Risk Assessment
Risk assessment evaluates the severity of cybersecurity vulnerabilities by considering both the likelihood of successful attacks and their potential impact. The probability of a cybersecurity threat transforming into a vulnerability depends on factors such as attack sophistication, technical expertise of personnel involved, complexity of attack equipment, duration of attack, and economic costs incurred. The harm caused by an attack can manifest in various ways including vehicle function failure, personal injury, property loss, and privacy breaches.
The scope determination is employed to restrict the testing scope and prevent testers from conducting unauthorized tests. Information collection serves as a crucial activity for comprehending the TOT, laying the groundwork for threat analysis. Threat analysis constructs a threat model of the TOT to assist testers in elucidating its cyber security risks. Testing scheme development generates specific test cases based on the cyber security threats encountered by the TOT. Testers strictly adhere to the test scheme during execution. Risk assessment scrutinizes test results to aid designers in devising solutions. These six activities synergistically ensure standardization of VPT processes.
3.3. Penetration Testing Case Set in ICVTest
From the perspective of the in-vehicle bus, cybersecurity attacks on ICVs can be categorized into two groups: accessible bus cybersecurity attacks and inaccessible bus cybersecurity attacks. Accessible bus cybersecurity attacks have the potential to infiltrate the bus network and target other ECUs within it. On the other hand, inaccessible bus cybersecurity attacks focus on targets that are not connected to the bus network. Even if these targets are compromised, they cannot access or pose a threat to other ECUs within the same network. In scenarios where vehicle dismantling is not involved, potential entry points for accessible bus cybersecurity attacks include:
The second on-board diagnostics (OBD-II) interface serves as a crucial interface for ICVs to offer diagnostic services and presents an accessible approach for daily maintenance and ECU programming [
32]. Various handheld scanning tools have been developed by automobile manufacturers and have been equipped with dedicated software that enables access to the in-vehicle bus network through the OBD-II interface for querying or programming ECUs within the in-vehicle network. However, if these tools fall under malicious control, unauthorized manipulation of ECU configuration becomes possible.
- 2.
Sensors
ICVs are equipped with a diverse range of sensors to perceive both the surrounding environment and their own state, encompassing tire pressure monitoring sensors, global positioning systems, and ultrasonic sensors, among others. These sensors collaborate with numerous ECUs to form an in-vehicle network. Consequently, malevolent attackers can exploit vulnerabilities within these sensors as a means to infiltrate the in-vehicle network.
- 3.
Infotainment system
ICVs offer infotainment functionalities such as navigation and multimedia playback. The ECU responsible for these functions exhibits a complex architecture, encompassing numerous physical and wireless communication interfaces, thereby expanding the potential attack surface. Once an adversary successfully compromises the infotainment system, they can exploit it to further infiltrate the in-vehicle network.
- 4.
Wireless Communication
Wireless communication units, such as Wi-Fi, Bluetooth, radio frequency identification (RFID), tire pressure monitoring system (TPMS), and dedicated short range communication (DSRC), can be directly connected to ECUs like infotainment systems and telematics boxes (T-Box). Attackers may exploit these ECUs through wireless access points to launch further attacks on the in-vehicle bus network.
The objective of cybersecurity attacks on ICVs is to manipulate the vehicular behaviors by attacking the ECU. Cybersecurity attacks on ICVs typically consist of two stages. In the first stage, when disassembling the vehicle is not feasible, attackers can only gain access to externally exposed interfaces provided by the vehicle. This includes wireless communication interfaces such as Wi-Fi, Bluetooth, and cellular networks, as well as physical interfaces like OBD-II and universal serial bus (USB). If any of these in-vehicle communication modules possess a cybersecurity vulnerability, an attacker can exploit it to target the specific ECU through these mentioned interfaces. Subsequently, in the second stage, once an ECU has been compromised successfully, it serves as a gateway for attackers to infiltrate and control other ECUs within the in-vehicle network.
The attack path in ICVs encompasses various nodes, including cloud platforms, mobile phone applications (APPs), sensors, key ECUs, as well as physical and wireless communication channels facilitating node-to-node communication. The nodes involved in the attack path of ICVs are abstracted into 10 fields within ICVTest: hardware boards, ECU firmware, ECU operating systems, in-vehicle buses, network communications, cloud platforms, mobile devices, sensors, and private data. ICVs may face cybersecurity threats across 10 fields depicted in
Figure 2. Within the vehicle, the on-board ECU consists of a hardware board with firmware or an operating system running on it. Different ECUs are interconnected through the vehicle bus to form an in-vehicle network. Outside the vehicle, sensors enable the automobile to perceive its surrounding environment and internal state. Sensor data is transmitted via the bus system to corresponding ECUs for analysis purposes. The radio communication module serves as a gateway between off-vehicle networks and the in-vehicle network. In addition to connecting vehicles with cloud service platforms, communication technology also facilitates connectivity with mobile terminal devices such as smartphones, introducing additional security risks for ICVs. Furthermore, the advent of ICVs brings forth significant concerns regarding data privacy while providing services.
Based on the threats identified in
Figure 2, ICVTest abstracts the vehicle cybersecurity test case set into 10 fields. The common cybersecurity threats that need to be verified by security tests are summarized in each test case set.
- 1.
Hardware Security Test Case Set
Hardware security threats encompass printed circuit board (PCB) vulnerabilities, processor chip weaknesses, memory chip susceptibilities, hardware debugging interface risks, and on-board bus vulnerabilities. In the case of PCB in ECU, there is potential for information leakage regarding integrated circuit (IC) chip models, interface details, and bus protocol. The processor chip may inadvertently disclose side channel information such as electromagnetic signals, timing data, and power consumption patterns [
33] thereby facilitating attackers to launch timing attacks [
34], power consumption attacks [
35], and electromagnetic attacks [
36]. Additionally, fault injection techniques [
37] pose significant security risks to processor chips. Attacks like voltage fault injection [
38], electromagnetic fault injection [
39], and laser fault injection [
40] can alter the operational logic of the processor. Data storage chips are also susceptible to data residue threats. For instance, flash memory can be read by a programmer. Furthermore, hardware debugging interfaces like joint test action group (JTAG), serial wire debug (SWD), and USB provide attackers with opportunities to access internal storage data of the system on chip (SoC). Attackers may also monitor buses such as serial peripheral interface (SPI) or inter-integrated circuit (I2C), leading to potential data leakage risks.
- 2.
Firmware Security Test Case Set
Private information, such as passwords, keys, network resource addresses, usernames, and email addresses, may be stored in plain text within the firmware. Malicious attackers can decipher the operational logic of the TOT through reverse engineering techniques. Furthermore, these attackers can even assess whether the target firmware possesses cybersecurity vulnerabilities within real-world operating environments using dynamic analysis.
- 3.
In-Vehicle Bus Security Test Case Set
ICVs employ various in-vehicle bus protocols, including CAN bus, FlexRay bus, local interconnection network (LIN) bus, media oriented system transport (MOST) bus, and in-vehicle ethernet bus. On one hand, certain in-vehicle bus protocols lack essential security mechanisms such as encryption, authentication, and integrity to meet the specific requirements of low-latency communication within vehicles. On the other hand, some application layer protocols for in-vehicle buses allow external individuals to access and control the vehicle’s functionalities. Examples include unified diagnostic services (UDS) and scalable service-oriented middleware over IP (SOME/IP). The absence of adequate security measures renders these in-vehicle buses vulnerable to potential attackers who can gain control over them, resulting in severe consequences.
- 4.
System Security Test Case Set
A secure ICV operating system necessitates stringent control over external entities’ access to resources within the system. Ensuring the security of the operating system entails not only safeguarding data through mechanisms like authority access control, encryption, and integrity verification during design but also mitigating cybersecurity risks arising from design and implementation defects via a series of configurations.
- 5.
Radio Security Test Case Set
ICVs are equipped with various wireless communication modules that operate on different radio frequencies to cater to diverse communication needs in scenarios such as Wi-Fi, Bluetooth, near field communication (NFC), RFID, cellular networks, and dedicated short-range communications (DSRC). However, the inherent openness of the wireless communication channel introduces vulnerabilities for ICVs. If the traffic transmitted over the radio channel is not encrypted, malicious attackers can intercept and manipulate user secrets by collecting and analyzing radio signals.
- 6.
Network Security Test Case Set
In contrast to radio security, network security primarily focuses on ensuring the security of the TCP/IP protocol stack at the network level, while the former concentrates on physical layer and link layer security. Malicious attackers have the potential to intercept sensitive data transmitted over a network, and they can also manipulate legitimate data obtained through replay attacks in order to deceive recipients. Furthermore, attackers may even exploit communication between entities to carry out man-in-the-middle attacks. Once successfully hijacked, these attackers can eavesdrop and tamper with communication information exchanged among authorized users.
- 7.
Web Security Test Case Set
The interaction between ICVs and the cloud platform may introduce cybersecurity risks to the ICVs. In the event of a compromise of the cloud platform by attackers, not only can private data such as user information be leaked, but also the ICVs can be invaded through remote wireless networks. Web applications, being crucial components of the cloud platform, face threats from both clients and servers. Client-side threats primarily encompass browser vulnerabilities, cross-site scripting (XSS) attacks, cross-site request forgery (CSRF) attacks, clickjacking exploits, and hypertext markup language (HTML) vulnerabilities. On the other hand, server-side threats mainly involve injection attacks, file upload vulnerabilities, authentication and session management weaknesses, access control issues, web framework susceptibilities, distributed denial-of-service (DDoS) attacks, and improper server configurations.
- 8.
APP Security Test Case Set
In the context of remote vehicle control scenarios, applications can potentially serve as a gateway for attacks, posing significant cybersecurity risks to ICVs. The assets that require protection within these applications encompass client files, local storage data, application processes, runtime data, interactive interfaces, and network communication. Client files may inadvertently expose system logic or sensitive information or be subject to malicious tampering. Certain applications may store critical information such as bank cards, ID cards, contacts, and account passwords locally. Without robust security mechanisms in place, application data is highly susceptible to compromise. Upon launching the application, it requests resources from the system and initiates a process to execute its main logic. An attacker could exploit vulnerabilities by forcibly terminating or hijacking a process or injecting malicious data into it, thereby impeding normal program execution.
- 9.
Sensor Security Test Case Set
ICVs heavily rely on sensors for achieving autonomous driving, thereby introducing a broader attack surface and potential cybersecurity risks to vehicles. Various types of sensors are embedded in ICVs, including cameras, Lidar, ultrasonic radar, millimeter-wave radar, GPS, etc. The GPS signal tends to weaken after long-distance transmission, making it susceptible to interference from simulated GPS signals that can cause location deception. Cameras can be rendered ineffective by arrays of powerful light sources. Similarly, millimeter-wave radar can be disrupted by signals with similar waveforms.
- 10.
Privacy Security Test Case Set
In intelligent driving scenarios, vehicles need to periodically transmit their state information to enhance driving efficiency. In the absence of encryption, adversaries can exploit wireless monitoring technology to acquire the status of the target vehicle. Furthermore, services such as danger warning and personalized recommendation necessitate access to sensitive data including vehicle identity information, user habits, and web browsing records.
Test cases in ICVTest are not limited to specific modules and systems in ICVs, such as infotainment systems or automatic driving control systems. Instead, ICVTest abstracts the software and hardware components involved in in-vehicle systems into 10 fields. These test cases comprehensively cover all 10 fields of automotive software and hardware, providing sufficient guidance for testers to conduct detailed cybersecurity evaluation for ICVs. Testers are relieved from the need to customize a penetration testing scheme for each TOT. Instead, they can simply reuse relevant test cases based on the software and hardware architecture of the TOT. As a result of leveraging threat analysis results, testers only need to select the appropriate cybersecurity test cases to assess whether exploitable vulnerabilities exist within the TOT. ICVTest significantly standardizes penetration testing content, reduces its complexity, and enhances overall efficiency.
3.4. Cybersecurity Testing Platform for ICVTest
The implementation of a cybersecurity testing platform for ICVTest aims to facilitate penetration testers, particularly inexperienced novices, in effectively carrying out testing tasks. As depicted in
Figure 3, the platform is divided into four components: ICVTest infrastructure, ICVTest tool and guide manual, ICVTest agent, and ICVTest web application.
- 1.
ICVTest Infrastructure
ICVTest infrastructure cabinet is equipped with essential tools for hardware security testing including oscilloscopes, multimeters, logic analyzers, and programmers. It also facilitates communication through various interfaces such as JTAG, USB, in-vehicle ethernet, and CAN bus. Additionally, the cabinet ensures uninterrupted operation during unexpected power loss by utilizing UPS power supply. Furthermore, the integrated server hosts ICVTest tool, ICVTest agents, and ICVTest web applications.
- 2.
ICVTest Tool and Guide Manuals
Due to the complexity of ICVs, there are numerous test cases that cannot all be automated. Therefore, ICVTest offers three distinct types of test case. ICVTest provides automated test scripts for test cases with well-defined criteria and high automation potential, ensuring a low false positive rate. For test cases with ambiguous judgment criteria, ICVTest integrates testing tools and allows testers to make manual judgments based on test data. Additionally, for test cases requiring extensive human participation the ICVTest integrated guide manuals assist testers in conducting tests and provide examples for making judgments.
- 3.
ICVTest Agent
The ICVTest agent serves as a crucial link between the ICVTest web application, ICVTest infrastructure, and ICVTest tool, offering comprehensive support for driving both hardware and software to accomplish automated testing. Upon receiving instructions from the ICVTest web application, the testing agent invokes the local testing tool to execute tests while providing real-time reporting of processed data and results back to the ICVTest web application.
- 4.
ICVTest Web Application
The ICVTest web application facilitates interaction among testers, test supervisors, and project supervisors by integrating features such as data board, user management, tool management, project management, task management, and test case management. The web application primarily focuses on collaborative multi-person test management to effectively handle personnel, equipment, software, projects, tasks, tests, and reports involved in the testing process. The ICVTest scalability is ensured through flexible configuration or online editing of test cases.
In the test scenario, testers log into the ICVTest web application with test terminals and select applicable test cases from the test case database based on the threat analysis results within the scope of authorization to form a penetration test plan. Once physically connected to the vehicle under test, the selected test case is executed. The ICVTest agent transmits the web-based test instructions to the ICVTest infrastructure and displays the test data on the ICVTest web application. Upon completion of the penetration test, an automated report is generated by the platform for review by the design team in order to address any cybersecurity vulnerabilities identified in relation to the vehicle.