Byzantine-Robust Multimodal Federated Learning Framework for Intelligent Connected Vehicle

Abstract

In the rapidly advancing domain of Intelligent Connected Vehicles (ICVs), multimodal Federated Learning (FL) presents a powerful methodology to harness diverse data sources, such as sensors, cameras, and Vehicle-to-Everything (V2X) communications, without compromising data privacy. Despite its potential, the presence of Byzantine adversaries–malicious participants who contribute incorrect or misleading updates–poses a significant challenge to the robustness and reliability of the FL process. This paper proposes a Byzantine-robust multimodal FL framework specifically designed for ICVs. Our framework integrates a robust aggregation mechanism to mitigate the influence of adversarial updates, a multimodal fusion strategy to effectively manage and combine heterogeneous input data, and a global optimization objective that accommodates the presence of Byzantine clients. The theoretical foundation of the framework is established through formal definitions and equations, demonstrating its ability to maintain reliable and accurate learning outcomes despite adversarial disruptions. Extensive experiments highlight the framework’s efficacy in preserving model performance and resilience in real-world ICV environments.

1. Introduction

The advent of Intelligent Connected Vehicles (ICVs) marks a transformative shift in transportation technology [1], promising to revolutionize road safety [2], traffic efficiency [3], and the overall driving experience [4]. ICVs leverage an intricate network of sensors, including high-resolution cameras, Light Detection and Ranging (LiDAR) systems [5], millimeter-wave radars [6], and Global Positioning System (GPS) receivers, to create a comprehensive understanding of their environment [7]. This sensor fusion, combined with advanced communication technologies such as Vehicle-to-Everything (V2X) protocols [8], enables ICVs to make informed decisions [9], navigate complex traffic scenarios [10], and interact seamlessly with other vehicles and infrastructure [11].

However, the proliferation of ICVs introduces unprecedented challenges in data management and processing [12]. The sheer volume of data generated by a single vehicle–estimated to be up to 25 gigabytes per hour–multiplied across millions of vehicles, creates a data deluge that traditional centralized computing paradigms struggle to handle efficiently [13,14,15]. Moreover, these data often contain sensitive information about vehicle locations, driving patterns, and potentially even biometric data of drivers, raising significant privacy concerns [16].

Federated Learning (FL) [17] has emerged as a promising solution to address these challenges. FL enables collaborative machine learning without the need for centralized data storage or processing [18]. In the context of ICVs, this means that vehicles can collectively train sophisticated models for tasks such as object detection [19], traffic prediction [14], and autonomous navigation [20,21], while keeping their raw sensor data securely on-board. This decentralized approach not only preserves privacy but also significantly reduces the bandwidth required for model training, as only model updates, rather than raw data, are transmitted [22].

Despite its potential, the application of FL in ICV scenarios faces several critical challenges that demand innovative solutions:

• Multimodal Data Integration. ICVs generate a diverse array of data types from various sensors [23,24]. Each sensor modality provides unique and complementary information. For example, cameras provide rich visual data that are critical for object recognition and scene understanding, while LiDAR provides precise depth information and 3D point clouds for accurate distance measurement and object localization [19]. These different modality types of data are extremely important for the proper operation of ICVs. Effectively fusing these heterogeneous data sources while maintaining their privacy-preserving nature in a FL setup is a complex challenge [25]. Traditional centralized fusion techniques are not directly applicable, necessitating novel approaches that can operate on distributed, privacy-sensitive data.
• Byzantine Attacks. In a distributed learning environment like FL, the system is vulnerable to Byzantine attacks, where malicious participants or compromised vehicles may inject false or manipulated data or model updates [26,27,28]. These attacks can take many forms, such as data poisoning [26,28], where the adversary injects crafted malicious samples into local training data, or model poisoning [29,30,31], where the adversary sends malicious model updates to corrupt the global model. The consequences of such attacks in an ICV context could be severe, potentially leading to erroneous object detection or navigation decisions that compromise road safety [32,33]. Developing robust defense mechanisms that can detect and mitigate these attacks without compromising the efficiency of the federated learning process is crucial.
• Communication Constraints. The mobility of vehicles presents unique challenges to the FL process, such as vehicles may experience periods of disconnection or weak signal strength, especially in rural or underground areas [7,9,34]. In addition, the network capacity available to vehicles may fluctuate widely depending on location, and network congestion and frequent high-bandwidth communications can put stress on the vehicle’s power system, especially in electric vehicles [5]. Designing a communication-efficient federated learning protocol that can adapt to these dynamic conditions while ensuring timely and effective model updates is essential.

To address these challenges, we propose a Byzantine-robust multimodal federated learning framework specifically designed for intelligent connected vehicles. Specifically, we first design a novel multimodal fusion architecture that can effectively integrate various sensor data while preserving privacy. The architecture adopts a hierarchical approach to first locally fuse data within each modality and then use a privacy-preserving cross-modal attention mechanism to integrate information across modalities. In addition, we design a Byzantine-robust aggregation algorithm based on gradient compression that can detect and mitigate the impact of malicious participants in the FL process while maintaining high communication efficiency. Our approach combines a statistical analysis of model updates with a reputation system that tracks the historical reliability of participants. Our framework not only addresses the immediate challenges facing intelligent connected vehicles, but also lays the foundation for building a scalable, secure, and efficient federated learning ecosystem in the broader context of intelligent transportation systems.

The contributions of this paper are listed as follows:

(1)

We develop a novel Byzantine-robust aggregation technique based on gradient compression, enhancing the resilience of federated learning against adversarial nodes.

(2)

We introduce an advanced cross-node multimodal alignment and fusion technique that efficiently combines data from diverse sensors to improve model performance in ICVs.

(3)

We implement top-k gradient compression to improve communication efficiency. This reduces the communication overhead between nodes and the central server, making the framework suitable for large-scale deployment.

(4)

We conducted extensive experiments on three public datasets for the proposed framework and evaluated prior work to demonstrate the advantages of the proposed framework. Our framework can achieve a better cost–utility trade-off.

The remainder of this paper is organized as follows: Section 2 provides a comprehensive review of related work in federated learning, multimodal fusion techniques, Byzantine-robust algorithms, and their applications in intelligent transportation systems. Section 3 presents our problem definition in detail, elaborating on the challenges and constraints. Section 4 presents our proposed framework in detail, including the cross-node multimodal alignment and fusion method, gradient compression-based Byzantine aggregation algorithm, and time complexity analysis. Section 5 discusses the results of our experiments, providing a comparative analysis with existing methods and an ablation study to quantify the impact of each component of our framework. Finally, Section 6 concludes the paper by summarizing our contributions, discussing the limitations of our approach, and outlining promising directions for future research in this rapidly evolving field.

2. Related Work

This section provides an overview of the existing literature relevant to our proposed Byzantine-robust multimodal FL framework for ICVs. We organize the related work into four key areas: federated learning in vehicular networks, multimodal learning for ICVs, Byzantine-robust federated learning, and communication-efficient federated learning.

2.1. Federated Learning in Vehicular Networks

FL has gained significant attention in the context of vehicular networks due to its ability to leverage distributed data while preserving privacy [35,36]. McMahan et al. [17] introduced the seminal FedAvg algorithm, which forms the basis for many federated learning approaches. In the vehicular domain, Du et al. [25] proposed a blockchain-based FL framework for securing data sharing in the Internet of Vehicles (IoVs). Their approach addresses trust issues in data sharing but does not consider multimodal data or Byzantine attacks. Samarakoon et al. [35] developed a FL approach for joint power control and resource allocation in vehicular networks. While their work demonstrates the potential of FL in optimizing network performance, it focuses on network-level optimization rather than perception and decision-making tasks. Lu et al. [37] introduced a FL framework for cooperative sensing in connected and autonomous vehicles. Their approach shows promise in improving sensing accuracy, but it does not address the challenges of multimodal data fusion or Byzantine robustness.

2.2. Multimodal Learning for ICVs

Multimodal learning is crucial for ICVs to effectively integrate data from various sensors [38]. Feng et al. [39] proposed a deep multimodal fusion framework for object detection in autonomous driving, combining data from cameras and LiDAR. However, their approach assumes centralized data processing, which is not suitable for privacy-preserving federated learning scenarios. Caesar et al. [23] developed a multimodal attention network for sensor fusion in autonomous vehicles. While their work demonstrates improved perception accuracy, it does not consider the distributed nature of data in federated learning settings. In the context of FL, Liu et al. [40] proposed a multimodal federated learning framework for medical image analysis. Although their work addresses privacy concerns in multimodal learning, it is not tailored to the specific challenges of vehicular networks and does not consider Byzantine attacks.

2.3. Byzantine-Robust Federated Learning

Byzantine robustness is critical for ensuring the reliability of FL systems, especially in safety-critical applications like ICVs [28,31,41]. Yin et al. [42] introduced the Byzantine-robust distributed learning algorithm, which can tolerate up to a certain fraction of Byzantine workers. However, their approach assumes a centralized parameter server, which may not be suitable for fully decentralized vehicular networks. Blanchard et al. [43] proposed the Krum algorithm for Byzantine-robust aggregation in FL [43]. While Krum provides theoretical guarantees against Byzantine attacks, it may not be computationally efficient for the large-scale and time-sensitive nature of ICV applications. More recently, Fung et al. [26] developed FoolsGold, a Byzantine-robust federated learning system that can defend against Sybil attacks [28,33]. Their approach shows promise in identifying and mitigating the impact of malicious clients, but it does not consider the multimodal nature of ICV data.

2.4. Communication-Efficient Federated Learning

Communication efficiency is paramount in vehicular networks due to bandwidth limitations and the mobile nature of vehicles [44]. Konečný et al. [34] proposed structured updates and sketched updates to reduce the communication cost in FL. While their methods show significant bandwidth savings, they do not address the specific challenges of vehicular networks. Sattler et al. [35] introduced Sparse Ternary Compression (STC) for communication-efficient federated learning. STC achieves high compression rates while maintaining model accuracy, but it does not consider the dynamic nature of vehicular network conditions. In the context of vehicular networks, Ye et al. [45] proposed an efficient federated learning scheme with adaptive model aggregation. Their approach considers vehicle mobility and network conditions but does not address multimodal data fusion or Byzantine robustness.

While the existing literature has made significant strides in various aspects of federated learning for vehicular networks, there remains a critical gap in addressing the combined challenges of multimodal data integration, Byzantine robustness, and communication efficiency in a unified framework for ICVs. By addressing these challenges simultaneously, our framework aims to provide a comprehensive solution for secure, efficient, and reliable federated learning in Intelligent Connected Vehicle systems.

3. Problem Definition

In this section, we formally define the problem of Byzantine-robust multimodal federated learning for ICVs. We outline the system model, the objectives of our framework, and the specific challenges we aim to address.

3.1. System Model

Consider a network of K ICVs, denoted as $V=\{v_1,v_2,\dots ,v_k\}$. Each vehicle $v_k$ is equipped with a set of M sensors $S=\{s_1,s_2,\dots ,s_m\}$, where each sensor captures a different modality of data (e.g., camera images, LiDAR point clouds, radar signals, GPS coordinates). Each vehicle $v_k$ maintains its local dataset ${\mathcal{D}}_k={\left\{({\mathbf{x}}_i,y_i)\right\}}_{i=1}^{n_k}$, where ${\mathbf{x}}_i$ represents the multimodal input data and $y_i$ the corresponding labels. As shown in Figure 1, the goal is to collaboratively train a global model $\mathbf{w}$ that can accurately perform a given task (e.g., object detection, traffic prediction) by aggregating local updates $\Delta {\mathbf{w}}_i$ from each vehicle without sharing raw data. The federated learning process can be formalized as follows:

$${\mathbf{w}}_{t+1}={\mathbf{w}}_t+\eta \sum _{i=1}^K{\alpha }_i\Delta {\mathbf{w}}_i^t,$$

(1)

where ${\mathbf{w}}_t$ is the global model at iteration t, $\eta$ is the learning rate, ${\alpha }_i$ is the weight assigned to the i-th vehicle, and $\Delta {\mathbf{w}}_i^t$ is the local model update from vehicle i at iteration t.

Multimodal Fusion. Each vehicle processes multimodal data, necessitating an effective fusion strategy to handle different types of input data. Let ${\mathbf{x}}_i$ denote the multimodal input data from vehicle $v_k$, comprising m modalities ${\mathbf{x}}_i=\{{\mathbf{x}}_i^1,{\mathbf{x}}_i^2,\dots ,{\mathbf{x}}_i^m\}$. The local model $f_i({\mathbf{x}}_i;{\mathbf{w}}_i)$ integrates these modalities to produce predictions ${\widehat{y}}_i=f_i({\mathbf{x}}_i;{\mathbf{w}}_i)$. The multimodal fusion within each vehicle can be formulated as follows:

$${\mathbf{h}}_i=\mathcal{F}({\mathbf{x}}_i^1,{\mathbf{x}}_i^2,\dots ,{\mathbf{x}}_i^m)$$

(2)

where $\mathcal{F}$ denotes the fusion function that combines the features from different modalities into a unified representation ${\mathbf{h}}_i$.

Byzantine Adversaries. In this setup, a subset of vehicles may act as Byzantine adversaries, sending arbitrary or malicious updates $\Delta {\mathbf{w}}_i^{t,\mathrm{adv}}$. These adversarial updates can significantly deteriorate the performance of the global model. Let $\mathcal{B}\subseteq \{1,2,\dots ,N\}$ denote the set of Byzantine vehicles, with $\left|\mathcal{B}\right|=b$. To mitigate the influence of Byzantine adversaries, we introduce a Byzantine-robust aggregation mechanism. The objective is to aggregate the local updates in a way that minimizes the impact of adversarial updates. Formally, the robust aggregation function $\mathcal{A}$ is defined as follows:

$${\mathbf{w}}_{t+1}={\mathbf{w}}_t+\eta \mathcal{A}\left({\{\Delta {\mathbf{w}}_i^t\}}_{i=1}^N\right).$$

(3)

The aggregation function $\mathcal{A}$ should satisfy the following properties: (1) Resilience: It should be resilient to at most b Byzantine adversaries. (2) Accuracy: It should ensure that the aggregated update is close to the mean of the non-adversarial updates.

Objective Function. The overall objective is to minimize the global loss function $\mathcal{L}\left(\mathbf{w}\right)$ over all vehicles, accounting for the presence of Byzantine adversaries:

$$\underset{\mathbf{w}}{min}\mathcal{L}\left(\mathbf{w}\right)=\sum _{i\notin \mathcal{B}}{\alpha }_i{\mathcal{L}}_i\left(\mathbf{w}\right),$$

(4)

where ${\mathcal{L}}_i\left(\mathbf{w}\right)$ is the local loss function for vehicle i. Thus, the proposed Byzantine-robust multimodal federated learning framework aims to ensure robust and efficient collaborative learning among ICVs, leveraging diverse data modalities while safeguarding against adversarial disruptions.

3.2. Challenges and Constraints

Implementing the above robust framework for ICVs poses several challenges and limitations that must be addressed to ensure the effectiveness and reliability of the system.

• Byzantine Robustness. Ensuring robustness against Byzantine adversaries is a significant challenge. Malicious nodes can send faulty updates that can severely degrade the performance of the global model. Designing efficient and effective robust aggregation methods to mitigate these attacks while maintaining high model performance is complex.
• Communication Overhead. FL inherently involves substantial communication between nodes and the central server. The gradient compression technique helps reduce this overhead, but finding the optimal balance between compression rate and model accuracy is crucial. Excessive compression can lead to the loss of important information, while insufficient compression can cause excessive communication delays.
• Heterogeneous Data. Multimodal datasets from different vehicles may vary in quality, resolution, and format. Ensuring effective data fusion across these heterogeneous sources without losing critical information is a key constraint.

Addressing these challenges and constraints requires continuous innovation and rigorous testing to ensure the framework’s reliability, efficiency, and security in real-world ICV applications.

4. Our Approach

This section presents the Byzantine-robust multimodal federated learning framework for ICVs. The framework includes a Byzantine aggregation algorithm based on gradient compression, a modality alignment fusion method across nodes, and an objective function designed to enhance learning performance despite adversarial interference, as shown in Figure 1.

4.1. Cross-Node Multimodal Alignment and Fusion

The cross-node multimodal alignment and fusion technique is designed to handle the diverse data modalities from different ICVs and align them into a consistent latent space for effective fusion. This technique ensures that the multimodal features from different nodes are comparable and can be effectively aggregated for federated learning.

Local Feature Extraction. Each vehicle $v_i$ extracts features from its local multimodal data using dedicated subnetworks. Let ${\mathbf{x}}_i=\{{\mathbf{x}}_i^1,{\mathbf{x}}_i^2,\dots ,{\mathbf{x}}_i^m\}$ represent the input data from m modalities. For each modality j, a feature extraction subnetwork ${\varphi }_j$ is used to obtain the local features:

$${\mathbf{h}}_i^j={\varphi }_j\left({\mathbf{x}}_i^j\right)$$

(5)

where ${\mathbf{h}}_i^j$ denotes the extracted feature vector for modality j from vehicle $v_i$.

Discussion. To effectively manage high-dimensional multimodal data in the proposed framework for ICVs, a combination of dimensionality reduction techniques, feature extraction, and multimodal fusion strategies is employed. Methods like the above Local Feature Extraction reduce the dimensionality of data from various sensors and cameras while preserving essential features. Sparse representations and low-rank approximations further minimize the complexity of high-dimensional inputs. Additionally, adaptive fusion strategies integrate the reduced representations of different modalities, allowing for efficient information aggregation across diverse data sources. This ensures that the framework can handle the high-dimensionality of multimodal data without overwhelming computational resources while maintaining robustness against adversarial attacks.

Global Modality Alignment. To ensure that the features from different nodes are aligned into a common latent space, we employ a modality alignment network ${\mathcal{A}}_j$ for each modality. This network aligns the local features ${\mathbf{h}}_i^j$ to a global latent space:

$${\tilde{\mathbf{h}}}_i^j={\mathcal{A}}_j\left({\mathbf{h}}_i^j\right),$$

(6)

where ${\tilde{\mathbf{h}}}_i^j$ is the globally aligned feature for modality j from vehicle i.

Alignment Network. The alignment network ${\mathcal{A}}_j$ can be implemented as a neural network trained to minimize the distance between features of the same class across different nodes. The loss function for training ${\mathcal{A}}_j$ could be a contrastive loss or a triplet loss:

$${\mathcal{L}}_{\mathrm{align}}=\sum _{i,k}{\left[\parallel {\mathcal{A}}_j\left({\mathbf{h}}_i^j\right)-{\mathcal{A}}_j\left({\mathbf{h}}_k^j\right){\parallel }^2-{\parallel {\mathcal{A}}_j\left({\mathbf{h}}_i^j\right)-{\mathcal{A}}_j\left({\mathbf{h}}_{\mathrm{neg}}^j\right)\parallel }^2+\alpha \right]}_{+},$$

(7)

where ${\mathbf{h}}_k^j$ is a feature from another vehicle with the same label as ${\mathbf{h}}_i^j$, ${\mathbf{h}}_{\mathrm{neg}}^j$ is a feature from a different class, and $\alpha$ is a margin parameter.

Feature Fusion. After aligning the features from all modalities, the next step is to fuse them into a single representation. This unified representation combines information from all modalities and serves as the input for the local prediction model. The fused representation ${\mathbf{h}}_i$ for vehicle i is obtained by concatenating the aligned features:

$${\mathbf{h}}_i=\mathrm{Concat}({\tilde{\mathbf{h}}}_i^1,{\tilde{\mathbf{h}}}_i^2,\dots ,{\tilde{\mathbf{h}}}_i^m),$$

(8)

where Concat denotes the concatenation operation across all m modalities.

Concatenation and Fusion Network. The concatenated feature vector ${\mathbf{h}}_i$ is then passed through a fusion network $\mathcal{F}$ to integrate the information from different modalities:

$${\mathbf{z}}_i=\mathcal{F}\left({\mathbf{h}}_i\right),$$

(9)

where ${\mathbf{z}}_i$ is the final fused feature vector used for prediction. The fused feature vector ${\mathbf{z}}_i$ is used by the local model $f_i$ to make predictions:

$${\widehat{y}}_i=f_i({\mathbf{z}}_i;{\mathbf{w}}_i),$$

(10)

where ${\mathbf{w}}_i$ are the local model parameters.

Training and Optimization. The local models are trained to minimize the empirical risk over their local datasets. The local loss function for vehicle i is defined as follows:

$${\mathcal{L}}_i\left(\mathbf{w}\right)={\displaystyle \frac{1}{n_i}}\sum _{j=1}^{n_i}\ell (f_i({\mathbf{z}}_{i,j};{\mathbf{w}}_i),y_{i,j}),$$

(11)

where $\ell (·,·)$ is the loss function (e.g., cross-entropy loss for classification tasks). The overall objective is to minimize the global loss function across all vehicles, accounting for the presence of Byzantine adversaries (see the following section):

$$\underset{\mathbf{w}}{min}\mathcal{L}\left(\mathbf{w}\right)=\sum _{i\notin \mathcal{B}}{\alpha }_i{\mathcal{L}}_i\left(\mathbf{w}\right).$$

(12)

4.2. Gradient Compression-Based Byzantine Aggregation

This section provides a detailed description of the Byzantine aggregation technique based on top-k gradient compression.

Local Gradient Calculation. Each vehicle $v_k$ computes the local gradient $\Delta {\mathbf{w}}_k$ based on its local dataset ${\mathcal{D}}_k$:

$$\Delta {\mathbf{w}}_k=\nabla {\mathcal{L}}_k\left({\mathbf{w}}_k\right)$$

(13)

where ${\mathcal{L}}_k\left({\mathbf{w}}_k\right)$ is the local loss function for vehicle $v_k$, and ${\mathbf{w}}_k$ are the local model parameters. Then we use top-k gradient compression to improve the communication efficiency, where top-k gradient compression involves retaining only the most significant elements of the gradient vector to reduce communication overhead. Given a gradient vector $\Delta {\mathbf{w}}_k$ from vehicle $v_k$, we can compute the magnitudes of all elements in $\Delta {\mathbf{w}}_k$:

$${\mathbf{magnitude}}_k=|\Delta {\mathbf{w}}_k|.$$

(14)

Top-k Selection. Here, we identify the indices of the top-k largest magnitudes by using the following equation:

$$\mathrm{topk}\_\mathrm{indices}=\mathrm{argsort}\left({\mathbf{magnitude}}_i\right)[-k:].$$

(15)

Then, we create a binary mask ${\mathbf{m}}_i$ where the positions corresponding to the top-k indices are set to 1, and the rest are set to 0:

$${\mathbf{m}}_i\left[j\right]=\left\{\begin{array}{cc}1\hfill & \mathrm{if}j\in \mathrm{topk}\_\mathrm{indices}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(16)

Gradient Compression. We apply the binary mask to the gradient vector:

$$\mathcal{C}(\Delta {\mathbf{w}}_k)=\Delta {\mathbf{w}}_k\odot {\mathbf{m}}_k$$

(17)

In this context, the compressed gradient $\mathcal{C}(\Delta {\mathbf{w}}_i)$ contains only the top-k elements of $\Delta {\mathbf{w}}_i$, reducing the communication load.

Robust Aggregation. To mitigate the influence of Byzantine adversaries, a robust aggregation method like trimmed mean aggregation is used at the central server. Given compressed gradients ${\left\{\mathcal{C}(\Delta {\mathbf{w}}_k)\right\}}_{k=1}^K$ from K vehicles:

• Dimension-wise Sorting and Trimming. For each dimension d of the gradient vector, we collect the k-th elements of the compressed gradients from all vehicles, i.e., ${\left\{\mathcal{C}{(\Delta {\mathbf{w}}_k)}_d\right\}}_{k=1}^K$. Then, we sort the collected values, i.e., $\mathcal{C}{(\Delta {\mathbf{w}}_{\left(1\right)})}_d\le \mathcal{C}{(\Delta {\mathbf{w}}_{\left(2\right)})}_d\le \cdots \le \mathcal{C}{(\Delta {\mathbf{w}}_{\left(K\right)})}_d$. After that, we trim the largest and smallest b values, where b is the estimated number of Byzantine adversaries.
• Mean Calculation. First, we compute the mean of the remaining values after trimming:

  $${\mathcal{A}}_d\left({\left\{\mathcal{C}(\Delta {\mathbf{w}}_k^t)\right\}}_{k=1}^K\right)={\displaystyle \frac{1}{K-2b}}\sum _{i=b+1}^{K-b}\mathcal{C}{(\Delta {\mathbf{w}}_k^t)}_d.$$

  (18)
        Then, we construct the aggregated gradient $\mathcal{A}\left({\left\{\mathcal{C}(\Delta {\mathbf{w}}_k^t)\right\}}_{k=1}^K\right)$ by applying ${\mathcal{A}}_d$ to each dimension d:

  $$\mathcal{A}({\{\mathcal{C}(\Delta {\mathbf{w}}_i^t)\}}_{i=1}^N)=\left({\mathcal{A}}_1({\{\mathcal{C}(\Delta {\mathbf{w}}_i^t)\}}_{i=1}^N),\dots ,{\mathcal{A}}_d({\{\mathcal{C}(\Delta {\mathbf{w}}_i^t)\}}_{i=1}^N)\right),$$

  (19)

  where d is the dimensionality of the gradient vector.
• Global Model Update. The server updates the global model using the robustly aggregated gradient:

  $${\mathbf{w}}_{t+1}={\mathbf{w}}_t+\eta \mathcal{A}\left({\left\{\mathcal{C}(\Delta {\mathbf{w}}_k^t)\right\}}_{k=1}^K\right),$$

  (20)

  where $\eta$ is the learning rate.

The Byzantine aggregation technique based on top-k gradient compression involves compressing local gradients by retaining only the top-k elements and using a robust trimmed mean aggregation method at the server. This approach effectively reduces communication overhead and mitigates the impact of Byzantine adversaries, ensuring resilient and accurate global model updates in the federated learning framework for ICVs. The detailed implementation steps (see Algorithm 1) and formulas provided establish the theoretical and practical foundations of the technique, ensuring it can be effectively applied in real-world scenarios.

4.3. Time Complexity Analysis

Analyzing the time complexity of the Byzantine-robust aggregation technique based on top-k gradient compression involves examining the computational cost of each step in the process. Here, we break down the time complexity for the key steps: local gradient calculation, gradient compression, transmission, robust aggregation at the server, and the global model update.

• Local Gradient Calculation. Each vehicle computes the local gradient $\Delta {\mathbf{w}}_i$ based on its local dataset. Assume the dataset has m samples and the model has d parameters. The time complexity for gradient computation is $\mathcal{O}(m·d)$. This is because each parameter gradient is typically calculated as a sum over the dataset, involving m operations per parameter.
• Top-k Gradient Compression. After computing the gradient, each vehicle compresses it by retaining the top-k elements, i.e., $\mathcal{O}\left(d\right)$ for magnitude calculation, $\mathcal{O}(dlogk)$ for top-k selection, and $\mathcal{O}\left(d\right)$ for binary mask creation and gradient compression. The overall time complexity for top-k gradient compression: $\mathcal{O}(d+dlogk+d)=\mathcal{O}(dlogk)$.
• Transmission. The transmission time depends on the communication bandwidth and is not typically considered in time complexity analysis. However, since only k elements are transmitted, the communication cost is $\mathcal{O}\left(k\right)$.
• Robust Aggregation at Server. The server aggregates the compressed gradients using the trimmed mean method, i.e., $\mathcal{O}(N·d)$ for dimension-wise collection, $\mathcal{O}(d·NlogN)$ for sorting, and $\mathcal{O}(d·(N-2b\left)\right)=\mathcal{O}(d·N)$ for trimming and mean calculation. Overall time complexity for robust aggregation: $\mathcal{O}(d·NlogN+d·N)=\mathcal{O}(d·NlogN)$.
• Global Model Update. The server updates the global model using the aggregated gradient. The time complexity for this step is $\mathcal{O}\left(d\right)$.

Combining all the steps, the total time complexity of the Byzantine-robust aggregation technique based on top-k gradient compression is dominated by the robust aggregation at the server, which involves sorting and averaging operations. The overall time complexity is

$$\mathcal{O}(m·d)+\mathcal{O}(dlogk)+\mathcal{O}\left(k\right)+\mathcal{O}(d·NlogN)+\mathcal{O}\left(d\right)$$

(21)

Since $\mathcal{O}(d·NlogN)$ is the dominant term, the overall time complexity is

$$\mathcal{O}(d·NlogN)$$

This complexity ensures that the framework can efficiently handle large-scale federated learning with numerous participants, provided the number of parameters d and the number of vehicles K remain manageable.

Algorithm 1: Byzantine-robust Multimodal Federated Learning Algorithm.

Input: Local model ${\omega }_k$ and local multimodal dataset ${\mathcal{D}}_k$. Output: Global model $\omega$ The server initializes the generator and global model and sends them to each vehicle;

5. Experiments

5.1. Experiment Setup

To evaluate the performance of our framework, we conducted extensive experiments on four benchmarking datasets. All experiments were implemented using Python 3.9 and PyTorch 1.12 and evaluated on a server with an NVIDIA A100 GPU.

Datasets. Our proposed framework was evaluated using three comprehensive multimodal datasets: KITTI [46], nuScenes [23], and KAIST Multispectral Pedestrian Detection Dataset [47]. KITTI provides RGB images, LiDAR point clouds, and GPS/IMU data for 2D and 3D object detection in urban and highway scenes. nuScenes offers a larger-scale dataset with additional RADAR data, covering diverse driving scenarios in multiple cities. The KAIST dataset focuses on pedestrian detection using RGB and thermal infrared images, challenging the framework with day/night variations. These datasets are adapted for federated learning by partitioning data among simulated ICVs, ensuring non-IID distributions, implementing privacy-preserving data handling, introducing Byzantine nodes, and simulating varying communication conditions. Together, they provide a robust testbed for evaluating our framework’s performance in multimodal fusion, Byzantine resilience, and adaptation to diverse ICV environments and object detection tasks.

Models. Our framework employs a diverse set of state-of-the-art models adapted for FL scenarios. These include PointPillars [48], which efficiently processes LiDAR point clouds and can be extended to incorporate RGB data; MVX-Net [49], designed for the multimodal fusion of LiDAR and image data; AVOD [50], a 3D object detection model that fuses LiDAR and RGB inputs; a custom KAIST Multispectral Pedestrian Detection Network for handling RGB and thermal infrared data; and YOLOv4-Multispectral, an extension of YOLOv4 [51] adapted for fast, multispectral object detection.

Parameters. In the proposed framework, key parameters include the number of participating nodes ($K=100$ vehicles), gradient compression rate ($r=100$ elements), and learning rate $\eta =0.001$. The framework is designed to handle $b=\{10,15,20,25\}$ Byzantine nodes. The model typically involves $d={10}^6$ parameters, with each node processing local datasets of $m=10,000$ samples. The datasets used include RGB images with resolutions of $1242\times 375$ (KITTI), $1600\times 900$ (nuScenes), and $1920\times 1280$ pixels (Waymo). LiDAR point clouds have densities of around 100,000 (KITTI), 300,000 (nuScenes), and 200,000 points per frame (Waymo). Radar data are captured at approximately 13 Hz, and GPS accuracy is within 1–2 m. These parameters ensure the framework’s efficiency, robustness, and effectiveness in real-world ICV scenarios.

Baselines. The proposed framework was evaluated using the following baselines:

• FedAvg [17]. FedAvg aggregates local models from all vehicles by averaging their parameters but does not account for Byzantine robustness.
• Krum [43] and Multi-Krum [52]. They are robust aggregation techniques designed to resist Byzantine attacks by selecting gradients that deviate the least from the majority.
• Trimmed Mean [53] and Median [28]. These aggregation methods enhance robustness by trimming extreme values and using median values to mitigate the influence of adversarial updates.
• Byzantine-resilient SGD (BrSGD) [54]. This approach focuses on detecting and excluding malicious updates during training.
• FLTrust [55]. This approach focuses on computing trust scores to select high-quality clients.

These baselines provide a comprehensive evaluation framework, allowing for a robust comparison of the proposed framework’s effectiveness in handling adversarial scenarios, maintaining accuracy, and ensuring efficient multimodal data fusion in ICV tasks.

Evaluation Metrics. Evaluating the proposed framework requires a comprehensive set of metrics to assess its performance across various dimensions. Key evaluation metrics include accuracy, which measures the proportion of correctly detected objects in multimodal datasets such as KITTI, nuScenes, and Waymo. Robustness metrics, i.e., the percentage of successful attacks detected and mitigated, are essential for evaluating the framework’s resilience against Byzantine adversaries. We use the communication overhead to assess the efficiency of the gradient compression and transmission processes, while convergence time measures how quickly the model reaches a satisfactory performance level. Together, these metrics ensure a thorough evaluation of the framework’s accuracy, robustness, efficiency, and overall effectiveness in real-world ICV scenarios.

5.2. Numerical Analysis

System Performance. This experiment aimed to study the system performance of the proposed framework. Specifically, we adopted local label-flipping attacks and Gaussian attacks as Byzantine attacks. We explored the performance of the proposed framework and baselines on different benchmark datasets under local label-flipping attacks. The experimental results are shown in

Table 1. Accuracy of the proposed framework and benchmarks on different datasets.

Method	KITTI	nuScenes	KAIST
FedAvg	65.4 ± 0.3	58.7 ± 0.2	67.8 ± 0.2
Krum	67.7 ± 0.1	62.4 ± 0.2	71.2 ± 0.1
Multi-Krum	68.9 ± 0.1	65.7 ± 0.3	74.1 ± 0.2
Trimmed Mean	66.8 ± 0.2	64.1 ± 0.2	72.6 ± 0.2
Mean	65.8 ± 0.1	59.7 ± 0.4	70.1 ± 0.2
BrSGD	71.2 ± 0.2	68.3 ± 0.3	75.4 ± 0.2
Ours	73.2 ± 0.2	72.5 ± 0.2	77.9 ± 0.2

, which shows that the proposed framework outperforms the baselines on different datasets, indicating that the proposed scheme can filter poisonous data well and maintain model performance. The experimental results show that the framework outperforms other baselines in terms of system performance and anti-poisoning attacks due to the efficient Byzantine defense mechanism and multimodal fusion design.

Secondly, we explored the impact of different numbers of Byzantine clients on the performance of the proposed framework and baselines under the above two attacks. Specifically, we set $b=\{10,15,20,25\}$. The experimental results are summarized in

Table 2. Accuracy of the proposed framework and benchmarks under different numbers of compromised clients.

Method	$\mathit{b}=10$	$\mathit{b}=15$	$\mathit{b}=20$	$\mathit{b}=25$
FedAvg	65.4 ± 0.1	63.8 ± 0.2	58.8 ± 0.3	52.4 ± 0.2
Krum	67.7 ± 0.1	65.6 ± 0.1	60.1 ± 0.2	54.8 ± 0.3
Multi-Krum	68.9 ± 0.2	67.1 ± 0.3	62.5 ± 0.2	58.7 ± 0.1
Trimmed Mean	66.8 ± 0.2	65.6 ± 0.4	63.7 ± 0.3	59.8 ± 0.1
Mean	65.8 ± 0.2	62.7 ± 0.1	58.9 ± 0.2	55.6 ± 0.1
BrSGD	71.2 ± 0.3	68.7 ± 0.2	66.5 ± 0.2	62.7 ± 0.1
FLTrust	72.2 ± 0.1	71.4 ± 0.2	68.7 ± 0.3	65.6 ± 0.1
Ours	73.2 ± 0.1	71.9 ± 0.2	69.7 ± 0.1	68.4 ± 0.2

and

Table 3. Accuracy of the proposed framework and benchmarks under different numbers of compromised clients for Gaussian attack.

Method	$\mathit{b}=10$	$\mathit{b}=15$	$\mathit{b}=20$	$\mathit{b}=25$
FedAvg	58.6 ± 0.2	54.2 ± 0.3	48.6 ± 0.3	42.7 ± 0.2
Krum	65.2 ± 0.2	63.2 ± 0.2	61.8 ± 0.3	56.7 ± 0.2
Multi-Krum	67.7 ± 0.2	65.4 ± 0.2	61.6 ± 0.2	56.5 ± 0.1
Trimmed Mean	64.6 ± 0.2	62.4 ± 0.3	58.7± 0.2	54.4 ± 0.1
Mean	62.7 ± 0.2	57.7 ± 0.1	55.4 ± 0.2	53.1 ± 0.1
BrSGD	68.7 ± 0.2	67.1 ± 0.2	64.8 ± 0.2	60.7 ± 0.1
FLTrust	71.7 ± 0.1	66.7 ± 0.2	65.4 ± 0.2	61.8 ± 0.1
Ours	74.2 ± 0.1	73.1 ± 0.2	70.8 ± 0.1	69.3 ± 0.2

. The experimental results show that the proposed framework is still more robust than other baselines under different numbers of attackers, which indicates that the proposed Byzantine defense based on gradient compression is very effective against these Byzantine attacks.

Communication Efficiency.

Table 4. Communication overhead of the proposed framework and baselines with different numbers of clients.

Method	$\mathit{K}=100$	$\mathit{K}=120$	$\mathit{K}=140$	$\mathit{K}=150$
FedAvg	4896 MB	5432 MB	5831 MB	6123 MB
Krum	4984 MB	5641 MB	6023 MB	6457 MB
Multi-Krum	5014 MB	5425 MB	5987 MB	6398 MB
Trimmed Mean	5021 MB	5531 MB	6015 MB	6157 MB
Mean	4974 MB	5324 MB	6074 MB	6248 MB
BrSGD	3697 MB	4125 MB	4897 MB	5324 MB
Ours	49.64 MB	53.24 MB	57.41 MB	60.23 MB

records the communication overhead results of the proposed framework and the baselines under different numbers of clients. The experimental results show that the proposed framework has a higher communication efficiency due to the use of a gradient compression scheme, which requires a small gradient size. In addition, cross-node multimodal alignment and fusion provide high-quality model update aggregation, thereby accelerating the convergence of the model.

Parameter Sensitivity. Here, we aim to explore the impact of the gradient compression rate parameter on the communication overhead of the proposed framework.

Table 5. Communication overhead of the proposed framework and baselines under compression rates.

Method	$\mathit{r}=100$	$\mathit{r}=110$	$\mathit{r}=120$	$\mathit{r}=150$
FedAvg	4896 MB	5432 MB	5831 MB	6123 MB
Krum	4984 MB	5641 MB	6023 MB	6457 MB
Multi-Krum	5014 MB	5425 MB	5987 MB	6398 MB
Trimmed Mean	5021 MB	5531 MB	6015 MB	6157 MB
Mean	4974 MB	5324 MB	6074 MB	6248 MB
BrSGD	3697 MB	4125 MB	4897 MB	5324 MB
Ours	49.64 MB	48.23 MB	46.65 MB	41.25 MB

shows the communication overhead results under different gradient compression rates, where the results show that the proposed framework can achieve a better communication efficiency.

Ablation Studies. Finally, we conducted ablation experiments to study the performance impact of different components within the proposed framework. Specifically, we explored the impact of the multimodal fusion mechanism and the Byzantine aggregation mechanism based on gradient compression on the framework, respectively.

Table 6. Ablation study results.

Method	$\mathit{K}=100$	$\mathit{K}=120$	$\mathit{K}=140$	$\mathit{K}=150$
w/o Fusion	68.9	67.7	66.8	65.7
w/o Aggregation	66.1	65.4	64.6	62.8
Ours	73.2	72.5	71.1	70.9

summarizes the experimental results. We observed that the performance of the proposed framework with and without the multimodal fusion mechanism was relatively close, indicating a good privacy–performance trade-off. In addition, we observed that the Byzantine aggregation mechanism based on gradient compression significantly improved the model’s robustness and performance.

Discussion. The proposed Byzantine-robust multimodal FL framework for ICVs can effectively scale to larger networks by leveraging adaptive robust aggregation mechanisms, hierarchical structures, and resource-aware optimization techniques. The dynamic network topology of ICVs is managed through asynchronous aggregation and buffer mechanisms, ensuring stability even with fluctuating connectivity. The framework’s multimodal fusion strategy accommodates varying data rates and heterogeneous sensor inputs by employing dynamic fusion weights and adaptive sampling. To handle increasing adversarial presence, hierarchical Byzantine-resilient aggregation, combined with reinforcement learning-based optimization, ensures that the system remains robust. Resource constraints are mitigated through bandwidth-aware compression and gradient sparsification, allowing for the framework to maintain efficiency even under communication limitations. Overall, these strategies enable the framework to scale effectively while preserving robustness, resilience, and performance in real-world ICV environments.

6. Conclusions

In this paper, we introduce a Byzantine-robust multimodal federated learning framework designed for ICVs. This framework addresses critical challenges in federated learning, particularly those related to data privacy, security, and robustness against adversarial attacks. By leveraging advanced techniques such as gradient compression and robust aggregation methods, the framework ensures efficient and secure training across multiple nodes, even in the presence of Byzantine adversaries. We highlighted the importance of multimodal data fusion by integrating diverse sensor data, including RGB images, LiDAR point clouds, radar data, and GPS/IMU measurements, to enhance the accuracy and reliability of object detection in autonomous driving. Through the use of benchmark datasets like KITTI, nuScenes, and Waymo Open Dataset, we demonstrated the framework’s capability to maintain high performance and robustness. Our evaluation metrics, including accuracy, precision, recall, robustness, communication overhead, and computational cost, provide a comprehensive assessment of the framework’s effectiveness. The proposed approach not only advances the state of federated learning in autonomous driving but also sets a foundation for future research on secure and resilient distributed machine learning systems.