Internet of Vehicles and Real-Time Optimization Algorithms: Concepts for Vehicle Networking in Smart Cities

Abstract

Achieving sustainable freight transport and citizens’ mobility operations in modern cities are becoming critical issues for many governments. By analyzing big data streams generated through IoT devices, city planners now have the possibility to optimize traffic and mobility patterns. IoT combined with innovative transport concepts as well as emerging mobility modes (e.g., ridesharing and carsharing) constitute a new paradigm in sustainable and optimized traffic operations in smart cities. Still, these are highly dynamic scenarios, which are also subject to a high uncertainty degree. Hence, factors such as real-time optimization and re-optimization of routes, stochastic travel times, and evolving customers’ requirements and traffic status also have to be considered. This paper discusses the main challenges associated with Internet of Vehicles (IoV) and vehicle networking scenarios, identifies the underlying optimization problems that need to be solved in real time, and proposes an approach to combine the use of IoV with parallelization approaches. To this aim, agile optimization and distributed machine learning are envisaged as the best candidate algorithms to develop efficient transport and mobility systems.

1. Introduction

As technologies, such as Internet of Things (IoT), self-driving vehicles, and 5G communications, are gaining momentum, transportation and mobility (T&M) operations in smart cities enter a new era with the so-called Internet of Vehicles (IoV) [1,2]. The efficient performance of IoV systems require online traffic data acquisition, ultra-broadband connections, alternative mobility modes, a sustainable use of energy, and low-latency optimization algorithms capable of finding high-quality solutions to complex operational challenges in real time (less than a second) [3]. In the context of smart cities, intelligent and sustainable vehicle networking (ISVN) constitutes a key development area that can significantly contribute to social and economic progress [4].

In these smart cities, large quantities of data are gathered in real time via electronic devices located inside vehicles and infrastructures (computer chips, sensors, traffic cameras, drones, etc.), transmitted over the Internet, and analyzed through information and expert systems [5]. The range of problems posed to manage the IoV, including the communications between nodes and the efficient processing of massive data, are vast. In addition, the activation of proper nodes is to be decided in a manner to optimize network performance, e.g., which radio side device to activate (Figure 1) [6]. In this context, we need a class of extremely fast, effective, and easily parallelizable optimization algorithms. These algorithms will allow for a coordinated and effective use of electric, unmanned, and self-driving vehicles in smart cities [7].

Agile optimization (AO) algorithms can be extremely fast in execution (thus allowing for real-time decision making), perfectly parallelizable (so they can be effectively executed on parallel threads), flexible (so they can be adapted to cope with different variants of routing and location problems, even those under dynamic or uncertain conditions), parameter-less (they should not require time-consuming fine-tuning processes), on-line (they have to be used iteratively as the data stream is updated), and still effective (they should be able to provide high-quality solutions to complex decision-making problems) [8,9].

Another area gaining momentum in smart city scenarios is the implementation of machine learning (ML) algorithms [10]. ML embraces a big family of different algorithms, each one tailored to specific requirements, in terms of data availability, latency, dynamicity, and so on. Recently, a novel paradigm in ML has been introduced, breaking from the traditional centralized approach where a big resource-unlimited node is able to execute an ML algorithm while a distributed scenario is exploited to implement distributed machine learning (DML) algorithms where multiple nodes interact among them to exchange their knowledge, experience, and environmental data in order to implement a faster and dynamic-aware approach [11]. This could be particularly interesting in the IoV scenario, which is still relatively unexplored, mainly due to the several challenges introduced by such a highly dynamic scenario.

The main contributions of this work are as follows:

• We analyze the main challenges associated with IoV and vehicle networking scenarios.
• We identify the associated optimization problems, some of which need to be solved in real time.
• We discuss how AO and DML approaches allow for the development of efficient transport and mobility systems.
• We provide some numerical evidence of the gains that can be obtained by employing the aforementioned approaches.

As previously mentioned, IoV and smart city scenarios are characterized by a high dynamicity and the generation of the significant amount of data required for their effective management [12]. This has required the introduction of optimization algorithms able to deal with stringent latency, high density, and big data. With this in mind, in this paper, we focus our attention on two main elements. On one side, AO, a novel concept that arises from the optimization algorithm parallelization, which allows us to reduce the time required to reach the problem solution. On the other side, there is recent momentum in ML algorithms, mainly due to their ability to solve complex problems; however, when several actors come into the system, traditional ML is unable to scale effectively—this is why we discuss distributed machine learning, a novel concept where traditional ML is considered to be distributed on several elements of the system.

This paper discusses, in detail, the challenges associated with IoV and how parallelization in optimization algorithms allow for the re-optimization of systems in just a few seconds. The rest of the paper is organized as follows: In Section 2, the main concepts related to IoV scenario and use cases identified by the standardization organizations are introduced. Section 3 reviews some related work on IoV, by focusing in particular on the optimization problems. Section 4 describes in more detail how AO algorithms work. Section 5 focuses on the DML algorithms and their application to IoV scenarios. Section 6 reviews some of the most popular vehicular networking models and provides a numerical analysis of different applications of AO algorithms. Finally, Section 7 summarizes the main contributions of the paper and provides future research lines.

2. The IoV Scenario and Use Cases

The Internet of Vehicles is envisaged as one of the most promising business areas in the near future [13]. Consequently, the automotive industry has been driving an intense research and development effort to identify business cases and technologies capable of supporting the new use cases requirements, which cannot be fully met with existing technologies. IoV is mainly characterized by the massive collection of data mainly provided by in-vehicle and infrastructure sensors, transferred to the network or shared among vehicles, and processed to feed intelligent algorithms to enhance network management and orchestration [14]. For instance, it is estimated that each connected vehicle can generate or consume more than 40 TB of data every eight hours [15], thus stretching the capacity of the networks to the limit and requiring the processing of a vast amount of data, in some cases in real time. Although this verticality has been gaining momentum during the last decade and has attracted the interest of the research community, it still presents technological challenges to be addressed. These challenges can be summarized as: (i) the large scale and extremely dynamic scenarios resulting from a large number of vehicles with high mobility; (ii) unstable connectivity caused by fast varying channel conditions; (iii) limited and distributed computational capacity; (iv) the need to support services with tight latency requirements or high data rate needs. Clearly, the aforementioned technological challenges fall into two research areas. On the one hand, the communication between nodes, which must be able to support high data rates, high reliability, and low latency communications. These communications are usually known as Vehicle-to-Vehicle (V2V), Vehicle-to-Infrastructure (V2I), Vehicle-to-Network (V2N), and Vehicle-to-Pedestrian (V2P) communications or, in general, Vehicle-to-Everything (V2X) communications [16]. On the other hand, the complexity, the scale, and the latency constraints of the problems call for real time optimization solutions which, in turn, must be adapted to the particularities of the network.

This work is aimed at discussing novel, efficient, and real time optimization approaches: AO and DML, both addressing the requirements of IoV. In order to set up the framework of the optimization problems and solutions, this section briefly overviews the communications technologies and the computational paradigm in IoV, and finally identifies the use cases proposed by standardization organizations in collaboration with the industry.

2.1. Communications and Computation

Standardization of vehicular communications took two different directions that still remain nowadays—one of them led by IEEE, and the other one carried out in the framework of 3GPP as a part of cellular technology [17]. Initially, the automotive industry, along with IEEE, led the standardization of dedicated short range communications (DSCR) with the development of IEEE 802.11p [18]. This standard is based on carrier sense multiple access/collision avoidance (CSMA/CA) and incorporates additional features to handle high mobility. However, and despite the new features adapted to vehicular scenarios, the medium access is still contention-based. In order to improve its performance, a new amendment has been developed as IEEE 802.11bd [19,20].

In parallel, and motivated by the importance of the automotive vertical, 3GPP incorporated the vehicular communications in Release 14, which has been enhanced since then in the successive releases until Release 17 [21,22,23], frozen in 2022. This standard is usually known in general as Cellular V2X (C-V2X), or LTE-V2X or 5G NR-V2X depending on the release. Details on C-V2X can be found in outstanding works [20,24,25,26]. Focusing on the 3GPP standard, two links are defined: the link between the vehicle and the base station, and the link between a vehicle and another vehicle or pedestrian, namely the sidelink. The resources of the link between the vehicle and the base station are managed by the base station, but for the sidelink, four different modes of operation are defined, two of them where resources are allocated without the support of the network, and two more where resources are granted by the network. The management of these resources is one of the key open research challenges.

As stated above, the operation of V2X is intimately bound up with the exchange and process of massive data, and therefore the computational paradigm is one of the key aspects in IoV. In [27], the two main computational paradigms for IoV, which are cloud computing and edge/fog computing, are identified and described. The main idea lying behind these two paradigms is the existence of centralized network computing capacity, able to aggregate data from different parts of the network, and edge computing capacity, which reduces the communication latency between the vehicle and the computational node. The centralization of computational capacity in the cloud benefits from cost-efficiency and scalability. However, moving computational capacity towards the edge of the network, enabled by technologies such as multi-access edge computing (MEC), allows instantiating application servers in the edge [28]. The implementation of MEC in the IoV architecture is also known as vehicular edge computing (VEC), and stands for a distributed computation between the vehicles themselves and the edge servers [29]. Some of the current research areas in VEC are the mobility management, since some computational tasks must be handed over and migrated when vehicles move away; the loss of connectivity between the task computing node (either a vehicle or an edge server) and the node that offloaded the task computation; the problem of allocating computational resources to each task in the existing nodes, either vehicles or servers [30]. In this context, optimization techniques must be able to take into account the IoV communications and network architecture nature in a convenient manner. Real time or near-real time approaches are required to keep the pace of the scenario dynamics.

2.2. Use Cases

3GPP has defined six groups of requirements for V2X, five of them describing major V2X scenarios and an additional one devoted to general aspects transversal to the five major areas [22]. Different analysis of the use cases or slightly modified use cases can be found [31,32,33]. Each of these areas is in turn divided into a set of use cases and scenarios [23]. The six areas are: (i) general requirements; (ii) vehicles platooning; (iii) advanced driving; (iv) extended sensors; (v) remote driving; (vi) vehicle quality of service support. The first area is devoted to general requirements such as location accuracy, the type of messages that must be supported, or the connection density. As for the last area, i.e., vehicle quality of service support, it defines the monitoring of the quality of service of V2X services as a key point in the management of the network and the services themselves. 3GPP defines this last area to make sure that real time data on the current quality of service is available for the network. In the following, the four remaining areas, from the second one to fourth one, are described and discussed focusing on the role AO and DML can play. In the following, the main VoI use cases are described as defined by 3GPP in [22,23].

2.2.1. Vehicles Platooning

Vehicle platooning is defined as the creation of groups of vehicles, one of them playing the role of leader, which exchange sensors’ information to increase the driving efficiency, reducing the inter-vehicle distance, reducing the fuel consumption, and allowing limited automated driving. The main research challenges in this area reside in how messages are disseminated within the members of the platoon and how the platoon leader/manager communicates with the network. The platoon must support up to five vehicles and, given that other V2X services can be run in the vehicles, 19 users can be using other V2X applications in the platoon. The reliability of communications must be between 90% and 99.99%, depending on the degree of automation, and the maximum latency is 10–20 ms upper bounded for platoon members’ cooperation.

2.2.2. Advanced Driving

The objective of advanced driving scenarios is providing the vehicles with the ability to interact with each other aiming at safer traveling, collision avoidance, and improvement of traffic efficiency. Differently from vehicle platooning, advanced driving assumes larger inter-vehicle distances and lacks a leader vehicle. All vehicles interact to announce driving maneuvers, such as slowing down, speeding up or lane changing, and exchanging relevant data for cooperative perception. The required reliability of communications is set to 99.999% for operations such as emergency trajectory alignment and 99.99% for most of the rest critical operations. Depending on the scenario, the maximum latency ranges from 3 ms for emergency alignment trajectory, to 10–15 ms for cooperative lane exchange, or 100 ms for the rest of scenarios.

2.2.3. Extended Sensors

Extended sensors area encompasses the collection and exchange of data (e.g., video, sensors, etc.) captured by any node in the scenario, including the network, vehicles, or pedestrians, to improve the perception composited by each vehicle. Requirements in terms of reliability and maximum latency are similar to the ones defined for advanced driving.

2.2.4. Remote Driving

This area sets the framework for UAVs and their remote operation. The area presents major differences with the previous ones for multifarious factors. Raw and processed data are gathered locally by the vehicle and are forwarded up to the cloud, where the remote driving platform resides. That is, data are not exchanged with nearby vehicles, but merged in the cloud with other vehicles, infrastructure data, etc., to enable their remote operation. Other aspects such as the calculation of efficient routes can be included into this area. The required reliability in the exchange of information between the vehicle and the V2X application server is 99.999% and the maximum allowed end-to-end latency is estimated around 5 ms.

3. Literature Review on IoV Analytics

The IoV field has been extensively studied in the literature. In this section, a review of the state-of-the-art is presented from a double perspective. Firstly, research works are grouped according to the topic under study, emphasizing the optimization framework used in each one. Then, optimization frameworks are discussed, which constitutes the second dimension and identifies the match between research challenges and optimization algorithms characteristics. The overview does not pretend to be exhaustive, but just to point out recent works that showcase the current research interests in the field of IoV.

3.1. Main Research Problems

Given the importance of both computational and communications in IoV, one of the main research topics addressed in the literature is the joint management of computational and communications resources, aiming at allowing a distributed and collaborative processing environment as well as optimizing the communication infrastructure.

Due the limited computational capacity of vehicular nodes and the need for intensive computation, computation offloading techniques have attracted the interest of the research community [34]. Task offloading is defined as the transferring of computational tasks from nodes with limited computational capacity—usually vehicles—to nodes with more extensive computational capacity—e.g., RSUs or the cloud servers. Several works address the task offloading problem from similar perspectives [35,36,37,38,39,40]. Liu et al. [35] propose a reinforcement learning-based solution to distribute tasks among the available MECs. Specifically, the policy gradient-based distributed computational offloading problem (PGDCO) is proposed and takes into account the dependency between tasks, parallelizing in different MEC servers, the execution of independent tasks, and serializing the computation of dependent tasks. Similarly, in [36], the problem is addressed without taking into account the tasks dependency, and a deep deterministic policy gradient solution is proposed. Zhang et al. [37] introduce a scenario where an SDN controller manages a set of MEC servers, each one allocated in an RSU. The distribution of tasks among MEC servers, along with the management of resources such as power transmission, is modeled and solved with a two level solution based on Q-learning and potential game theory. Further, by modeling the task offloading problem with game theory, Wang et al. [38] set the framework of a problem to decide whether a task is not offloaded and executed in the vehicle, or it is offloaded to neighboring vehicles or edge servers of cloud servers. Prathiba et al. [39] propose a solution based on stochastic network calculus to compute the delay upper bounds and a federated Q-learning algorithm to distribute the tasks. Finally, Bozorgchenani et al. [40] propose a network selection strategy for optimizing the computation offloading between vehicles and RSUs, acting as VEC nodes. The problem has been solved resorting to both on-line and off-policy algorithms by exploiting the multi-armed bandit theory.

As for the communication resource allocation problem, two outstanding surveys have been published [41,42]. Focusing on the specific resources allocation problems, the distribution of resources in sidelink communications operated in a distributed fashion—i.e., modes 2 and 4 of operation—is addressed in some works [43,44,45]. Thus, Yoon et al. [43] pose a simulation based analysis of scheduling performance in mode 2 sidelink communication. In particular, the work studies a scenario where part of the users have full-sensing capacity and part of the users have limited sensing capacity. It is shown that defensive transmission strategy employed by full-sensing users mitigates the packet delivery ratio degradation. Instead, Yoon and Kim [44] propose a resources reservation for mode 2 aimed at efficient allocation for aperiodic arrivals. Further, focused on the distributed reservation of resources for sidelink communications, in this case in mode 4, Zang and Shikh-Bahaei [45] propose a semi-persistent scheduling solution combining full-duplex transmission/reception and a deep Q-learning algorithm. In [46,47], deep reinforcement learning solutions are proposed for power and channel allocation. In particular, Kumar et al. [47] introduce a multi-agent reinforcement learning solution to predict the position of the vehicles and allocate the channels in a distributed manner. Conversely, stochastic geometry is used to analyze the performance of distributed resources allocation in ultra-dense scenarios, such as traffic jams, in [48].

The ‘softwarization’ of mobile networks has been one of the enablers of network slicing as a way to provide services with the required quality of service. In this context, and given the different requirements of V2X applications, as described in Section 2, network slicing has also attracted the interest of academia and industry in the framework of IoV [49,50,51,52]. Khan et al. [49] analyze how distributed remote radio heads and centralized baseband unit controllers must be paired to guarantee the requirements of best-effort and ultra-reliable low-latency communications slices in the IoV framework. The solution, based on genetic algorithms, takes into account capacity and delay of the network. Okic et al. [50] propose the deployment of an emergency network slice to prioritize critical mission operations. The proposal is based on the popular deep recurrent neural network long-short term memory (LSTM), known as an effective solution for the forecasting of time series, and characterize how road events propagate over distant base stations. Nassar and Yilmaz [51], instead, address a more classical network slicing problem in the edge of IoV, where a Q-learning solution is proposed as the solution. Further, based on deep reinforcement learning, Yu et al. [52] propose a soft actor-critic solution combined with an algorithm denoted as alternative slicing ratio search (ASRS) to provide radio access network slices with the required resources.

Different from the research problems discussed above, researchers have also been active in the problem of user association and RSU distribution. Regarding the user association problem, Wang et al. [53] develop a user association solution based on deep learning. In that work, feature learning is used to learn the link quality, and the strategy learning—either Q-learning or actor-critic—decides the association of each user. Moving a step forward, Cesarano et al. [6] formulate the user association problem in vehicular networks as an uncapacitated facility location problem (UFLP) for jointly solving the RSU-to-vehicle allocation problem while managing the RSUs switch-on and -off processes. Differently from traditional UFLP approaches, the authors propose a fast-heuristic approach based on a dynamic multi-period time scale mapping. Alablani and Arafah [54] estimate the dwelling time of vehicular nodes in each cell, and a handover procedure based on these estimates is proposed. A problem slightly different is addressed in Roger et al. [55], where not only user association is studied through a simulation-based analysis, but also the antenna selection in the vehicle side.

The massive data exchanged in IoV brings the capacity of the network to the limit; thus, risking the ability of the network of meeting the required quality of service. In this context, edge caching has also been investigated [56,57,58,59]. Wang et al. [56], given the massive amount of data to be shared among vehicles, propose caching the shareable content in RSUs to reduce the V2N traffic. The problem is modeled as an integer nonlinear program (INLP). Instead, Sanghvi et al. [58] present a reinforcement learning-based solution, namely Res6Edge. Reinforcement learning is one of the most popular approaches to address the content caching update. Further, heuristics have been proposed to solve the edge caching problem. For instance, in [59] a caching solution aimed at the distribution of large size files.

The large dynamics of vehicular networks pose challenging requirements for routing algorithms. Although routing has been extensively studied in the past, there are some recent works proposing routing algorithms for IoV. The connectivity of vehicular ad hoc networks has been analyzed though a stochastic geometry approach when the distribution of vehicles is not uniform over the whole scenario in [60]. The impact of connectivity and energy efficiency is pointed out. In [61], instead, a graph-theory-based routing algorithm is presented. Results are promising, but the main drawback of the proposal has to do with scalability of the problem and the required computational time. Meng et al. [62] propose an algorithm to control the routing of an official business vehicle flow. The algorithm makes use of data provided by the in-vehicle network environment. The authors make use of an expert system, and employ the K-means clustering algorithm to process traffic flow data. In the context of highly dynamic UAV networks, Arafat and Moh [63] analyze different routing protocols considering factors such as network topology, UAV positions, hierarchies, and uncertainty. Similarly, in the context of vehicular ad hoc networks, Nazib and Moh [64] also analyze several routing protocols for UAVs. The interested reader could refer to [65] for an overview of the different routing protocols considered to be used in IoV scenarios.

Other less popular research topics can be found in the literature. For instance, the vehicle software over the air update is dealt in [66]. The objective of the problem is finding the minimum set of base stations from which software updates should be downloaded. To do so, the authors perform traffic pattern analysis to predict the over the air update delay by using a transfer learning approach, and then the K-means algorithm to cluster the base stations. Another example is the service migration. The idea of micro-services instantiated when and where needed becomes more challenging when the mobility of the scenario is large. In [67], a Lyapunov optimization approach is proposed to select where services must be instantiated, and how many instances of the service are required to guarantee the service continuity. The prediction is performed with a recurrent neural network.

The mobility prediction has also been studied aiming at creating reliable mobility models. In the framework of IoV, road safety has been addressed as a mobility prediction problem in [68]. Specifically, LSTM is executed in the MEC to predict the trajectory of the vehicles and assist road safety with proactive actions. Similarly, Gupta et al. [69] analyze dangerous road locations and propose a multi-hop protocol to guarantee vehicles’ safety. The proposal is modeled with a clustering approach.

Optimizing the traffic flow and corresponding congestion management is an important aspect of modern vehicular networks. Here, we discuss some of the promising studies from recent times that are analyzing the congestion problem. Dhanare et al. [70] have analyzed various bio-inspired optimal routing algorithms for IoV scenarios from recent studies. Comparisons are drawn based on the strengths, weaknesses, and various other critical characteristics leading towards a potential combined model of a multi-modular bio-inspired approach to IoV routing. In another case, Aung et al. [71] proposed a new vehicular traffic congestion pricing system based upon reward/penalty policies. The system tends to reward the good behavior of users through the proposed T-Coin being a virtual currency and penalizing the users with bad behavior. The proposed virtual currency (T-Coin) can also be used to manage road reservations. Aung et al. [72] have combined the IoV resources with the virtual currency for creating a reward/penalty-based framework for optimizing the traffic flow through congestion management. A hybrid optimization technique based upon the modified ant colony and firefly optimization techniques is proposed in [73] for finding the optimal path impacting the reduced traveling time. Zhou et al. [74] have effectively utilized edge computing facilities for optimizing traffic management via adequate traffic light control. In particular, a decentralized reinforcement learning (distributed deep Q-learning) scheme is proposed for solving the problem at the edge.

3.2. Optimization Tools

As described in Section 3.1, the range of research issues in the field of IoV is vast and is being actively researched. However, although there is consensus on the research open issues, there is not actually an agreement on which optimization tools are the most convenient ones to meet the requirements of IoV while, in parallel, fitting the network architecture, characterized by distributed nodes, most of them with limited computing capacity, and with large dynamics. Recently, artificial intelligence (AI) is gaining ground as the key optimization tool for the challenges dealt in IoV [75]. Yet, the range of different optimization possibilities goes far beyond AI. In the following, the main optimization approaches mentioned in the previous section are listed and discussed.

AI has emerged as a promising approach to face IoV. As discussed previously, both supervised and unsupervised machine learning are proposed in two different directions. On the one hand, unsupervised learning is mainly used for clustering vehicles or base stations. Conversely, recurrent neural networks, such as LSTM, are becoming popular as supervised machine learning solutions able to forecast time series [76,77,78]. The main drawback of these approaches in IoV is two-fold. First, the amount of data and time required to train the network can be an obstacle in some scenarios, particularly when some aspects of the scenario vary. For instance, the installation of an additional RSU, the blockage of some streets, etc. Secondly, machine learning approaches do not perform properly when unexpected events occur. If low probability but critical events are not identified and conveniently processed in the training phase, the performance of the solution is degraded. In parallel, (deep) reinforcement learning has also undergone a considerable boom as a model-free approach to solve optimal control problems in IoV [74,79]. Although reinforcement learning is appropriate for IoV problems, it suffers from slightly different drawbacks from supervised/unsupervised learning. In this case, reinforcement learning is able to adapt to changes that occur in the environment, though at the expense of extended convergence time, which may degrade the performance if changes are frequent. However, recently a new ML training paradigm in the form of DML has emerged as a promising technique, especially in the case of latency-critical vehicular networking scenarios. Various DML techniques such as federated learning (FL) [80], decentralized learning (DL) [11], and collaborative learning (CL) [81] have shown great promises in terms of reduced training costs and frequent/online ML model updates for reducing the model drift, allowing their use in dynamic scenarios.

Game theory is also a popular approach to deal with IoV optimization problems [82]. It is a particularly appropriate framework for decision making when there is a clear interdependence between the decisions of the different ‘players’ involved. However, game theory can present scalability issues and it assumes that all users have knowledge on their own pay-off functions, which is not always the case.

The optimization framework can rely on a model. This is the case of approaches such as stochastic geometry, graph theory, integer programming, or Lyapunov optimization. In all these cases, the need of an accurate model limits the actual real life application. Moreover, in integer programming and graph theory, scalability must be carefully analyzed when the expected number of vehicles is large.

Table 1. Main challenges and methodologies employed.

Challenge	Methodology	References
Clustering vehicles/base stations	Unsupervised learning	Song et al. [83]
Time series forecasting	Recurrent neural networks	Hewamalage et al. [84]
Optimal control problems	Reinforcement learning	Zhou et al. [74]
Latency-critical vehicular networking	Distributed machine learning	Muscinelli et al. [11]
IoV optimization problems	Game theory	Sun et al. [82]
IoV optimization problems	Integer programming	Salahuddin et al. [85], Ning et al. [86]

summarizes the main challenges identified in the literature on IoV analytics, as well as the methodologies usually employed to cope with each of them.

4. Agile Optimization Algorithms

Different optimization algorithms could be used to optimize problems, such as exact algorithms that find the optimal solution to a problem. However, these algorithms are limited to small-size problems. For larger instances of problems and NP-hard problems, different heuristics are recommended. These heuristics construct solutions in an iterative and greedy procedure. A candidate for a solution is selected from a list of candidates at each iteration. The list of candidates is sorted according to pre-defined criteria, such as costs or traveled distance. In the end, a solution is constructed that is typically promising. Examples of these heuristics can be found in different variants of the vehicle routing problem [87] and the traveling salesman problem [88]. Local searches and perturbation movements could be adapted to improve the solutions constructed by the heuristics.

These heuristics are deterministic. Thus, the same solution is constructed at each execution. Different solutions could be found by selecting candidates according to a non-uniform distribution. This approach introduces a biased randomized behavior [89]. In this approach, more promising candidates obtain a higher probability of being selected while constructing a solution. These promising candidates are found at the top of the sorted candidate list.

Assigning non-uniform probabilities to the lists of candidates plays a role in keeping the heuristic logic [90]. A theoretical probability distribution could be used to assign the non-uniform distribution of solution candidates. These theoretical distributions help to minimize the computation time required compared to defining an empirical distribution. Examples of these distributions are the geometric distribution and the decreasing triangular distribution. These distributions have few parameters and are easy to implement using analytical expressions. The geometric distribution has a single parameter p. Figure 2 and Figure 3 show the probability distribution of a geometric distribution with different p values. It can be noticed that the most promising candidate obtains the highest selection probability that equals the value of the parameter p. The other candidates obtain a lower probability depending on p. For example, The fourth candidate in the list has a selection probability of $0.1$ compared to the probability of around $0.07$ under the parameters values $0.2$ and $0.5$, respectively. On the one hand, the probability distribution of small parameter values ($p\to 0$) approaches a uniform distribution, and many elements in the list are assigned a selection probability. On the other hand, larger parameter values $p\to 1$ assign considerable different probabilities to the top elements in the candidate list, and the probabilities are more likely to represent the greedy behavior of the heuristic. The geometric distribution has been successfully utilized to introduce the biased randomized behavior in heuristics used to solve problems in mobile cloud computing [91] and food logistics [92].

As a result, many executions of the biased randomized heuristic result in different solutions; these executions could be performed in a parallel or serial manner. In each execution, a solution is constructed according to the heuristic logic, and the solutions form biased randomized variations of the greedy solution. At least one of these solutions might outperform the greedy solution constructed by the heuristic [90]. In addition, these solutions help to explore the search space of the problem being solved. Thus, the biased randomized heuristic is a candidate to be integrated into a multi-start framework [93] or used in parallel programming. In the multi-start framework, the heuristic constructs a solution at each new start (iteration). The best solution to the problem being solved is defined by comparing constructed solutions with the so-far best-found solution.

Running parallel executions of the biased randomized heuristic is illustrated in Figure 4. Computer threads could be utilized; thus, each thread executes one run of the biased randomized heuristic with a different seed of the random number generator and resulting in forming an ‘embarrassingly parallel’ algorithm [94]. Embarrassingly parallel algorithms might be executed by parallel processing architectures [95]; moreover the recently introduced MEC and VEC paradigms inherently consider the possibility of parallelizing the task execution. Hence, the VN environment results to be in a suitable candidate for the execution of AO. Each execution of the heuristic constructs a solution within a short time (an instantaneous solution). The greedy procedure of the heuristic constructs the reference solution (baseline in Figure 4), and the solutions resulting from the biased randomized heuristic execution vary corresponding to the baseline. One of these solutions might outperform the baseline. In addition, the executions might be carried out by multi-core processors or graphical processing units (GPUs) [96]. The GPU consists of multiple low-energy cores and has advancements in performance and energy efficiency relative to traditional processors. These architectures are suitable for running embarrassingly parallel algorithms. In addition, one core in a multi-processor can execute several threads at once. Multi-processor units could be found in small chips [97], enabling different applications in smart cities to solve raised problems in real-time.

The illustration shown in Figure 4 demonstrates an AO algorithm, in which a solution to an optimization problem could be found in a reasonable short wall-clock time. In this context, AO refers to the parallel execution of a biased randomized heuristic. Thus, AO algorithms are characterized as (i) easily parallelizable, (ii) fastly executed, (iii) easily tuned (few or no parameter is needed), (iv) flexible in solving different problems, and (v) allowing re-optimization of solved problems. Many hundreds of biased randomized heuristics are run concurrently, and many solutions are constructed and evaluated during the same wall-clock time. In smart cities, unmanned aerial vehicles or self-driving vehicles might benefit from AO, from embarrassingly parallel algorithms on a multi-processor in a single chip.

5. Distributed Machine Learning Algorithms

While traditional ML approaches have demonstrated their limits when applied to dynamic and variable environments, such as the IoV scenario, a new class of algorithm has been recently introduced allowing to cast for their usage in dynamic scenarios.

They are usually referred as distributed machine learning (DML) algorithms. While the concept is quite broad, they mostly refer to a new class of algorithms able to gain from the presence of multiple nodes that do not only act as input/output for the optimization tool while they are active parts of the optimization process. In such a distributed system, the different nodes composing the system are able to run a fraction of an ML algorithm, enabling them to work closer to the data production/sensing so as to enable a closer interaction with data and a more reactive approach with respect to them.

While DL refers to a general view, in the following, we focus more on some specific distributed implementations, known as decentralized learning, collaborative learning, and federated learning, and how they can be usefully considered for the IoV use-cases optimization.

5.1. Centralized Learning

In centralized ML, more commonly referred to as ML, the idea is that there is a central unit, characterized by a high-performance processing unit, a GPU or an ML dedicated processing unit, capable of running ML specific algorithms. Such processing unit takes input data from the nodes for training, elaborating, and exploiting the ML algorithm to be used. The ML algorithm could be any supervised, unsupervised, or reinforcement learning, where different approaches are used for elaborating the optimal strategy. For any additional information, the reader could refer to [10].

5.2. Decentralized Learning

In decentralized learning, or DL, the main idea is that multiple learning agents are distributed within the scenario. Such distributed agents can interact, or not, among them by creating different types of distribution. DL approaches have been recently developed, gaining from the introduction of novel communication paradigms, as those defined in 5G, B5G, and forthcoming 6G systems [11]. At the same time, the use of a distributed approach can solve several issues arising from the centralized implementations. Wireless and intermittent connections, energy issues, and privacy concerns raise the necessity of developing algorithms that should not rely on the presence of a centralized node, but can still work independently or in a decentralized fashion.

As an example, Xu et al. [98] propose the use of an MEC infrastructure to deploy several deep learning agents. Each of the agents is capable of training a deep neural network for image processing. However, to avoid a very long convergence time, several agents can collaborate among them by elaborating a subset of the image so that the distributed fashion is able to completely train the network exploiting the parallelism. The output of the training can then be used for IoV applications. Dong et al. [99] propose instead to deploy a system of distributed deep learning agents in an IoV scenario. Those nodes can interact among them by exchanging data and parameters so as to create a DL approach. However, due to the intermittent nature of the vehicular environment, a proper node selection could be performed. In the paper, a proper policy is developed allowing to select which of the nodes should collaborate in order to respect proper timeliness requirements. Zhou et al. [74] propose instead a multi-layer hierarchical DL scenario. By exploiting the presence of different nodes (i.e., vehicles, VEC and cloud) able to execute reinforcement learning algorithms, the authors propose to exploit them for collecting data from the lower layer and send the outcome of the RL algorithm to the upper layer to create scenario-wide learning. The main challenges reside here in the connection among nodes, which is supposed to be implemented through proper V2X communication technologies. Ma et al. [100] propose to map a DL algorithm on a platoon on vehicle where the different nodes train the same model and exchange the parameters. Moreover, depending on the processing capability and availability of the different vehicles, proper scheduling is performed allowing us to optimize the distribution mechanism.

5.3. Collaborative Learning

While DL refers to a general approach where multiple agents works in parallel sharing, broadly speaking, in the ML effort, whether it is for training or testing, collaborative learning can be considered as a specific type of decentralized learning where multiple agents collaborate to implement an ML algorithm. This is a very efficient way to implement a DML algorithm in an IoV scenario since V2X connection can be used. Let us focus indeed on a scenario where multiple vehicles travel around; thanks to proper V2V connections, they can exchange ML data, allowing a tighter interconnection among vehicles.

Balkus et al. [81] survey on the importance of collaborative learning techniques in IoV scenario exploiting 5G communication systems. In particular, they focus on the importance of this approach when dealing with autonomous driving use cases. Five main questions are solved in the paper regarding the implementation of an autonomous driving system, that are: (1) How can autonomous vehicles (AVs) effectively use wireless communications for transmitting data on the road? (2) How can AVs manage the shared data? (3) How can shared data be used for improving AVs environment perception? (4) How can shared data be used to drive more safely and efficiently? (5) How can shared data privacy be protected and cyberattacks prevented? The authors have shown how vehicles can collaborate for improving the machine learning process aiming at better complying with environmental perception and data fusion. When information is exchanged through different vehicles, the environment is better understood and AV can move around relying with a higher safety and collision avoidance behavior as well as employing collaborative traffic analysis.

Kumar et al. [101] introduced, for the first time, the possibility of collaboration among different sources for better estimating the best route to be taken by vehicles in an urban scenario. In particular, through the presence of different learning automata, the information coming from different sensors is collaboratively used to find the best route. A similar scenario is also considered by Wu et al. [102], where the best route selection problem is solved. Here, instead, the use of an edge computing layer is considered. Through a strict collaboration between vehicles, edge nodes, and cloud computing nodes, a collaborative learning solution is proposed.

5.4. Federated Learning

While collaborative learning mainly relies on the exchange of information among vehicles and their co-located learning agent, in federated learning (FL), the approach is a bit different. Indeed, the exchange of data among nodes could impact the learning convergence a lot, since a large amount of data should be transferred. Despite the fact that new 5G technologies allow high data rate transfer, collaborative learning still need to transfer huge amounts of data. In FL, instead, the idea is that of exploiting, in a more inherent way, the ML algorithm’s parameters. In that context, the simplest FL scenario is defined through the presence of a central unit aiming at only merging the ML parameters, where the ML agents are implemented in a distributed fashion. Such unit, which can be RSUs or vehicles, should execute a partial training process on their own data, while the partial model is uploaded to the central node. After having merged all the nodes, the central node sends the general ML model parameters back to the agents. The process is repeated several times up to achieving convergence.

Such an approach is convenient in a distributed scenario, such as IoV, where multiple nodes can parallelize the ML algorithm execution, while the global model is evaluated at a central point. In addition, since only the model parameters are exchanged, privacy concerns are avoided, where only the ML parameters are exchanged among nodes.

The importance of using FL in IoV scenarios is evident by looking at several papers recently published on FL application to the vehicular environment. In particular, for a better understanding of the main solutions and challenges, the interested reader could refer to [103,104,105]. More recently, FL has been further extended with more challenging IoV scenarios. As an example, Zhou et al. [106] extend the traditional two-layers model to a three-layer model, where vehicles, RSUs, and the cloud interact to create a hierarchical FL structure. In particular, the RSUs perform an intermediate aggregation to limit the burden on the cloud when several updates are sent simultaneously. Liang et al. [107], propose to use a semi-synchronous approach to limit the detrimental effect of dynamicity in the vehicular scenario. In this approach, only a subset of the nodes update the model depending on their characteristics, while others can update the model while the aggregation is performed. In this way, the convergence speed would be much higher while most of the parameters coming from the vehicles are considered. The use of edge computing is also considered by Li et al. [108], where a four-layers infrastructure is supposed to be implemented. The intermediate MEC layer is deployed for computation offloading operations, while it is proposed to be used also for implementing the FL process. Bao et al. [109] propose an algorithm for selecting the nodes that should participate in the FL process. An edge-computing-based joint client selection and networking scheme for IoV is proposed by assigning some vehicles as edge vehicles, employing a distributed approach, and using vehicles as FL clients to conduct the training of local models, which learns optimal behaviors based on the interaction with environments. Sun et al. [110] instead consider the FL for solving the scheduling problem when trying balance the computation load among different nodes. Thanks to FL, the process can be more effectively balanced considering also the user mobility and their constraints. Saputra et al. [111] propose an economic framework for improving the advantages from the internet service provider point of view in helping the vehicles. This is achieved through the use of an FL platform that is able to select the best vehicles to be part of the FL process. Shinde et al. [112] propose instead a different approach that considers the possibility of jointly exploiting the VEC nodes for both FL and offloading process. To this aim, a proper trade-off between the two processes should be considered to allow the FL to properly converge while the proper amount of data are offloaded.

Phung et al. [113] propose to use a fog computing platform as an intermediate layer between the vehicles and the cloud. Originally proposed for deploying different services to the users, such an infrastructure is demonstrated to be also useful for the FL process, which is here considered as a task to be executed over the fog infrastructure. A similar approach is considered by Hammoud et al. [114], where a technique for creating stable fog federations of nodes is considered, allowing a more stable FL training phase. This is applied to an IoV scenario where the mobility of the nodes could negatively impact on the federated agent deployment if not correctly considered.

As mentioned, another advantage when using FL is related to the possibility of preserving the secrecy of the data. This is exploited by Zhao et al. [115], where a social IoV environment is considered. The users exchange their data for improving the system efficiency, while at the same time their secrecy is preserved thanks to the FL process. An intrusion detection system fer vehicular environments is instead considered by Liu et al. [116]. In particular, federated learning and blockchain technologies are jointly used with this aim, where the secrecy of data exchange for the FL training is enforced by the blockchain technology.

Pokhrel and Choi [117], instead consider the impact of FL on the network architecture. To this aim, we propose to use a specific TCP implementation for compelling with detrimental effect of mobility and long distance transmission in an IoV FL scenario.

5.5. Distributed Learning for ISVN

With these previous studies, it is inherently clear that distributed learning can play a huge role in enabling ISVN. Various distributed learning frameworks can perform efficient learning operations with the help of distributed vehicular data. In the case of latency-critical vehicular scenarios, it is important to perform ML training operations with limited resources. A pretrained ML model can often become outdated over time, incurring the issues such as model drift. Work performed in [112] highlights the importance of performing the continuous model training operations with new vehicular data. In particular, the authors have proposed the optimization approach for joint computation offloading and the FL training process. The trade-off between the FL training iterations vs. offloading performance is well explored. Additionally, vehicular mobility being one of the bottlenecks during vehicular processing operations is also taken into account. Such studies can surely motivate the vehicular community for adapting the online distributed learning operations by taking into account the resource constraints. Optimizing the distributed learning process based on the local vehicular environment can be a key idea for enabling the ISVN.

Additionally, hierarchical learning approaches such as distributed FL process can also be useful while diving into the upcoming 6G world and corresponding intelligent vehicular services [118]. In such cases, the averaging process of FL can be distributed over multiple computing platforms/layers enabling the possibility of a large number of users to participate in the learning process with limited resource requirements and link failure probabilities.

Integrating the meta-learning approaches in the traditional distributed learning process can also be one area to explore, especially for the ISVN case. Different meta-learning approaches can effectively reduce the training resource requirements and create high-quality ML models by exploring previous learning experiences.

Thus adapting the distributed learning process based upon the limited vehicular resources, and vehicular mobility, considering the multilayered edge computing platforms (i.e., joint terrestrial and non-terrestrial networks) for distributing the training process, integrating the advanced learning techniques such as meta-learning into traditional learning approaches are some of the major future directions that can be useful for enabling the ISVN in coming days.

The following

Table 2. Distributed ML models and corresponding characteristics.

Characteristics	Centralized Learning	Decentralized Learning	Collaborative Learning	Federated Learning
Learning Entity	Centralized Server	Distributed Servers/Devices	Distributed Servers and/or devices	On Device (VUs)
Scalability	Low	High	High	Very High
Energy Cost (VU Side)	Low	High	High	High
Latency	High	Low	Low	Medium
VUs Sensitive Data Privacy	Medium	Limited	Limited	High
Pros	Training Fairly Complex ML models (i.e., DNN with limited number of layers)	Training Complex ML Models	Training Models with Limited Complexity	Training Models with Limited Complexity
Main Challenges over ISVN	High Resource Requirements, larger training cost/latency	High resource Requirements, Server/Device Selection	VUs mobility, Server/Device Selection with Proper Data	VUs mobility, Proper Device/Server Selection, Model Accuracy

compares the different learning approaches in terms of their typical characteristics, especially for the vehicular user (VU) case.

6. Computational Results using AO Algorithms in ISVN

The development of AO algorithms was motivated by the need to address real-life optimization problems characterized by high dynamism. AO algorithms combine heuristic, biased-randomization techniques, and parallel computing to deal with the dynamic part of the problem. These algorithms belong to the field of parallelized heuristics, and can be used to generate efficient solutions to large-scale combinatorial optimization problems in real time. In the T&M field, these dynamic elements can be, for example, changing demands or travel times. In real life, we face a huge variety of optimization problems, including vehicle routing problems (VRP) [119], facility location problems (FLP) [7], arc routing problems (ARP) [120], team orienteering problems (TOP) [121], etc. Typically, the goal of each problem is to maximize the total reward or to minimize the total cost associated with the activity. Figure 5 presents an illustrative example of the structure of each of these main optimization problems. For example, the objective function in an VRP is usually to minimize the total cost, which is the sum of fixed plus variable cost. In the TOP, the objective is to maximize the rewards collected by vehicles when visiting customers. Many of the problems that arise in real-life T&M are dynamic in nature. These challenges make the problem troublesome.

Figure 6 shows a cross-problem analysis of the performance of AO algorithms when compared with the best-known solution (BKS), which might require minutes or even hours of computation, and the solution provided by a greedy heuristic—which, as the AO, usually requires less than a second of computation. The problems analyzed are: the uncapacitated facility location problem in an Internet of Vehicles context (UFLP-IoV) [7], the team orienteering problem (TOP) [122], the permutation flow-shop problem with deadlines and payoffs (PFSP-DP) [123], the vehicle routing problem (VRP) [90], the basic permutation flow-shop problem (PFSP) [90], the basic uncapacitated facility location problem (UFLP) [124], and the arc routing problem (ARP) [125].

Notice that, while the greedy heuristic typically offers solutions with a gap between $2\%$ and $8\%$ with respect to the BKS, the AO algorithm is capable of generating solutions with a gap lower than $2\%$ in most cases. This is a noticeable result if we take into account that these gaps are obtained in wall-clock times below one second.

7. Conclusions and Future Research

In this paper, we have reviewed IoV concepts and discussed the need for AO and DML algorithms to address vehicle networking problems in smart cities. IoV is becoming a predominant research topic due to the emergency of self-driving vehicles and the exponential development experimented in IoT and communication technologies, including 5G and newer developments. At the same time, this technology not only opens a wide range of possibilities in the field of urban transportation and mobility, but it also raises a series of optimization problems which are large-scale and NP-hard. In addition, these problems need to be solved in real-time. Moreover, and due to the dynamism of the traffic in any major city, the models need to be re-optimized every few minutes, which calls for the need of AO and DML algorithms.

Hence, the paper has reviewed the state-of-the-art in IoV main uses, putting special emphasis in protocol requirements and vehicle routing operational challenges. Further, the most frequent optimization tools employed so far to deal with these operational challenges have been discussed. AO algorithms have been introduced next as a combination of constructive heuristics, biased randomization techniques, and parallel computing. Some computational results, gathered from previously published articles, contribute to illustrate the potential of AO algorithms in terms of computing time and quality performance. These results cover some of the main optimization models in the literature of vehicle routing, including: vehicle routing problems, arc routing problems, team orienteering problems, and facility location problems. Distributed/decentralized training algorithms for ML models are discussed in detail. In particular, techniques such as collaborative and federated learning being promising solution methods for IoV problems are presented along with the important benefits and past studies.

Despite AO and DML approaches can be extremely useful to address vehicle networking problems in smart cities, their practical implementation is not trivial since it requires from the availability of reliable and valuable data on the status of the city transport and mobility systems. In addition, even when we consider AO and DML to be two of the most relevant methodologies in the context of IoV, the number of challenges related to ISVN is huge, and no single approach can solve all of them. Finally, IoV scenarios might vary depending on each city, and further analytical studies might be required in order to quantify the potential benefits that could be obtained with the proposed methodologies.

Some future work lines are described next: (i) the design of distributed AO algorithms that can run inside each vehicle instead of in a centralized computer system, and the development of the associated communication protocols in order to guarantee a fluent coordination among the vehicles; (ii) the use of edge computing as an intermediate solution that provides better response times than a centralized approach while, at the same time, facilitates coordination of vehicles at a local scale; (iii) the exploration of other optimization and machine learning methodologies that can constitute an alternative to the use of constructive heuristics and biased randomization techniques.