Spatial-Temporal Flows-Adaptive Street Layout Control Using Reinforcement Learning

Abstract

Complete streets scheme makes seminal contributions to securing the basic public right-of-way (ROW), improving road safety, and maintaining high traffic efficiency for all modes of commute. However, such a popular street design paradigm also faces endogenous pressures like the appeal to a more balanced ROW for non-vehicular users. In addition, the deployment of Autonomous Vehicle (AV) mobility is likely to challenge the conventional use of the street space as well as this scheme. Previous studies have invented automated control techniques for specific road management issues, such as traffic light control and lane management. Whereas models and algorithms that dynamically calibrate the ROW of road space corresponding to travel demands and place-making requirements still represent a research gap. This study proposes a novel optimal control method that decides the ROW of road space assigned to driveways and sidewalks in real-time. To solve this optimal control task, a reinforcement learning method is introduced that employs a microscopic traffic simulator, namely SUMO, as its environment. The model was trained for 150 episodes using a four-legged intersection and joint AVs-pedestrian travel demands of a day. Results evidenced the effectiveness of the model in both symmetric and asymmetric road settings. After being trained by 150 episodes, our proposed model significantly increased its comprehensive reward of both pedestrians and vehicular traffic efficiency and sidewalk ratio by 10.39%. Decisions on the balanced ROW are optimised as 90.16% of the edges decrease the driveways supply and raise sidewalk shares by approximately 9%. Moreover, during 18.22% of the tested time slots, a lane-width equivalent space is shifted from driveways to sidewalks, minimising the travel costs for both an AV fleet and pedestrians. Our study primarily contributes to the modelling architecture and algorithms concerning centralised and real-time ROW management. Prospective applications out of this method are likely to facilitate AV mobility-oriented road management and pedestrian-friendly street space design in the near future.

1. Introduction

The complete streets scheme is a mainstream engineering solution to improve road sharing for all road users [1,2]. It balances all users’ public right-of-way (ROW) and canalises road proportions according to respective travel demands [3,4]. A balanced ROW through the implementation of a complete street scheme could accommodate all modes of travel with rational road shares, an efficient operational environment and safe travel experiences [5,6].

Evidence shows that the complete streets scheme has considerably contributed to reducing road hazards, especially inter-modes traffic accidents, while maintaining relatively high transport efficiency [7,8]. However, their rigid and canalised thoroughfares have been criticised by a significant proportion of planners and geographers for the imbalanced ROW assignment [9]. They have a strong belief that the complete streets scheme particularly priorities motorised mobility while diminishing the shares for active travel modes. They demand that active modes of transport and street events should be granted larger road space shares compared with the status quo [10]. This appeal deeply roots in humanitarian urbanism and profoundly underpins a storm of movements that shifts supremacy from automobiles to non-motorised users within neighbourhoods or at critical public realms [11]. This reclaim of street space represents a force that is currently reshaping complete streets through measurements like traffic calming [12], temporarily or permanently occupying driveways [13,14].

Autonomous Vehicles (AVs) and Shared Autonomous Vehicles (SAVs) are emerging as the next solution to urban mobility. A substantial number of studies have predicted their potential to transform the conventions of travel patterns fundamentally [15]. Furthermore, a significant proportion of them share concerns on such changes as exogenous challenges faced with entire road networks and complete streets [16]. They fear that the induced demand by AVs transport could overload the streets of city centres during morning and evening commutes [17]. However, the positive effects include a considerable reduction in the throughput pressure of neighbourhood roads [18], suggesting opportunities to re-balance the ROW usage among road users. Simultaneously, Pick-Up and Drop-Off (PUDO) practices of SAVs [19], fast electric charging services [20], deployment of Roadside Units (RSUs) [21] and automated logistics [22] are likely to redefine the functions of streets in the AVs era. These new street norms would foster a revolution of the ongoing complete street schemes to be more inclusive, pedestrian-aware, and flexible.

The challenges, as mentioned earlier, call for a careful rethink of the state-of-the-art design protocols of streets and the development of novel techniques to manage the road space usage in a more innovative way. With the advancement of intelligent and connected road infrastructures, real-time control over the ROW could be considered as a new engineering approach. Furthermore, assisted with artificial intelligence (AI), road traffic management has experienced significant progress, particularly traffic signal control [23,24] and traffic operations control [25]. Despite these advancements, the optimal control models focusing on the assignment of ROW are seldom present in the literature. Thus, this research was proposed to contribute to this area.

To solve this optimal control problem, we have introduced a Reinforcement Learning (RL) method, namely a Deep Deterministic Policy Gradient (DDPG) algorithm for the real-time road space assignment to corresponding road users. The goal is to realise a traffic flow-responsive and pedestrian-friendly street layout by altering the ROW proportions assigned to driveways and sidewalks. A four-legged intersection and a 24-h AVs-pedestrians combined travel plan constitute the basic modelling settings. This DDPG algorithm incorporates the open-source microscopic traffic simulator, Simulation of Urban Mobility (SUMO), for retrieving the operational AVs’ and pedestrians’ dynamics, and the states of the road environment. In addition, both symmetric and asymmetric road layouts are simulated to measure the effectiveness of the proposed method.

The contribution of this study is four-folded. First, a novel RL-based approach is proposed and validated to address the real-time ROW assignment problem, benefiting the inclusiveness and efficiency of future streets. Second, the model coordinates the seemingly conflicting objectives of place-making and transport efficiency, combining urban planning and traffic engineering appeals. Third, our model is likely to be scalable concerning measuring simple road geometries, which are generally defined by limited edges, but potentially city-level road grids. Last, as a seminal building block of the Intelligent Transport System (ITS), this proposed method could further incorporate peer controlling technologies to accommodate potential challenges raised by the introduction of AVs mobility.

2. Literature Review

This section provides a systematic review of peer studies concerning three seminal aspects. First, we surveyed the complete streets scheme, which is the most widely implemented guide for street design practices. Second, we reviewed challenges facing the implementation of this scheme. Third, essential objectives concerning streets design and management are summarised.

2.1. Public Right-of-Way of Complete Streets Scheme

In transportation, the public ROW, or the ROW of road space, defines the legal right or the priority of specific types of road users to pass along a route through the street space [1]. These road users include not only motorised vehicles, but also vulnerable groups such as pedestrians and cyclists [2]. The purposes of balancing the ROW include improving traffic efficiency, engaging all modes of transport, and reducing potential inter-mode conflicts [6].

The complete streets scheme has risen as a mainstream engineering solution to balance the ROW [3,4]. It satisfies the basic demands of accommodating all road users with corresponding shares of space, but simultaneously canalise such space as per distinctive modes of travel [5,6]. The complete streets scheme principally comprises a driveway zone and a streetside zone. Figure 1 demonstrates four examples of its ROW plan.

Both the driveway and streetside can be further subdivided into different functional sections [26]. For instance, a driveway zone comprises several driving and curb lanes (Z${}_1$), and possibly a median (Z${}_2$). Meanwhile, a streetside sits between the driveway and private lands, primarily serving non-vehicular mobility and providing accessibility to venues, comprising cycle lanes (Z${}_5$) and sidewalks (Z${}_3$). It also includes a variety of road facilities in facility belts (Z${}_4$), and lively street activities in the front zones (Z${}_6$) [9,27].

Due to differences in street functions, locations and throughput capacities, the sectional widths vary significantly [9]. On one hand, roads can be categorised into four types according to their functions: commercial, residential (lanes, mews), landscape (boulevard, parkway) and trafficking. On the other hand, concerning their serving capacity, the ROW can be classed into four grades: main roads, secondary roads, branches, and laneways [28]. For instance, boulevards are landscape-functional main roads, and the commercial avenues are commercial-oriented main roads [29].

Based on a holistic survey of the state-of-the-art available complete streets scheme worldwide, we compared and summarised the following underlying principles of designing a street. First, the sidewalk should be planned no less than 1.5 m, and it is encouraged to be between 2 m and 2.5 m for new development residential streets or downtown commercial streets [27,28,29,30]. Second, the width of a driving lane is considered 3 m to 3.5 m for passenger cars, freight vans and trucks in urban areas. Third, a curb lane is suggested to bear a width of 3 m for on-street parking operations [27]. Fourth, the provision of cycle lanes can be flexible as it could occupy an independent lane with a width between 1.5 m to 2.5 m; Alternatively, driving lanes could accommodate those cycling demands. Fifth, the comprehensive facility belt should at least be assigned with 1.5 m to 2 m in width [27,28]. Finally, any street should secure a clear path in a width of 3.5 m to ensure the operations of emergency vehicles.

2.2. Challenges Facing Complete Streets Scheme

It is widely acknowledged that the complete streets scheme has achieved seminal contributions with regards to road safety and the operational efficiency of traffic flows [7,8]. However, these rigid and canalised patterns handicap their flexibility and resilience in responding long term’s endogenous and exogenous challenges facing road space.

The endogenous challenges emerge from the priority of usage between vehicles and the other modes of travel. A wide range of planners and geographers have criticised the complete street as ’incomplete plans’ concerning streets as public space [10]. They claimed that street events, pedestrians, e-scooter riders and cyclists should be granted more space than and priority to cars [11]. Temporary measurements, such as closure of driveways in a short period of time [13], and some permanent remedies, like traffic calming [12], shared road surface [31] have firmly responded to such appeal. The recent Covid-19 pandemic also fostered the reclaiming of driveways for non-vehicular traffic operations or as extensions of indoor activities [14]. To take placemaking and urban design into account, such flexibility of road space usage could potentially support diverse street activities and reinforce the public recognition of streets as public space [32].

The prominent exogenous challenges could be the deployment of Autonomous Vehicles (AVs) and Shared Autonomous Vehicles (SAVs) mobility [33]. It is expected that the future urban mobility and goods logistics could be replaced almost entirely by these disruptive modes of transport around 2040 to 2060 [34]. One of the early research jointly conducted by the Boston Consulting Group (BCG) and the World Economic Forum (WEF) found that with a moderate 60% of market penetration, AVs mobility might induce considerable trips to downtown areas during the morning and evening commute peaks. This could overload streets of city centres, raising travel costs by at least 5.5%, while broadly alleviating traffic in suburban neighbourhoods by 12% [18].

An increasing proportion of studies on AV transport demonstrates the disruptive impact of SAVs, which may transform our current transport into the Autonomous Mobility-on-Demand (AMoD) system [35]. By adopting SAVs, it is estimated that 16% of current vehicular fleet could suffice daily mobility [36]. While 85% of current off-street parking space land can be liberated [37], while frequent Pick-Up and Drop-Off (PUDO) events would require more curb parking areas to be installed and efficiently managed [19]. In addition, some emerging new road infrastructures, including the rapid charging facilities, may also disrupt the conventional street functions and demand for new spatial plans to accommodate new ROW desires [20,33].

These potential changes signal the urgency to revisit the current design protocols and renovate road management techniques. With the promising advancement of intelligent and connected road infrastructures, real-time optimal control over ROW might present a novel solution to this problem. Although new methods and algorithms have been developed, with some even tested on roads [38,39], those pioneering practices are still confined within limited road infrastructures, such as traffic signals and roadside units [21,40]. Moreover, the status-quo models and algorithms still fall far short in supporting the dynamic control of public ROW.

2.3. Performance Metrics Regarding Street Design and Management

Good street space is underpinned by objectives from a broad spectrum of domains [9]. In other words, the design and management of streets usually correspond to a multi-objective decision-making process given a collection of goals [41]. These domains include place-making, health and environmental, connectivity and accessibility, traffic efficiency, construction and maintenance, and safety [9,41], as summarised in Figure 2.

Regarding our study, we first treat users’ safety as the baseline. Namely, the requirement of collision-free was encoded as a priority in our model. Then, among the rest objectives, we approach to balance place-making and transport efficiency, hoping to coordinate traffic engineering and urban planning appeals. On the one hand, evidence proves that suffice the territory of sidewalks can effectively enhance the safety perception of pedestrians, contributing to more comfortable walking experiences, and simultaneously engaging street lives [9,10,12]. On the other hand, the operational efficiency represents a significant indicator measuring the primary trafficking performance of roads [42].

2.4. Summary of Literature Review

The essence of designing the road space is to engage all modes of transport and ensure their travel safety and traffic efficiency. As a prevalent solution to this engineering problem, the complete streets scheme canalises road space per distinctive modes of travel to accommodate all road users. However, despite the ensured traffic safety and efficiency, the complete streets scheme presents a lack of flexibility and resilience concerning challenges introduced by AV mobility. In addition, such a solution is yet balanced enough to echo the demands for truly pedestrian-friendly road space.

Traffic engineers, urban designers, and planners need to cooperate in upgrading the current street design protocols and renovating road management techniques for a flexible, human-oriented ROW control scheme. To the best of our knowledge, the status quo literature probed into limited solutions to traffic signals control problem and roadside units control problem while offering no clear methodology, either theoretically or practically, to the ROW control problem concerned by our study.

3. Preliminaries of Reinforcement Learning

3.1. Reinforcement Learning and Markov Decision Process

Reinforcement Learning (RL) represents one of the cutting-edge branches in Artificial Intelligence (AI) methods. It includes a wide range of popular algorithms that address optimal control problems via a trial-and-error procedure [43]. Concretely, intelligent agents embedded with RL controllers learn optimal policies to interact with the environment and take actions. They retrieve feedback and update their decision-making machinery for higher scored movements in the subsequent attempts [44]. RL has been widely applied in solving real-world problems such as robot control, dispatch management problem, the Travel Salesman Problem (TSP) and production scheduling problem [45]. Furthermore, it has been intensively practised in transportation studies for traffic signal control [23,24,38], vehicles routing [46], movements control [47] and traffic operations control [48].

By convention, a control problem could be defined using a Markov Decision Process (MDP), denoted as a tuple $M=<N,\mathbf{S},\mathbf{A},P,R,\gamma >$. N is the number of agents in the MDP, and an agent is indexed by $n=[1,2,\dots N]$. Agents interact with the stochastic environment and observe the states. Let $\mathbf{S}=S^1\times S^2\times \dots S^n\dots S^N$ define the joint state space, where each state element indicates a high-dimensional state space. Let T be the time horizon, the component state-space $S^n$ comprises individual observation $s_t^n$ of agent n at time step $t=1,2\dots T$, herein $s_t^n\in S^n$. Similarly, $\mathbf{A}=A^1\times A^2\times \dots A^n\dots A^N$ defines the joint action space. A state-action space $\tau$ describes a trajectory of agent-environment interaction, which is expressed as $s_1\times a_1\times s_2\times a_2\dots \times s_T\times a_T$. Given a trajectory $\tau$, P defines a stationary transition probability of being in a state s given an action a at time t and transforming to the next state $s^{{}^{\prime }}$ at time $t+1$. Let $r=s\times a\to \mathbb{R}$ defines the immediate reward from the environment once the agent observed state s and take the action a. The controlling objective is to maximise the cumulative reward of all agents $R={\sum }_{n=1}^N{\sum }_{t^{{}^{\prime }}=t}^T{\gamma }^{t^{{}^{\prime }}-t}{r}_{n,t^{{}^{\prime }}}$ from $t^{{}^{\prime }}$ to the temporal limitation. Here, $\gamma \to (0,1)$ is a discounted factor representing the decaying contribution of the expected immediate reward.

3.2. Deep Deterministic Policy Gradient (DDPG) Algorithm

Policy Gradient (PG) algorithms represent a vital branch of RL methods that address optimisation problems characterised by a continuous action space [49,50]. Let $\pi$ define a stochastic policy that executes an action a observing a state s, and ${\pi }_{\theta }\left(a\right|s)=\mathbb{P}\left(a\right|s;\theta )$ represents the probabilistic distribution of taking this action under such policy.

The Stochastic Policy Gradient (SPG) algorithm is an early representative of PG algorithms. The discounted future reward of SPG at time t with a discount factor $\gamma$ is expressed as $G_t$ following Equation (1). Whereas Equations (2)–(4) present the state value function, state-action function and advantage function of SPG respectively. SPG conducts a gradient ascend approach to optimise the policy parameter $\theta$, which is expressed in Equation ().

$$\begin{array}{ccc}\hfill G_t& =\sum _{t^{{}^{\prime }}=t}^T{\gamma }^{t^{{}^{\prime }}-t}{R}_{t^{{}^{\prime }}}\hfill & \hfill \end{array}$$

(1)

$$\begin{array}{ccc}\hfill V^{\pi }\left(s_t\right)& =\mathbb{E}\left[G_t\right|s_t=s]\hfill & \hfill \end{array}$$

(2)

$$\begin{array}{ccc}\hfill Q^{\pi }(s,a)& =\mathbb{E}\left[G_t\right|s_t=s,a_t=a]\hfill & \hfill \end{array}$$

(3)

$$\begin{array}{ccc}\hfill A^{\pi }(s,a)& =Q^{\pi }(s,a)-V^{\pi }\left(s\right)\hfill & \hfill \end{array}$$

(4)

$$\begin{array}{ccc}\hfill {\nabla }_{\theta }J\left(\theta \right)& \approx \mathbb{E}\left[A^{\pi }(s,a){\nabla }_{\theta }lo{g}_{{\pi }_{\theta }}\left(a\right|s)\right]\hfill & \hfill \end{array}$$

(5)

The deterministic policy gradient (DPG) method approximates the optimal policy using a deterministic policy $\mu \left(s\right)=a$ rather than the random approach adopted by SPG. In other words, DPG is a ’limiting analogue’ of the stochastic counterpart, according to its theorem [51]. Let $\sigma$ indicate the deviation corresponding to the probabilistic distribution $\mathbb{P}\left(a\right|s;\theta )$. As Equation (6) expresses, DPG is the condition where $\sigma =0$. DPG can either be modelled in an on-policy or off-policy way. An on-policy DPG usually requires a large number of samples for learning [52]. Instead, an off-policy DPG is self-sufficient regarding training samples because of the embedded perturbation mechanism.

$$\begin{array}{ccc}\hfill {\pi }_{{\mu }_{\theta },\sigma =0}\left(a\right|s)& =a\hfill & \hfill \end{array}$$

(6)

The Deep Deterministic Policy Gradient (DDPG) algorithm is a model-free off-policy DPG, which inherits features from both DPG and Deep Q-network (DQN) [53]. Namely, it embeds deep neural networks to enable an end-to-end learning ability [54]. The DDPG has improved significantly from three aspects in comparison with DPG and DQN. First, an exploration policy that generates random noises is implemented in DDPG to suffice exploration. Second, an experience replay buffer trains neural network parameters $\theta$ in an off-policy manner. Third, conventional DQN updates the target action-value function $\widehat{Q}$ through periodically coping from the current action-value function Q. In contrast, DDPG updates its parameters of both target networks at each iteration and proportionately at a soft update rate $\eta$.

4. Methodology

In this section, we first formulate this dynamic ROW assignment problem as an optimal control problem. Our proposed solution model combines the SUMO traffic simulator and a DDPG algorithm. Then, a detailed explanation is provided covering the state, action, reward, experience replay buffer and essential parameters of the actor-critic architecture of DDPG.

4.1. Modelling Formulation

Consider a simple network represented by a bi-directed graph $\mathcal{G}=<N,E>$, where $w_e$ denotes the width of an edge $e\in E$ and $b_e$ indicates the width of a facility belt. Let ${\beta }_e$ be the proportion of a driveway. As such, the width of the driveway equals ${\beta }_e{w}_e$, whereas the counterpart streetside has a width of ($1-{\beta }_e)w_e$. Figure 3 highlights the spatial relationship among these variables, given a ROW plan of an edge. Provided a consecutive pair-wise AV traffic demands $N{V}_t$ and pedestrian travel demand $N{T}_t$ at $t\in [0,T]$, the problem is to optimise the driveway ratio ${\beta }_{e,t}$ of discrete edge and at each time slot, to improve travel efficiency of road users and a higher ratio of sidewalks.

Table 1. Notations of Variables and Parameters.

	Notation	Specification	Value
	E	Set of edges
	V	Set of vehicles
	P	Set of pedestrians
	OD	Set of origin-destination pairs
Sets	V${}_e$	Set of vehicles on edge e	V${}_e\subset$ V
	P${}_e$	Set of pedestrians on edge e	P${}_e\subset$ P
	V${}_{od}$	Set of vehicles with $od$ assignment	V${}_{od}\subset$ V
	P${}_{od}$	Set of pedestrians with $od$ assignment	P${}_{od}\subset$ P
	CF${}_v$	following cars of vehicle v	$v\in V$
	e	an edge	$e\in$ E
	v	a vehicle	$v\in$ V
Indices	p	a pedestrian	$p\in$ P
	t	a time slot	$t\in [0,T]$
	$od$	an origin-destination pair	$od\in$ OD
	$l_e$	number of lanes on edge e	${\mathbb{Z}}_{+}^{*}$
	$f_{od,v,t}$, $f_{od,p,t}$	unit travel demand of a vehicle/a pedestrian	$v\in$ V, $p\in$ P
	$\overrightarrow{{\chi }_v}$, $\overrightarrow{{\chi }_p}$	position of a vehicle/ a pedestrian	$v\in$ V, $p\in$ P
	${\overrightarrow{d}}_e$	direction of edge e	-
	$h{s}_{u,v}$	longitudinal distance between $u,v$	$u,v\in$ V${}_e$
Variables	$c_{v,t}$, $c_{p,t}$	velocity of a vehicle/ a pedestrian	$v\in$ V, $p\in$ P
	${\dot{c}}_{v,t}$, ${\dot{c}}_{p,t}$	acceleration of a vehicle/ a pedestrian	$v\in$ V, $p\in$ P
	$\lambda \left(t\right)$	arrival rate at t	$t\in [0,T]$
	$\zeta \left(t\right)$	departure rate at t	$t\in [0,T]$
	${\beta }_{e,t}$	(Decision Variable) driving lane ratio of e at t	${\mathbb{R}}_{+}$
	$w_e$	edge width of edge e	${\mathbb{R}}_{+}$
	$b_e$	width of a facility belt of edge e	${\mathbb{R}}_{+}$
	k	length of a time slot	30 $\mathrm{m}\mathrm{i}\mathrm{n}$
	T	Simulation time period	3600 s
	$NE$	number of edges	8
	$N{V}_t$	vehicular travel demand at t	${\mathbb{Z}}_{+}$
	$N{P}_t$	pedestrian travel demand at t	${\mathbb{Z}}_{+}$
	$gs$	minimal space gap between pedestrians	0.25 m
Parameters	$DI{S}_{od}$	distance between origin o and destination d	${\mathbb{R}}_{+}$
	$\varphi$	reward amplifier	1000
	$len$	vehicle length	4.5 m
	$v_{max}$	maximum speed of AV	13 m/s
	$p_{max}$	maximum speed of pedestrian	1.2 m/s
	$\psi$	speed deviation	0.05
	h	time headway	0.6 s
	$ac{c}^{veh}$	maximum acceleration of vehicles	2.6 m/s2
	$ac{c}^{ped}$	maximum acceleration of a pedestrian	0.3 m/s2

demonstrates the notation conventions applied for the formulation of this optimal control problem. Following previous studies [55,56,57,58], we consider the following physical parameters for the simulation: the maximum operational speed and maximum acceleration of an AV are 13 m/s and 2.6 m/s², respectively. The vehicular length is set at 4.5 m. Pedestrian walking speed is limited to under 1.2 m/s to ensure the testing scenario for intersections.

Our objective function (7) is composed of three sub-objectives which are expressed in Equations (8)–(13). Namely, at each time slot t, the objective maximises reward from vehicular traffic efficiency $R_t^{veh}$, pedestrian traffic efficiency $R_t^{ped}$ and sidewalk ratio $R_t^{side}$. Here, an amplifier $\varphi$ is applied to scale the total objective $R_t$.

Equation (8) presents that $R_t^{veh}$ equals the ratio between the free-flow travel time $H^{veh}$ and the actual travel cost $H^{veh}+D_t^{veh}$, where $D_t^{veh}$ indicates the total traffic delay of the fleet in time slot t. Equation (9) estimates that $H^{veh}$ as a sum of travel costs of all AVs operating at their respective maximum speed in the system. Equation (10) expresses that the traffic delay $D_t^{veh}$ equals the gap between the integral of the arrival rate and that of the departure rate. In this equation, $\lambda \left(t\right)$ and $\zeta \left(t\right)$ represent functions of the arrival rate and the departure rate controlled by time t given $od\in OD$.

The calculation of the pedestrian traffic efficiency $R_t^{ped}$ follows a similar rule as that of $R_t^{veh}$, as described in Equation (11). The individual free-flow walking time $H^{ped}$ and pedestrian traffic delay $D_t^{veh}$ are acquired following Equations (12) and (13). Following Equation (14), the third sub-objective component $R_t^{side}$ maximises the cumulative sidewalk ratio, namely $1-{\beta }_{e,t}-\frac{be}{w_e}$, at time t of edge e. Here, $b_e$ indicates the width of the facility belt.

$$\begin{array}{cccc}\hfill \mathbf{maximise}& R_t=\varphi (R_t^{veh}+R_t^{ped}+R_t^{side})\hfill & \hfill \forall t\in [0,T]& \hfill \end{array}$$

(7)

$$\begin{array}{cccc}\hfill \mathbf{where}& R_t^{veh}=\frac{H^{veh}}{H^{veh}+D_t^{veh}}\hfill & & \hfill \end{array}$$

(8)

$$\begin{array}{cccc}& H^{veh}=\sum _{od\in OD}\sum _{v\in V_{od}}\frac{DI{S}_{od,v}}{v{m}_v}\hfill & & \hfill \end{array}$$

(9)

$$\begin{array}{cccc}& D_t^{veh}=\sum _{od\in OD}{\left({\int }_{k(t-1)}^{kt}\lambda \left(t\right)dt-{\int }_{k(t-1)}^{kt}\zeta \left(t\right)dt\right)}_{od}^{veh}\hfill & & \hfill \end{array}$$

(10)

$$\begin{array}{cccc}& R_t^{ped}=\frac{H^{ped}}{H^{ped}+D_t^{ped}}\hfill & & \hfill \end{array}$$

(11)

$$\begin{array}{cccc}& H^{ped}=\sum _{od\in OD}\sum _{p\in P_{od}}\frac{DI{S}_{od,p}}{v{m}_p}\hfill & & \hfill \end{array}$$

(12)

$$\begin{array}{cccc}& D_t^{ped}=\sum _{od\in OD}{\left({\int }_{k(t-1)}^{kt}\lambda \left(t\right)dt-{\int }_{k(t-1)}^{kt}\zeta \left(t\right)dt\right)}_{od}^{ped}\hfill & & \hfill \end{array}$$

(13)

$$\begin{array}{cccc}& R_t^{side}=\frac{1}{NE}\sum _{e\in E}(1-{\beta }_{e,t}-\frac{be}{w_e})\hfill & & \hfill \end{array}$$

(14)

The problem is constrained by Equations (15)–(24). Constraint (15) requires that the distribution of operational speed of the fleet to follow a normal distribution, which has a mean of $v_{max}=$ 13 m/s multiplying a speedFactor = 1.0, and a deviation $\psi$ = 0.05. Constraint (16) sets the upper bound of the real-time speed of an AV, expressed as $c_{v,t}$, which should not exceed the designated operational rate. Let ${\dot{c}}_{v,t}$ indicate the acceleration of an AV at t, Constraint (17) limits this acceleration as below $ac{c}^{veh}=$ 2.6 m/s². Let (${\overrightarrow{\chi }}_v$) and (${\overrightarrow{\chi }}_p$) denote the coordinates of an AV and pedestrian respectively, where $v\in$ V and $p\in$ P. Constraint (18) demands the minimal longitudinal distance of the proceeding car $u\in C{F}_v$ to suffice the clearance gap $hs$ requirement. Constraint (19) further estimates such clearance gap using time headway h, the mean velocity of the two vehicles and vehicle length $le{n}_u$.

$$\begin{array}{cccc}\hfill \mathbf{where}& v{m}_v\sim \mathcal{N}(v_{max},\psi )\hfill & \hfill \forall v\in V& \hfill \end{array}$$

(15)

$$\begin{array}{cccc}& 0\le c_{v,t}\le v{m}_v\hfill & \hfill \forall t\in [0,T]& \hfill \end{array}$$

(16)

$$\begin{array}{cccc}& {\dot{c}}_{v,t}\le ac{c}^{veh}\hfill & & \hfill \end{array}$$

(17)

$$\begin{array}{cccc}& \mid {\overrightarrow{\chi }}_v-{\overrightarrow{\chi }}_u\mid \cos [({\overrightarrow{\chi }}_v-{\overrightarrow{\chi }}_u),{\overrightarrow{d}}_e]\le hs\hfill & \hfill \forall v\in V_e,u\in C{F}_v& \hfill \end{array}$$

(18)

$$\begin{array}{cccc}& hs=\frac{h}{2}(c_{v,t}+c_{u,t})-le{n}_u\hfill & \hfill \forall v\in V_e,u\in C{F}_v& \hfill \end{array}$$

(19)

We apply similar constraints on the simulation of pedestrian dynamics. Concretely, Constraint (20) regulates that the real-time velocity of a pedestrian $c_{p,t}$ be bounded by the maximum speed $p_{max}=$1.2 m/s. Constraint (21) ensures that the acceleration of a pedestrian at t should not exceed $ac{c}^{ped}=$0.3 m/s². In addition, Constraint (22) defines a minimal inter-person space as the norm of the vector between two pedestrians $p,q$, in which $gs$ equals 0.25 m in this study.

$$\begin{array}{cccc}& 0\le c_{p,t}\le p_{max}\hfill & \hfill \forall p\in P,t\in [0,T]& \hfill \end{array}$$

(20)

$$\begin{array}{cccc}& {\dot{c}}_{p,t}\le ac{c}^{ped}\hfill & \hfill \forall p\in P_e& \hfill \end{array}$$

(21)

$$\begin{array}{cccc}& \mid {\overrightarrow{\chi }}_p-{\overrightarrow{\chi }}_q\mid \ge gs\hfill & \hfill \forall p,q\in P,p\ne q& \hfill \end{array}$$

(22)

The hourly flow rates of AV transport and pedestrians are generated in alignment with the bimodal travel time distributions, following the description in [59]. A travel plan jointly synthesises pedestrian trips and AVs trips at the OD-pair level in 48-time slots of a day. Let $N{V}_t$ and $N{P}_t$ denote the vehicular and pedestrian travel demands per OD pair at time t. The sizes of AV fleets ${\sum }_{od}{\sum }_v{f}_{od,v,t}$ and pedestrians ${\sum }_{od}{\sum }_p{f}_{od,p,t}$ subject to corresponding travel plans, as expressed in Constraints (23) and (24).

$$\begin{array}{c}\sum _{od\in OD}\sum _{v\in V_{od}}{f}_{od,v,t}=N{V}_t\hfill \end{array}$$

(23)

$$\begin{array}{cc}\sum _{od\in OD}\sum _{p\in P_{od}}{f}_{od,p,t}=N{P}_t\hfill & \end{array}$$

(24)

Constraint (25) describes the boundaries assigned to the decision variable. Its lower limit represents the edging case where only one emerging lane is reserved, and the upper bound ensures the extra provision of at least 1.5 m sidewalk. Constraint (26) determines the number of lanes $l_e$ regarding ${\beta }_e$ and $w_e$. The final condition corresponds to the case of the multi-driving lane.

$$\begin{array}{ccc}& 3\le {\beta }_e\le \frac{w_e-b_e-1.5}{w_e}\forall e\in E\hfill & \hfill \end{array}$$

(25)

$$\begin{array}{ccc}& l_e=\left\{\begin{array}{cc}0,\hfill & \hfill if{\beta }_e{w}_e<3\\ 1,\hfill & \hfill if3\le {\beta }_e{w}_e<6.5\\ \lfloor ({\beta }_e{w}_e-3)/3.5\rfloor +1,\hfill & \hfill otherwise\end{array}\right.\hfill & \hfill \end{array}$$

(26)

4.2. SUMO Traffic Simulation Incorporated DDPG Modelling Framework

We proposed a SUMO-incorporated Deep Deterministic Policy Gradient model to address this optimal control problem, namely the SUMO-DDPG. Figure 4 presents the modelling framework, which essentially comprises two interactive components: a SUMO traffic simulator and a DDPG controller. Initially, SUMO generates network configurations using default ROW settings. It also generates discrete vehicular and pedestrian trips according to the pre-defined travel plans. Then, we use the Traffic Control Interface (TraCI), a SUMO built-in API, to calibrate road geometries at each time slot and acquire states of all AVs, pedestrians and environments.

The DDPG algorithm comprises an actor-critic model and an experience replay buffer. First, the actor-critic model consists of four neural networks: an actor online network $\mu \left(\theta \right)$, an actor target network ${\mu }^{{}^{\prime }}\left({\theta }^{{}^{\prime }}\right)$, a critic online network $Q\left(\omega \right)$ and a critic target network $Q^{{}^{\prime }}\left({\omega }^{{}^{\prime }}\right)$. Here, we use $\theta$, ${\theta }^{{}^{\prime }}$, $\omega$ and ${\omega }^{{}^{\prime }}$ to indicate the parameter sets of these four neural networks, respectively.

This DDPG is trained in $EP=$ 150 episodes and at the edges level, meaning discrete edges make individual decisions. One training episode comprises a number of time slots bounded by the starting time slot and $T=$ 3600 s. Each time slot is equivalent to a length of $k=30$ min in the real world.

Meanwhile, the DDPG controller interacts with SUMO via TraCI at an interval of $I=36$ s, namely 50 visits in a single slot. During the interactions, a set of edges evaluates the environment state ($S_t$), executes a collection of individual actions ($A_t$), receives a joint reward ($R_t$), and observes a future environment state ($S_{t+1}$). This information is then utilised to update the neural network parameters of the actor-critic model.

4.3. DDPG Algorithm Structure

The following subsection details the state, action, reward, experience replay buffer, and parameters applied for both actor and critic networks.

Table 2. Hyperparameters of SUMO-DDPG Model.

Notation	Specification	Value
$OU$	Ornstein–Uhlenbeck noise	-
$\delta$	mean rate of noise regression	0.15
$\sigma$	standard deviation of noise distribution	0.2
$ϵ$	mean of noise distribution	0
$\eta$	soft update parameter	0.005
$\gamma$	discount factor	0.99
B	replay buffer capacity	100,000
m	mini-batch size	64
$EP$	Episodes	150
I	Intervals per Step	36 s

lists the key hyper-parameters concerned by our SUMO-DDPG model.

4.3.1. State

The observed state includes the number of operational AVs and pedestrians on each edge. Let $S_e$ denote this two-dimensional state vector per edge, $S_e=s_e^{veh},s_e^{ped}$, whereas $s_e^{veh}$ and $s_e^{ped}$ indicate the real-time operational numbers of AVs and pedestrians on edge e. Thus, at a time slot t, the joint state space is defined as $S_t=[S_{1,t},S_{2,t}\dots S_{e,t}\dots S_{NE,t}]$.

4.3.2. Action

The action taken by an edge e at time step t is denoted as ${\beta }_{e,t}$, representing the proportion of the link occupied by the driveway. Note that the action space of ${\beta }_{e,t}$ is continuous in a range of $[3,\frac{w_e-b_e-1.5}{w_e}]$. Thus, the joint action space for the whole system is expressed using a $NE$-dimensional vector $A_t=[{\beta }_{1,t},{\beta }_{2,t}\dots {\beta }_{e,t}\dots {\beta }_{NE,t}]$. Equation (27) expresses that the actor online network determines a deterministic action $A_t^{{}^{\prime }}$ following the policy ${\mu }_{\theta }$.

$$\begin{array}{cccc}\hfill A_t^{{}^{\prime }}& ={\mu }_{\theta }\left(S_t\right|\delta )\hfill & & \hfill \end{array}$$

(27)

Equation (28) presents an Ornstein–Uhlenbeck process that generates a time-dependent noise to sufficiently explore the action space. By convention, it usually comprises a Gauss–Markov procedure and a white noise perturbation [60]. The Gauss–Markov noise distribution is defined by its mean $ϵ$, the regression rate to its mean $\delta$ and a standard deviation $\sigma$. The white noise is modelled as $\mathcal{N}(ϵ,0)$, which describes a normal distribution with a mean of $ϵ$ and a standard deviation of 0. Following that, Equation (29) expresses the eventual joint action $A_t$ taken by edges, which equals the sum of the deterministic part and the value of $OU$ noise.

$$\begin{array}{cccc}\hfill O{U}_t& =\delta (ϵ-O{U}_{t-1})+\sigma \mathcal{N}(ϵ,0)\hfill & & \hfill \end{array}$$

(28)

$$\begin{array}{cccc}\hfill A_t& =A_t^{{}^{\prime }}+O{U}_t\hfill & & \hfill \end{array}$$

(29)

Figure 5 presents a record of the obtained Ornstein–Uhlenbeck noises from eight tested discrete edges (green dots) and their average value per episode (blue squares). The horizontal red line highlights that mean noises $\overline{O{U}_t}$ fluctuate around 0.

4.3.3. Reward

A reward quantifies the feedback from the environment to evaluate an executed action [44]. In this current study, this reward is estimated using the value of $R_t$ following specific rules as explained in Formula (30). This formula regulates that this reward is 0 if more than one of the following illegal occasions happens: (1) the value assigned for decision variable ${\beta }_{e,t}$ exceeds either limit; (2) uncompleted trips due to disconnected routes. For instance, if a person could not complete the assigned trip from o to d due to an unconnected sidewalk-crossing system, then this reward equals 0.

$$R_t=\left\{\begin{array}{cc}0,\hfill & \hfill \exists {\beta }_e\notin [3,\frac{w_e-b_e-1.5}{w_e}],\forall e\in E\\ 0,\hfill & \hfill \exists f_{od,v}=0,\forall od\in OD,v\in V,e\in E\\ \varphi (R_t^{veh}+R_t^{ped}+R_t^{side}),\hfill & \hfill otherwise,\forall t\in [0,T]\end{array}\right.$$

(30)

4.3.4. Experience Replay Buffer

An experience replay buffer stores past transition tuples $<S_t$, $A_t$, $R_t$, $S_{t+1}>$ for further off-line learning. A random m-sized mini-batch is randomly sampled from the buffer to feed both target networks in the off-line learning process. Then a transition trajectories for learning is denoted as $<S_i$, $A_i$, $R_i$, $S_{i+1}>$, where $i\in [0,M]$ index samples of mini-batch.

4.3.5. Actor Networks

The actor online network approximates to the optimal policy ${\mu }_{\theta }$ by minimising the gradient loss of $\theta$. The corresponding loss function $\mathcal{J}\left(\theta \right)$ and the optimisation function for updating the gradients ${\nabla }_{\theta }\mathcal{J}$ are described in Equations (31) and (32) respectively.

$$\begin{array}{cccc}\hfill \mathcal{J}\left(\theta \right)& =-\frac{1}{M}\sum _{i\in [0,M]}Q(S_i,A_i|\theta )\hfill & & \hfill \end{array}$$

(31)

$$\begin{array}{cccc}\hfill {\nabla }_{\theta }\mathcal{J}& \approx \frac{1}{M}\sum _{i\in [0,M]}{\nabla }_{a}Q(S_i,A_i|\omega ){\nabla }_{\theta }\mu \left(S_i\right|\theta ))\hfill & & \hfill \end{array}$$

(32)

The actor target network updates the network parameter ${\theta }^{{}^{\prime }}$ using a soft update strategy controlled by a specific parameter $eta$, following Equation (33).

$$\begin{array}{cccc}\hfill {\theta }^{{}^{\prime }}& :=\eta \theta +(1-\eta ){\theta }^{{}^{\prime }}\hfill & & \hfill \end{array}$$

(33)

4.3.6. Critic Networks

The target part of critic networks estimates a target Q value $y_i$. Then, following the Bellman Equation [61], it sums the sampled reward $R_i$ and a discounted future state-action values using the network policy parameter ${\omega }^{{}^{\prime }}$ and a discount factor $\gamma$, as expressed in Equation (34). In parallel, the critic online network first calculates the Q value and then updates parameter $\omega$ by minimising the gradient loss between $y_i$ and this Q value. Equations (35) and (36) express this loss function $\mathcal{L}\left(\omega \right)$ and gradient update approximation ${\nabla }_{\omega }\mathcal{L}$ respectively.

$$\begin{array}{cccc}\hfill y_i& =R_i+\gamma Q^{{}^{\prime }}(S_{i+1},A_{i+1}|{\omega }^{{}^{\prime }})\hfill & \hfill \forall i\in [0,M]& \hfill \end{array}$$

(34)

$$\begin{array}{cccc}\hfill \mathcal{L}\left(\omega \right)& =\frac{1}{M}\sum _{i\in [0,M]}{\left[(y_i-Q(S_i,A_i|\omega ))\right]}^2\hfill & \hfill \forall i\in M& \hfill \end{array}$$

(35)

$$\begin{array}{cccc}\hfill {\nabla }_{\omega }\mathcal{L}& =\frac{1}{M}\sum _{i\in [0,M]}(y_i-Q(S_i,A_i|\omega )){\nabla }_{\omega }Q(S_i,A_i|\omega )\hfill & \hfill \forall i\in M& \hfill \end{array}$$

(36)

The critic target network updates its parameter ${\omega }^{{}^{\prime }}$ controlled by $\eta$ following Equation (37).

$$\begin{array}{cccc}\hfill {\omega }^{{}^{\prime }}& :=\eta \omega +(1-\eta ){\omega }^{{}^{\prime }}\hfill & & \hfill \end{array}$$

(37)

4.3.7. Keras Neural Networks

Both actor and critic networks were implemented in Keras. An actor network comprises two hidden layers, which inputs a $(NE,2)$-sized observed states layer and outputs $NE$$(1,1)$-dimensional actions. The rectified linear activation function (ReLU) and tangent activation function (TanH) are implemented as calculation rules for the input and output layers respectively.

A critic network has six hidden layers in total. Initially, a $(NE,2)$-sized state layer and $NE$$(1,1)$-dimensional action layer are input as dual-sourced layers. Then, both layers are activated using ReLU and later concatenated as a joint layer. Finally, the eventual output layer is featured in the size of $(1,1)$ for the control task of each edge.

5. Model Training and Results

5.1. Specification of Travel Plans and Testing Case

Figure 6 demonstrates the OD-pair based travel demands at 48-time slots, with error marks representing stochastic perturbations (±10 puc/h or person/h) based on their respective mean demands.

The mean OD-pair level travel demands of AVs and pedestrians are 114 pcu/h and 21 person/h respectively. Concretely, two peaks of AVs traffic flow appear at 07:30 (175 pcu/h) and 18:00 (140 pcu/h). Meanwhile, pedestrian flow rates reach the highest demands at 07:30 (41 person/h) and 18:00 (40 person/h).

The prototypical road layout for testing originates from a four-legged intersection. This road layout has four 100 m roads comprising eight connected edges, $NE=8$. We have further calibrated this prototype into one symmetric layout case and an asymmetric case to ensure the usability of our proposed SUMO-DDPG model under varied geometric conditions.

The symmetric case has homogeneous edge widths, namely [16, 16, 16, 16, 16, 16, 16, 16 (m)], whereas their facility belt are in the widths of [1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5 (m)]. For the asymmetric case, the edge widths are [14, 14, 16, 16, 18, 18, 14, 14 (m)], which are heterogeneously configured. The widths of their facility belts are [1.5, 1.5, 1.5, 1.5, 2, 2, 1.5, 1.5 (m)]. Figure 7 demonstrates the configuration of two settings in SUMO simulation environment.

Different geometric settings lead to distinctions in the upper and lower bounds of ${\beta }_{e,t}$. Despite the variations, a uniform randomness seed is shared throughout the training course over two cases. This arrangement ensures that travel plans, perturbation noises, and actor-critic models’ initial parameters are identical for both cases.

5.2. Training Performance of SUMO-DDPG

Figure 8 demonstrates the training performances of learning curves obtained from the two tested cases. The X-axis index 150 training episodes, while Y-axis represents the edge-level accumulative average reward. The blue and green curves log the generally ascending trends of such rewards obtained from the symmetric and asymmetric cases.

The reward in the symmetric case has increased by 10.39% since the initial episode. It reaches the peak of 1700.65 at an incremental rate of +1.07/ep. The learning curves of three training stages–the early stage (Ep. 0–49), middle stage (Ep. 50–99) and later stage (Ep. 100–149), present distinctive convergence patterns.

The early stage records a significant surge from 1540.61 to 1632.82, leaping a gap of 92.20 with an incremental rate of +1.84/ep. Its mean and standard deviation are 1599.26 and 22.45, indicating a rapid convergence process. In the middle stage, the reward gap drops by 68.67, from 1624.97 to 1693.63, while its standard deviation slightly rises to 24.30. Additionally, the mean reward value situates at 1660.21, with the incremental rate declines to +1.37/ep. In the last stage, the reward pattern reaches a plateau marked by the lowest standard deviation (6.15), the even narrowest mean value (18.42) and the slowest incremental rate +0.37/ep. This stage-wise declining tendency suggests the approximation to the optima.

Accordingly, the learning curve of the asymmetric case demonstrates similar convergence patterns. It has a general 9.09% in reward surge at an average incremental rate of +0.96/ep. The lowest reward (1590.73) is obtained at Ep.2, while the highest reward (1735.33) at Ep. 100. Examined stage-wise, the corresponding average reward gaps of the three stages are 95.60, 60.92 and 15.48, whereas the incremental rates are +1.91/ep, +1.22/ep and +0.31/ep. Their standard deviations are 30.24, 22.61, and 4.5. These figures signal that the obtained reward is approximate to the optima, which is consistent with that of the symmetric case.

In general, both learning curves display synchronised and incremental patterns throughout their independent training courses. These patterns evidence the effectiveness of the model as input arbitrary road geometries.

The training pattern of the symmetric case has a slightly lower episodic reward (40.01/ep) than its asymmetric counterpart, indicating a moderately better convergence performance of our model on optimising the asymmetric road layout. However, rewards of the symmetric case outperform the asymmetric ones concerning the overall reward growth rate (+1.30% higher) and the stage-wise incremental rate (+0.11/ep higher). Furthermore, despite the differences, the rewards gap of the two cases narrows down from 69.57 at Ep.0 to 38.89 at the last episode, with a mean gap of 40.01. These facts evidence that our proposed SUMO-DDPG model could effectively optimise ROW plans under symmetric and asymmetric road layout conditions.

5.3. Improved ROW Assignment Strategies

As another performance metric of our model, we would like to understand to what extent has our proposed model optimised the executed actions throughout the training course.

Figure 9 contains eight graphs to demonstrate the transformations of ROW decisions for each edge between their early stage and later stage. The X-axis indicates the time slots of a day, and the Y-axis shows the action values, namely the road proportion assigned for driveways ($\beta e,t$). The early-stage actions are highlighted in grey dots, while their later counterparts are in black. We use blue and red vertical bars to indicate the tendency of declining or rising in action values.

Consequently, a majority (90.16%) in action values decline, with only 9.84% showing a slight increase. The time slot-wise differences are not apparent due to these principal trends of reduction. Combined with the rising patterns of learning curves, the declination in action values suggests that our model can effectively alter the road proportions assigned to driveways to sidewalks. Such efforts in re-balancing the ROW between the driveways and sidewalks reinforce as controllers receive higher rewards from the environment feedback.

We further examined detailed transformation in distributions of actions. Figure 10 demonstrates the density distributions and their corresponding Gaussian Probability Density Functions (PDF). Optimised by our model, the actions’ density distributions get flattened throughout the training course. Their mean value transforms from 0.78 in the early stage to 0.69, shifting away from the upper limit and approaching towards the lower bound.

The road space assigned to the driveway reduces by 9% on average, namely 1.26 m, 1.44 m or 1.62 m corresponding to edges in widths of 14 m, 16 m or 18 m respectively. Additional space assigned to sidewalks is beneficial to relieve the pressure of pedestrian traffic while potentially encouraging street activities. More street activities are likely to induce demand for pedestrian flows at certain levels, requiring control models, like ours, to be deployed to manage road space assignment dynamically.

Another critical change is the notable increase in standard deviations, which has increased by 1.75 times from 0.04 during the early stage to 0.07 at the last stage. This shift indicates an even wider spread of actions after optimisation. Moreover, it provides a larger strategy pool of optimal actions to realise a more flexible, responsive and pedestrian-friendly mode road layout.

Compared with the conventional complete streets scheme, both the early and later phases liberates road space assigned for the driveways to sidewalks. At the early stage, approximately a standard lane-width (3.5 m) equivalent space is re-assigned to the sidewalk in one out of 48 time slots, of which the probability estimates 2.08%. By optimisation, this ratio significantly rises to 9 out of 48 (18.22%) in the later stage. This improvement demonstrates that our SUMO-DDPG model has effectively learnt to allocate fewer proportions to AVs trafficking while increasing the share of sidewalks under different traffic conditions of a day.

6. Conclusions

In this current study, we proposed a SUMO traffic simulator-incorporated Deep Deterministic Policy Gradient Algorithm (SUMO-DDPG) to realise the optimal control of the ROW plan. The modelling objective maximises the edge-based traffic efficiency of both AVs fleet and pedestrians while maximising the ratio of sidewalks.

This proposed model has been trained in 150 episodes using a four-legged intersection. The synthesised travel plans include OD pair-based travel demands of AVs and pedestrians. Training results demonstrate that our model is efficient in convergence to optima under both the symmetric road geometries and the asymmetric settings.

Throughout the training course, episodic rewards increase by 10.39%. Meanwhile, 90.16% of the edges reduce the driveways supply and raise their sidewalk ratios. During 18.22% of all simulated time slots, additional lane-width space is shifted from driveways to sidewalks while maintaining high standard traffic efficiency. The ROW layout solutions expand 1.75 times given arbitrary traffic patterns, contributing to more flexible, responsive, and active mode friendly road layout strategies.

A key strength of this research lies within the fact that our model coordinates the multi-objectives from both fields of traffic engineering and urban planning. This SUMO-DDPG modelling framework successfully resolved the synchronical ROW assignment optimisation problem facing multiple roads conditions in real-time. Furthermore, the centralised training and distributive actions execution accelerate the learning process while decreasing computational cost, which seems quite promising to be applied to address such ROW optimal control problem on a city-level network scale.

However, some limitations of this initial study should also be acknowledged. On the one hand, the presented testing samples could only represent a limited range of traffic conditions, whereas more testing scenarios are expected. On the other hand, the efficacy of using a centralised training machine compared to a distributed one is yet to be answered. We also acknowledge that a wide range of urban design concerns could be measured and quantified as our training objectives, targeting a sustainable street space. Despite these limitations, the methodology is original and proved effective to solve the public ROW assignment problem. Meanwhile, findings present essential insights to both road infrastructure management and urban design.

Following this preliminary study, our further research plan will focus on (1) Real urban setting-based experiments using this model. (2) Comparison among a broad spectrum of reinforcement learning algorithms that can be implemented into the control model, including distributively architect DDPG, Q-learning and Asynchronous Advantage Actor-Critic (A3C). (3) Incorporate this ROW optimal control method with other intelligent transport control techniques such as traffic signal control and roadside unit management.