Enhanced Safety in Autonomous Driving: Integrating a Latent State Diffusion Model for End-to-End Navigation
Abstract
:1. Introduction
- We integrate latent diffusion models for state representation learning, enabling the forecasting of future observations, rewards, and actions. This capability allows for the simulation of future trajectories within the model framework, facilitating the proactive assessment of rewards and risks through controlled model roll-outs.
- We further extend our approach to include advanced prediction of future state value distributions, incorporating the estimation of worst-case scenarios. This ensures that our model predicts and prepares for potential adverse conditions, enhancing system reliability.
- Our experimental results demonstrate the efficacy of our proposed approaches in simulated and real-world environments, which can also guarantee safe policy exploration in unpredictable scenarios.
2. Related Work
2.1. Safe Reinforcement Learning
2.2. Reinforcement Learning with Latent State
- State Transition Equation:
- Observation Equation:
- Reward Function:
2.3. Diffusion-Model-Based Reinforcement Learning
3. Problem Modeling
4. Methodology
4.1. Constrained Markov Decision Process Formulation
4.2. Build the Latent Diffusion Model for State Representation
- Representation Model: The representation model establishes a robust latent space based on past experiences. The representation model is formalized as , predicting the next state by integrating information from the current state, action, and observation. The representation loss is quantified by assessing the accuracy of state and reward predictions.
- Transition Model: This model outputs a Gaussian distribution, defined as . The transition model’s accuracy is evaluated using the Kullback–Leibler (KL) divergence between the predicted and actual distributions, signifying the latent imagination and the environment’s real response, respectively.
- Reward Model: The reward model enhances learning by computing expected rewards based on the current state, . This model is crucial for the agent to enhance actions and maximize environmental returns.
4.3. Build Safety Guarantee
4.4. VaR-Based Soft Actor-Critic for Safe Exploration
Algorithm 1 Pseudocode for ESAD-LEND |
|
5. Experiments
5.1. Environmental Setup
Experimental Setup in CARLA Simulator
- Traffic Negotiation: Multiple vehicles interact at a complex intersection, testing the vehicle’s ability to negotiate right-of-way and avoid collisions.
- Highway: Simulates high-speed driving conditions with lane changes and merges, assessing the vehicle’s decision-making speed and accuracy.
- Obstacle Avoidance: Challenges the vehicle to detect and navigate around sudden obstacles such as roadblocks.
- Braking and Lane Changing: Tests the vehicle’s response to emergency braking scenarios and rapid lane changes to evade potential hazards.
- Town 6: Features a typical urban grid that simplifies navigation while testing adherence to basic traffic rules.
- Town 7: Incorporates winding roads and a central water feature, introducing complexity to navigation tasks and necessitating advanced path planning.
- Town 10: Represents a dense urban environment with numerous intersections and limited maneuvering space, ideal for testing advanced navigation strategies.
5.2. Design of the Reward Function
5.2.1. Velocity Compliance Reward ()
5.2.2. Lane Maintenance Reward ()
5.2.3. Orientation Alignment Reward ()
5.2.4. Exploration Incentive Reward ()
5.2.5. Composite Reward Calculation
5.3. Evaluation Metrics
- Route Completion (RC): This metric quantifies the percentage of each route completed by the agent without intervention. It is defined as follows:
- Infraction Score (IS): Capturing the cumulative effect of driving infractions, this score uses a geometric series, with each infraction type assigned a specific penalty coefficient:Coefficients are set as , , , and for infractions involving pedestrians, vehicles, static objects, and red lights, respectively.
- Driving Score (DS): This primary evaluation metric combines route completion with infraction penalties:
- Collision Occurrences (COs): This metric quantifies the frequency of collisions during autonomous driving, providing a key measure of the safety and reliability of the driving algorithm. A lower CO value indicates better collision avoidance, which is critical for the safe operation of autonomous vehicles. This metric is defined as follows:
- Infractions per Kilometer (IPK): This metric normalizes the number of infractions by the distance driven, providing a measure of infractions per unit distance:
- Time to Collision (TTC): This metric estimates the time remaining before a collision occurs, assuming the current velocity and trajectory of the vehicle and any object or vehicle in its path remain unchanged. It critically measures the vehicle’s ability to detect and react to potential hazards in its immediate environment:
- Collision Rate (CR): This metric quantifies the frequency of collisions during autonomous operation:Expressed as collisions per kilometer, this metric evaluates the efficacy of collision avoidance systems integrated into autonomous driving algorithms.
5.4. Baseline Setup
- Dreamer [39]: A reinforcement learning agent designed to tackle long-horizon tasks using latent imagination in learned world models. It distinguishes itself by employing deep learning to process high-dimensional sensory inputs and learn intricate behaviors.
- LatentSW-PPO [43]: Wang et al introduced a novel RL framework for autonomous driving that enhances safety and efficiency. This framework integrates a latent dynamic model that captures environmental dynamics from bird’s-eye view images, thereby improving learning efficiency and mitigating safety risks through synthetic data generation. Additionally, it incorporates state-wise safety constraints using a barrier function to ensure safety at every state during the learning process.
- Diffuser [35]: Janner et al proposed a novel approach to model-based reinforcement learning that integrates trajectory optimization into the modeling process, addressing the empirical shortcomings of traditional methods. They utilize a diffusion probabilistic model to plan by iteratively denoising trajectories, making sampling and planning nearly identical. In contrast to their proposed model, we further enhanced it with safety considerations.
- Safe Autonomous Driving with Latent End-to-end Navigation (SAD-LEN ): This version excludes the latent diffusion component, relying solely on traditional latent state representation, the primary focus of which is on evaluating the impact of the latent state representation on navigation performance without the enhancements provided by diffusion processes.
- Autonomous Driving with End-to-end Navigation and Diffusion (AD-END): This version removes the safety guarantee mechanisms, focusing on the integration of end-to-end navigation with diffusion models. It aims to assess the contribution of safety constraints to overall performance and safety.
6. Results and Analysis
6.1. Evaluating Prediction Performance
6.2. Evaluating Safety and Efficiency During Exploratory
6.3. Evaluate Generalization Ability
6.4. Bridging the Gap between Simulation and Real-World
6.5. Evaluate Robustness
7. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Yurtsever, E.; Lambert, J.; Carballo, A.; Takeda, K. A survey of autonomous driving: Common practices and emerging technologies. IEEE Access 2020, 8, 58443–58469. [Google Scholar] [CrossRef]
- Shi, T.; Chen, D.; Chen, K.; Li, Z. Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement. arXiv 2021, arXiv:2110.07067. [Google Scholar]
- Kuutti, S.J. End-to-End Deep Learning Control for Autonomous Vehicles. Ph.D. Thesis, University of Surrey, Guildford, UK, 2022. [Google Scholar]
- Xiao, Y.; Codevilla, F.; Gurram, A.; Urfalioglu, O.; López, A.M. Multimodal end-to-end autonomous driving. IEEE Trans. Intell. Transp. Syst. 2020, 23, 537–547. [Google Scholar] [CrossRef]
- Kiran, B.R.; Sobh, I.; Talpaert, V.; Mannion, P.; Al Sallab, A.A.; Yogamani, S.; Pérez, P. Deep reinforcement learning for autonomous driving: A survey. IEEE Trans. Intell. Transp. Syst. 2021, 23, 4909–4926. [Google Scholar] [CrossRef]
- Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A.A.; Veness, J.; Bellemare, M.G.; Graves, A.; Riedmiller, M.; Fidjeland, A.K.; Ostrovski, G.; et al. Human-level control through deep reinforcement learning. Nature 2015, 518, 529–533. [Google Scholar] [CrossRef] [PubMed]
- Kim, B.; Lee, K.H.; Xue, L.; Niu, X. A review of dynamic network models with latent variables. Stat. Surv. 2018, 12, 105. [Google Scholar] [CrossRef] [PubMed]
- Achiam, J.; Held, D.; Tamar, A.; Abbeel, P. Constrained policy optimization. In Proceedings of the International Conference on Machine Learning (PMLR), Sydney, NSW, Australia, 6–11 August 2017; pp. 22–31. [Google Scholar]
- Alan, A.; Taylor, A.J.; He, C.R.; Ames, A.D.; Orosz, G. Control barrier functions and input-to-state safety with application to automated vehicles. IEEE Trans. Control. Syst. Technol. 2023, 31, 2744–2759. [Google Scholar] [CrossRef]
- Verstraete, D.; Droguett, E.; Modarres, M. A deep adversarial approach based on multi-sensor fusion for semi-supervised remaining useful life prognostics. Sensors 2019, 20, 176. [Google Scholar] [CrossRef]
- Hao, Z.; Zhu, H.; Chen, W.; Cai, R. Latent Causal Dynamics Model for Model-Based Reinforcement Learning. In Proceedings of the International Conference on Neural Information Processing, Changsha, China, 20–23 November 2023; Springer: Berlin/Heidelberg, Germany, 2023; pp. 219–230. [Google Scholar]
- Wei, J.; Dolan, J.M.; Snider, J.M.; Litkouhi, B. A point-based MDP for robust single-lane autonomous driving behavior under uncertainties. In Proceedings of the 2011 IEEE International Conference on Robotics and Automation, Shanghai, China, 9–13 May 2011; pp. 2586–2592. [Google Scholar]
- Shalev-Shwartz, S.; Shammah, S.; Shashua, A. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv 2016, arXiv:1610.03295. [Google Scholar]
- Garcıa, J.; Fernández, F. A comprehensive survey on safe reinforcement learning. J. Mach. Learn. Res. 2015, 16, 1437–1480. [Google Scholar]
- Schulman, J.; Levine, S.; Abbeel, P.; Jordan, M.; Moritz, P. Trust region policy optimization. In Proceedings of the International Conference on Machine Learning (PMLR), Lille, France, 7–9 July 2015; pp. 1889–1897. [Google Scholar]
- Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; Klimov, O. Proximal policy optimization algorithms. arXiv 2017, arXiv:1707.06347. [Google Scholar]
- Deisenroth, M.P.; Fox, D.; Rasmussen, C.E. Gaussian processes for data-efficient learning in robotics and control. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 37, 408–423. [Google Scholar] [CrossRef]
- Koller, T.; Berkenkamp, F.; Turchetta, M.; Krause, A. Learning-based model predictive control for safe exploration. In Proceedings of the 2018 IEEE Conference on Decision and Control (CDC), Miami Beach, FL, USA, 17–19 December 2018; pp. 6059–6066. [Google Scholar]
- Zanon, M.; Gros, S. Safe reinforcement learning using robust MPC. IEEE Trans. Autom. Control 2020, 66, 3638–3652. [Google Scholar] [CrossRef]
- Ye, F.; Cheng, X.; Wang, P.; Chan, C.Y.; Zhang, J. Automated lane change strategy using proximal policy optimization-based deep reinforcement learning. In Proceedings of the 2020 IEEE Intelligent Vehicles Symposium (IV), Las Vegas, NV, USA, 19 October–13 November 2020; pp. 1746–1752. [Google Scholar]
- Hewing, L.; Wabersich, K.P.; Menner, M.; Zeilinger, M.N. Learning-based model predictive control: Toward safe learning in control. Annu. Rev. Control Robot. Auton. Syst. 2020, 3, 269–296. [Google Scholar] [CrossRef]
- Brüdigam, T.; Olbrich, M.; Wollherr, D.; Leibold, M. Stochastic model predictive control with a safety guarantee for automated driving. IEEE Trans. Intell. Veh. 2021, 8, 22–36. [Google Scholar] [CrossRef]
- Pek, C.; Zahn, P.; Althoff, M. Verifying the safety of lane change maneuvers of self-driving vehicles based on formalized traffic rules. In Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA, 11–14 June 2017; pp. 1477–1483. [Google Scholar]
- Wang, J.; Zhang, Q.; Zhao, D.; Chen, Y. Lane change decision-making through deep reinforcement learning with rule-based constraints. In Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary, 14–19 July 2019; pp. 1–6. [Google Scholar]
- Sewell, D.K.; Chen, Y. Latent space models for dynamic networks. J. Am. Stat. Assoc. 2015, 110, 1646–1657. [Google Scholar] [CrossRef]
- Sarkar, P.; Moore, A.W. Dynamic social network analysis using latent space models. ACM SIGKDD Explor. Newsl. 2005, 7, 31–40. [Google Scholar] [CrossRef]
- Padakandla, S. A survey of reinforcement learning algorithms for dynamically varying environments. ACM Comput. Surv. (CSUR) 2021, 54, 1–25. [Google Scholar] [CrossRef]
- Levine, S. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv 2018, arXiv:1805.00909. [Google Scholar]
- Lee, K.; Seo, Y.; Lee, S.; Lee, H.; Shin, J. Context-aware dynamics model for generalization in model-based reinforcement learning. In Proceedings of the International Conference on Machine Learning (PMLR), Virtual, 13–18 July 2020; pp. 5757–5766. [Google Scholar]
- Chen, J.; Li, S.E.; Tomizuka, M. Interpretable end-to-end urban autonomous driving with latent deep reinforcement learning. IEEE Trans. Intell. Transp. Syst. 2021, 23, 5068–5078. [Google Scholar] [CrossRef]
- Li, Y.; Song, J.; Ermon, S. Inferring the latent structure of human decision-making from raw visual inputs. arXiv 2017, arXiv:1703.08840. [Google Scholar]
- Wang, H.; Gao, H.; Yuan, S.; Zhao, H.; Wang, K.; Wang, X.; Li, K.; Li, D. Interpretable decision-making for autonomous vehicles at highway on-ramps with latent space reinforcement learning. IEEE Trans. Veh. Technol. 2021, 70, 8707–8719. [Google Scholar] [CrossRef]
- Gulcehre, C.; Colmenarejo, S.G.; Sygnowski, J.; Paine, T.; Zolna, K.; Chen, Y.; Hoffman, M.; Pascanu, R.; de Freitas, N. Addressing Extrapolation Error in Deep Offline Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), Vienna, Austria, 4 May 2021. [Google Scholar]
- Venkatraman, S.; Khaitan, S.; Akella, R.T.; Dolan, J.; Schneider, J.; Berseth, G. Reasoning with latent diffusion in offline reinforcement learning. arXiv 2023, arXiv:2309.06599. [Google Scholar]
- Janner, M.; Du, Y.; Tenenbaum, J.B.; Levine, S. Planning with Diffusion for Flexible Behavior Synthesis. In Proceedings of the International Conference on Machine Learning, Baltimore, MD, USA, 17–23 July 2022. [Google Scholar]
- Zhang, Z.; Li, A.; Lim, A.; Chen, M. Predicting Long-Term Human Behaviors in Discrete Representations via Physics-Guided Diffusion. arXiv 2024, arXiv:2405.19528. [Google Scholar]
- Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; pp. 10684–10695. [Google Scholar]
- Kumar, A.; Zhou, A.; Tucker, G.; Levine, S. Conservative q-learning for offline reinforcement learning. Adv. Neural Inf. Process. Syst. 2020, 33, 1179–1191. [Google Scholar]
- Hafner, D.; Lillicrap, T.; Ba, J.; Norouzi, M. Dream to control: Learning behaviors by latent imagination. arXiv 2019, arXiv:1912.01603. [Google Scholar]
- Yang, Q.; Simão, T.D.; Tindemans, S.H.; Spaan, M.T. WCSAC: Worst-case soft actor critic for safety-constrained reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Virtual Event, 2–9 February 2021; Volume 35, pp. 10639–10646. [Google Scholar]
- Mavrin, B.; Yao, H.; Kong, L.; Wu, K.; Yu, Y. Distributional reinforcement learning for efficient exploration. In Proceedings of the International Conference on Machine Learning (PMLR), Long Beach, CA, USA, 9–15 June 2019; pp. 4424–4434. [Google Scholar]
- Haarnoja, T.; Zhou, A.; Abbeel, P.; Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning (PMLR), Stockholm, Sweden, 10–15 July 2018; pp. 1861–1870. [Google Scholar]
- Wang, C.; Wang, Y. Safe Autonomous Driving with Latent Dynamics and State-Wise Constraints. Sensors 2024, 24, 3139. [Google Scholar] [CrossRef]
- Daza, I.G.; Izquierdo, R.; Martínez, L.M.; Benderius, O.; Llorca, D.F. Sim-to-real transfer and reality gap modeling in model predictive control for autonomous driving. Appl. Intell. 2023, 53, 12719–12735. [Google Scholar] [CrossRef]
- Leurent, E. An Environment for Autonomous Driving Decision-Making. 2018. Available online: https://github.com/eleurent/highway-env (accessed on 20 April 2024).
- Shi, T.; Wang, J.; Wu, Y.; Miranda-Moreno, L.; Sun, L. Efficient connected and automated driving system with multi-agent graph reinforcement learning. arXiv 2020, arXiv:2007.02794. [Google Scholar]
- Wang, J.; Shi, T.; Wu, Y.; Miranda-Moreno, L.; Sun, L. Multi-agent Graph Reinforcement Learning for Connected Automated Driving. In Proceedings of the International Conference on Machine Learning (ICML) Workshop on AI for Autonomous Driving, Virtual Event, 13–18 July 2020. [Google Scholar]
Metric | ESAD-LEND | Dreamer | LatentSW-PPO | SAD-LEN | AD-END | SAC | Diffuser |
---|---|---|---|---|---|---|---|
DS (%) | 91.2 ± 1.5 | 91.7 ± 2.1 | 86.5 ± 2.4 | 83.3 ± 2.7 | 80.4 ± 2.9 | 78.2 ± 3.1 | 87.6 ± 2.3 |
RC (%) | 98.3 ± 0.5 | 97.2 ± 0.7 | 95.8 ± 0.9 | 94.1 ± 1.1 | 92.0 ± 1.3 | 90.1 ± 1.6 | 96.4 ± 0.8 |
IS (%) | 0.5 ± 0.1 | 0.7 ± 0.2 | 0.9 ± 0.2 | 1.1 ± 0.3 | 0.4 ± 0.3 | 1.6 ± 0.4 | 0.9 ± 0.3 |
CO (%) | 0.5 ± 0.1 | 0.8 ± 0.2 | 1.0 ± 0.2 | 1.3 ± 0.3 | 1.6 ± 0.4 | 1.9 ± 0.5 | 0.7 ± 0.2 |
CR (%) | 0.4 ± 0.1 | 0.6 ± 0.2 | 0.8 ± 0.2 | 1.0 ± 0.3 | 1.3 ± 0.4 | 1.5 ± 0.5 | 0.6 ± 0.2 |
TTC (%) | 0.4 ± 0.1 | 0.6 ± 0.2 | 0.8 ± 0.2 | 1.0 ± 0.3 | 1.3 ± 0.4 | 1.5 ± 0.5 | 0.6 ± 0.2 |
Metric | ESAD-LEND | Dreamer | LatentSW-PPO | SAD-LEN | AD-END | SAC | Diffuser |
---|---|---|---|---|---|---|---|
DS (%) | 95.3 ± 1.2 | 87.4 ± 2.6 | 84.1 ± 3.0 | 80.5 ± 3.5 | 78.9 ± 3.8 | 76.8 ± 4.1 | 85.6 ± 5.5 |
RC (%) | 99.2 ± 0.3 | 96.5 ± 1.1 | 94.3 ± 1.4 | 92.1 ± 1.7 | 89.8 ± 2.0 | 87.6 ± 2.4 | 96.4 ± 0.9 |
IS (%) | 0.4 ± 0.05 | 0.7 ± 0.1 | 1.0 ± 0.15 | 1.2 ± 0.18 | 1.5 ± 0.22 | 1.8 ± 0.25 | 0.3 ± 0.1 |
CO (%) | 0.2 ± 0.03 | 0.6 ± 0.09 | 0.9 ± 0.13 | 1.2 ± 0.16 | 1.5 ± 0.20 | 1.8 ± 0.24 | 0.5 ± 0.2 |
CR (%) | 0.2 ± 0.04 | 0.5 ± 0.08 | 0.7 ± 0.11 | 0.9 ± 0.13 | 1.2 ± 0.16 | 1.4 ± 0.19 | 0.4 ± 0.1 |
TTC (%) | 0.3 ± 0.05 | 0.6 ± 0.09 | 0.9 ± 0.13 | 1.1 ± 0.16 | 1.4 ± 0.19 | 1.6 ± 0.22 | 0.4 ± 0.2 |
Planning Algorithm | Length of Path (m) | Maximum Curvature | Training Time (min) | Failure Rate (%) |
---|---|---|---|---|
ESAD-LEND | 44.2 | 0.48 | 89 | 3 |
Dreamer | 46.7 | 0.60 | 160 | 7 |
LatentSW-PPO | 45.5 | 0.43 | 155 | 4 |
SAD-LEN | 47.9 | 0.66 | 170 | 9 |
AD-END | 46.3 | 0.58 | 165 | 11 |
SAC | 48.1 | 0.72 | 160 | 14 |
Diffuser | 45.0 | 0.50 | 140 | 5 |
Speed | 1 m/s | 2 m/s | 3 m/s | ||||||
---|---|---|---|---|---|---|---|---|---|
Algorithm | Fail Rate (%) | Avg. Time (s) | Safety Score | Fail Rate (%) | Avg. Time (s) | Safety Score | Fail Rate (%) | Avg. Time (s) | Safety Score |
ESAD-LEND | 1 | 120 | 95 | 3 | 130 | 93 | 5 | 140 | 90 |
Dreamer | 5 | 140 | 90 | 7 | 150 | 88 | 10 | 170 | 85 |
LatentSW-PPO | 3 | 130 | 93 | 5 | 140 | 90 | 8 | 160 | 87 |
SAD-LEN | 4 | 135 | 92 | 6 | 145 | 89 | 9 | 165 | 86 |
AD-END | 7 | 150 | 88 | 10 | 160 | 85 | 14 | 180 | 82 |
SAC | 6 | 145 | 89 | 9 | 155 | 87 | 13 | 175 | 84 |
Diffuser | 2 | 125 | 94 | 4 | 135 | 91 | 6 | 150 | 88 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Chu, D.-T.; Bai, L.-Y.; Huang, J.-N.; Fang, Z.-L.; Zhang, P.; Kang, W.; Ling, H.-F. Enhanced Safety in Autonomous Driving: Integrating a Latent State Diffusion Model for End-to-End Navigation. Sensors 2024, 24, 5514. https://doi.org/10.3390/s24175514
Chu D-T, Bai L-Y, Huang J-N, Fang Z-L, Zhang P, Kang W, Ling H-F. Enhanced Safety in Autonomous Driving: Integrating a Latent State Diffusion Model for End-to-End Navigation. Sensors. 2024; 24(17):5514. https://doi.org/10.3390/s24175514
Chicago/Turabian StyleChu, De-Tian, Lin-Yuan Bai, Jia-Nuo Huang, Zhen-Long Fang, Peng Zhang, Wei Kang, and Hai-Feng Ling. 2024. "Enhanced Safety in Autonomous Driving: Integrating a Latent State Diffusion Model for End-to-End Navigation" Sensors 24, no. 17: 5514. https://doi.org/10.3390/s24175514