Skip to Content
SymmetrySymmetry
  • Article
  • Open Access

2 July 2021

16 Pages

Transition Based Discount Factor for Model Free Algorithms in Reinforcement Learning

,
,
and
1
Department of Computer Science, PDPM Indian Institute of Information Technology Jabalpur, Madhya Pradesh 482005, India
2
Department of Computer Science, JNU, Delhi 110001, India
3
Department of Computer Science, IIT (BHU) Varanasi, Uttar Pradesh 221005, India
*
Author to whom correspondence should be addressed.
This article belongs to the Section A: Computer Science

Abstract

Reinforcement Learning (RL) enables an agent to learn control policies for achieving its long-term goals. One key parameter of RL algorithms is a discount factor that scales down future cost in the state’s current value estimate. This study introduces and analyses a transition-based discount factor in two model-free reinforcement learning algorithms: Q-learning and SARSA, and shows their convergence using the theory of stochastic approximation for finite state and action spaces. This causes an asymmetric discounting, favouring some transitions over others, which allows (1) faster convergence than constant discount factor variant of these algorithms, which is demonstrated by experiments on the Taxi domain and MountainCar environments; (2) provides better control over the RL agents to learn risk-averse or risk-taking policy, as demonstrated in a Cliff Walking experiment.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.