Deep Reinforcement Learning for Trading—A Critical Survey

Abstract

Deep reinforcement learning (DRL) has achieved significant results in many machine learning (ML) benchmarks. In this short survey, we provide an overview of DRL applied to trading on financial markets with the purpose of unravelling common structures used in the trading community using DRL, as well as discovering common issues and limitations of such approaches. We include also a short corpus summarization using Google Scholar. Moreover, we discuss how one can use hierarchy for dividing the problem space, as well as using model-based RL to learn a world model of the trading environment which can be used for prediction. In addition, multiple risk measures are defined and discussed, which not only provide a way of quantifying the performance of various algorithms, but they can also act as (dense) reward-shaping mechanisms for the agent. We discuss in detail the various state representations used for financial markets, which we consider critical for the success and efficiency of such DRL agents. The market in focus for this survey is the cryptocurrency market; the results of this survey are two-fold: firstly, to find the most promising directions for further research and secondly, to show how a lack of consistency in the community can significantly impede research and the development of DRL agents for trading.

1. Introduction

Predicting and analysing financial indices has been of interest in the financial community for a long time, but recently, there has been a wide interest in the machine learning (ML) community to make use of and benefit from the more advanced techniques, such as deep networks or deep reinforcement learning. In addition, the cryptocurrencies market has gained a lot of popularity worldwide, among which Bitcoin makes the top news almost weekly now.

In this survey article, we explore different aspects of using DRL for trading, by first using Google Scholar to obtain a glimpse of what the corpus of papers which use DRL for trading looks like, and then going through the main concepts of interest at the intersection of trading and DRL, such as using different measures of risk for reward shaping and embedding some form of forecasting or prediction into the agent’s decision making. We then explain the contributions of the first DRL paper in more detail [1] and continue by going through the specifics of applying DRL to trading markets. Then, we go deeper into the technologies which look most promising, hierarchical DRL and model-based RL, and finish by stating some of the commonalities and methodological issues we found in the corpus explored. We also state some of the limitations of the current study.

We begin by referring to a few survey articles which are in some way related to this work:

• Ref. [2] looks at model-free methods and is thus limiting. Moreover, it discusses some aspects from a financial-economical point of view: the portfolio management problem, the relation between the Kelly criterion, risk parity, and mean variance. We are interested in the RL perspective, and the need for risk measures is only to feed the agents with helpful reward functions. We also discuss the hierarchical approach which we consider to be quite promising, especially for applying a DRL agent to multiple markets.
• Ref. [3] looks only at the prediction problem using deep learning.
• Ref. [4] has a similar outlook as this work, but the works discussed are quite old, very few using deep learning techniques.
• Ref. [5] looks at more applications in economics, not just trading. Additionally, the deep reinforcement learning part is quite small in the whole article.

We focus more on the cryptocurrency markets, but without loss of generality, since trading is very similar among financial markets, with the only thing differing being the volatility of assets in each market type, with cryptocurrencies being the most volatile. We do advise the curious reader to take a look at the mentioned works, since they provide different perspectives and plenty of resources to further one’s research. Following [6], we take a similar approach to our investigation. We use Google Scholar to search and download all relevant papers (we log in through the respective portal wherever needed using the university subscription), and then we search through them using recoll https://www.lesbonscomptes.com/recoll/, accessed on 8 November 2021 (a free desktop application for advanced searching in documents) to be able to construct the plots in Figure 1 and Figure 2. This is possible for three reasons:

• All papers using DRL for trading will most probably cite the first paper that introduced DQN [1].
• Because of the above reason, we can use the feature in Scholar which searches through the citing articles.
• Google Scholar returns results also present on all other portals such as IEEE, Scopus, WebOfScience, etc.

Figure 1. Using Google Scholar to explore 152 papers using DRL for trading. (a) Type of action space. (b) Type of market. (c) Frequency of data used. (d) Model-free vs model-based DRL.

Figure 1. Using Google Scholar to explore 152 papers using DRL for trading. (a) Type of action space. (b) Type of market. (c) Frequency of data used. (d) Model-free vs model-based DRL.

Figure 2. Various keywords presence.

Figure 2. Various keywords presence.

The following data are obtained through this methodology (see Figure 3). The total number of articles citing [1] is 16,652 today (18 October 2021). By adding trading market reward action state as search words, we narrow this down to 401. We have to mention here that we had to put every word between quotation marks; otherwise, Google Scholar seemed a bit stochastic sometimes, in the sense that by changing the position of market in the search query, we obtained a different number of results. By inspecting them, we realise that we can remove some more by eliminating papers which deal with the electricity markets (this is because these markets are quite different than the financial markets discussed), and we can also remove papers which contain the word ’robot’, since these are probably referring to robotics papers. Before this process, we did extract 16 papers which are relevant for our work. After this, we added -electricity -robot, giving us 182 articles, which we then sift through manually and download 98 of them. We independently conducted a separate search, without looking through the citations of the DQN papers. In this way, we obtained 38 articles. In total, we obtained 152 articles. By then searching through them, we investigated the type of action space, state space, market type, and input data and if the papers use some form of prediction or not by using representative keywords (e.g., continuous actions vs. discrete actions). Of course, the following results should be taken with a grain of salt, in the sense that some keywords might appear in the paper, but as references or mentions, rather than actual uses. We think these results provide a rough idea of the underlying structures but definitely not the precise ones. More details are found in Figure 1 and Figure 2.

The cryptocurrency market has been of wide interest for economic agents for some time now. Everything started with the whitepaper on Bitcoin [7]. This is in part due to the fundamentally different principles on which it is functioning (i.e., coins are minted/mined, they are decentralised, etc.) [8], but it is also due to the extremely high volatility the market is experiencing [9]. This means there is room for significant profit for the smart agent but also the possibility of huge losses for the speculative trader. Moreover, the market capitalisation is increasing almost continuously, which means new actors are entering and investing actively in the crypto markets. When looking at the problem of automatically trading on these markets, we can define a few, quite different problems one is facing and which have been tackled separately:

• The prediction problem, where we want to accurately predict some time-steps in the future based on the past and on the general dynamics of the time-series. What we mean by this is that, for example, gold or oil do not have such large fluctuations as the crypto markets. When one is predicting the next point for the price, it is advisable to take this information into account. One way to do this is to consider general a priori models, based on large historical samples, such that we embed this possible, historical dynamics into our prediction model. This would function as a type of general prior information; for example, it is absurd to consider models where gold will rise 100% overnight, as this has never happened and probably never will. However, for the crypto markets, this might not be that impossible, and some of them did in fact rise thousands of percent overnight, with many still rising significantly every day https://coinmarketcap.com/, accessed on 7 October 2021. Deep learning models have been successfully applied to such predictions [3,10,11,12] as well as many other ML techniques [13,14,15]. In the context of RL, prediction is usually done through model-based RL [16], where a simulation of the environment is created based on data seen so far. Then, prediction amounts to running this simulation to see what comes next (at least according to the prediction model). More on model-based RL can be found in Section 10.1. For a comprehensive survey, see [17].
• The strategy itself might be based on the prediction but embed other information as well, such as the certainty of the prediction (which should inform us how much we would like to invest, assuming a rising or bull future market) or other external information sources, such as social media sentiments [18] (e.g., are people positive or negative about an asset?) or news [19], volume, or other sources of related information which might influence the decision. The strategy should be also customisable, in the sense that we might want a risk-averse strategy or risk-seeking one. This cannot be done from a point prediction only, but there are some ways of estimating risk for DRL [20]. For all these reasons, we consider the strategy or decision making a quite different problem than the prediction problem. This is the general trend in the related literature [21,22].
• The portfolio management problem is encountered when dealing with multiple assets. Until now we have been talking only about a single asset, but if some amount of resources is being split among multiple concurrent assets, then we have to decide how to split these resources. At first, the crypto markets were highly correlated (if Bitcoin was going up, then most other cryptocurrencies would go up, so this differentiation did not matter too much), but more recently, it has been shown that the correlation among different pairs is significantly decreasing https://charts.coinmetrics.io/correlations/, accessed on 11 October 2021. Thus, an investing strategy is needed to select among the most promising (for the next time-slice) coins. The time-slice for which one does rebalancing is also a significant parameter of the investing strategy. Rebalancing usually means how to split resources (evenly) among the most promising coins. Most of the work in trading using smart techniques such as DRL is situated here. See, for example, [2,23,24,25,26,27,28]. Not all the related works we are citing deal with cryptocurrency markets, but since all financial markets are quite similar, we consider these relevant for this article.
• The risk associated with some investments is inevitable. However, through smart techniques, a market actor might minimize the risk of investment while still achieving significant profits. Additionally, some actors might want to follow a risk-averse strategy while other might be risk-seeking. The risk of a strategy is generally measured by using the Sharpe ratio or Sortino ratio, which take into account a risk-free investment (e.g., U.S. treasury bond or cash), but even though we acknowledge the significance of these measures, they do have some drawbacks. We will discuss in more detail the two measures, giving a formal description and proposing alternative measures of risk. Moreover, if one uses the prediction in the strategy, by looking at the uncertainty of prediction, one can also infuse a strategy with a measure of risk given by this certainty, i.e., if the prediction of some asset going up is very uncertain, then the risk is high, and, on the other hand, if the prediction is certain, the risk is lower.

We show an overview of the above-mentioned financial concepts, the relations between them, and their RL correspondents in Figure 4.

2. Prediction

Prediction or forecasting of a time-series is the process of estimating what will happen in the next time period based on what we have seen so far. Will the price go up or down, by how much, and for how long? A comprehensive prediction provides a probability distribution (if it is not a point estimate) of these quantities. Many older works deal with the prediction problem in a financial market, assuming the strategy is clear once the prediction is accurate. It is, indeed, so, but quite often the predictions are not correct, and this can significantly impact the performance of the trading strategy. However, it has been shown that some techniques exist which can predict even quasi-periodic time-series, which is quite remarkable, considering the fundamentally chaotic nature of such time-series [29,30]. Depending on the nature of the time-series, the prediction interval is generally one time-step, and for each new time-point prediction, the ground truth is fed back to the network. If the prediction is fed such that multiple time-steps are predicted, the prediction degrades rapidly, due to accumulating errors. However, there are some exceptions (see, for example, [31]). One way to overcome this limitation, is to look at data at a higher granularity (at coarser data). For example, if one looks at minute data (by this, we mean that the prediction machinery is fed with this type of data) to predict 30 min in the future, one needs to predict 30 time-points, whereas if one looks at 30 min data (meaning each point is a summarization of a 30 min time-slice), then there is only need for a one-step prediction. Any strategy built on top of a prediction module is critically dependent on this sampling period. By using multiple different time-scales, one can have a much better picture of the future, especially if the predictions at different levels are combined (for example, using ensembles). Surprisingly, there are not many related works which use such a technique, but there are some [28].

3. Strategy for RL

The strategy for trading can vary widely, depending on its scope, the data sampling rate, the risk quantification choice, and more. We distinguish a few ways to organise the type of strategy:

• Single or multi-asset strategy: This needs to take into account data for each of the assets, either time-based or price-based data. This can be done individually (e.g., have one strategy for each asset) or aggregated.
• The sampling rate and nature of the data: We can have strategies which make decisions at second intervals or even less (high frequency trading or HFT [32]) or make a few decisions throughout a day (intraday) [33], day trading (at the end of each day all open orders are closed) [34], or higher intervals. The type of data fed is usually time-based, but there are also price-based environments for RL [35].
• The reward fed to the RL agent is completely governing its behaviour, so a wise choice of the reward shaping function is critical for good performance. There are quite a number of rewards one can choose from or combine, from risk-based measures to profitability or cumulative return, number of trades per interval, etc. The RL framework accepts any sort of reward: the denser, the better. For example, in [35], the authors test seven different reward functions based on various measures of risk or PnL (profit and loss).

There are also works which discover or learn algorithmic trading rules through various methods such as evolutionary computation [36] or deep reinforcement learning [37].

Benchmark Strategies

When evaluating this type of work using backtests, one needs to have some benchmark strategies to compare with. We have a wide variety to choose from, but we restrict ourselves to just a few for brevity. We list them below in increasing order of complexity:

• Buy and Hold is the classical benchmark [38,39,40] when trading on cryptocurrency markets, since the markets are significantly increasing; thus, simply buying an asset and holding it until the end of the testing period provides some informative baseline profit measure.
• Uniform constant rebalanced portfolio (UCRP) [41] rebalances the assets every day, keeping the same distribution among them.
• On-line moving average reversion (OLMAR) [42] is an extension to the moving average reversion (MAR) strategy [43] but overcomes some of its limitations. MAR tries to make profits by assuming the asset will return (revert) to the moving average value in the near future.
• A deep RL architecture [27] has been very well received by the community and with great results. It has also been used in other related works for comparison since the code is readily available https://github.com/ZhengyaoJiang/PGPortfolio, accessed on 10 October 2021.

4. Portfolio Management

In a financial market, one of the main issues is which assets to invest and which coins to buy. There are many available assets; on Binance (the largest platform for cryptocurrencies by market cap), for example, there are hundreds of coins, and each one can form a pair with a few major stable coins such as USDT (Tether). Thus, the choices are numerous, and the decision is hard. The portfolio management problem deals with allocating cash resources (this can be any stable currency which does not suffer from the fluctuations of the market, a baseline) into a number of assets m. Formally, this means finding a set of weights

$$\mathbf{w}=\{w_0,w_1,w_2,...,w_m\}\mathrm{such}\mathrm{that}\sum _{i=0}^m{w}_i=1$$

(1)

where at the beginning, the weights are $\mathbf{w}=\{1,0,0,\dots ,0\}$, which means we only have cash. The goal is to increase the value of the portfolio by selecting the assets which give the highest return. If at the time-step t, the overall value of the portfolio is given by

$$V_t=\sum _{i=1}^m{p}_i^t{w}_i^t$$

(2)

where $p_i$ is the sell price of the asset i, then $V_{t+1}>V_t$. Prices and asset quantities change during trading periods, and when switching back to cash (this is what the above equation does), this new amount should be larger than the previous cash value of the portfolio.

5. Risk

Risk has been quantified (measured) and/or managed (fed as a reward to the DRL agent or part of the reward) through various measures throughout the years. We state below what we consider to be some of the most important ones.

5.1. Sharpe Ratio and Sortino Ratio

One of the most ubiquitous is the Sharpe ratio [44] or the related Sortino ratio. These are performance measures of investments which take into account the risk by considering an extra term, which is a risk-free asset (originally, it was suggested that U.S. treasury bonds be used here). Formally, the Sharpe ratio is defined as

$$S_a=\frac{E\left[R_a-R_b\right]}{\sqrt{var\left[R_a-R_b\right]}}$$

(3)

where $R_a$ is the asset return, and $R_b$ is the risk-free return, with E being the expectation over the excess return of the asset return compared to the risk-free return. $var$ is the variance of the respective excess, so in the denominator, we have the standard deviation of the excess. To accurately measure the risk of an investment strategy, this should be taken over comprehensive periods, where all types of transactions happen; thus, we need a representative sample for the Sharpe ratio to give an accurate estimate of the risk.

5.1.1. Sortino Ratio

The Sortino ratio is a modified version of the Sharpe ratio where negative outcomes (failures) are penalized quadratically. This can be written as

$$S=\frac{R-T}{DR}$$

(4)

where R is the average return of the asset of interest, T is the target rate of return (also called MAR—minimum acceptance return), and DR is called the downside risk, is the target semi-deviation (square root of the target semi-variance), and is formally given by

$$DR=\sqrt{{\int }_{-\infty }^T{(T-r)}^{2}f\left(r\right)dr}$$

(5)

where r is the random variable representing the return (this might be annual), and $f\left(r\right)$ is the distribution of returns.

5.1.2. Differential Sharpe Ratio

In an online learning setting, as is the case for RL (if using this measure as a reward shaping mechanism), a different measure is needed, which does not require all the returns to obtain the expectation over the returns, which are obviously not available in an online setting. To overcome this limitation, [45] has derived a new measure given by

$$D_t=\frac{B_{t-1}\Delta A_t-\frac{1}{2}{A}_{t-1}\Delta B_t}{{(B_{t-1}-A_{t-1}^2)}^{3/2}}$$

(6)

where $A_t$ and $B_t$ are exponential moving estimates of the first and second moment of $R_t$, where $R_t$ is the estimate for the return at time t ($R_T$ is the usual return used in the standard Sharpe ratio, with T being the final step of the evaluated period), so a small t usually refers to the current time-step. $A_t$ and $B_t$ are defined as

$$\begin{array}{c}\hfill A_t=A_{t-1}+\eta \Delta A_t=A_{t-1}+\eta (R_t-A_{t-1})\\ \hfill B_t=B_{t-1}+\eta \Delta B_t=B_{t-1}+\eta (R_t^2-B_{t-1})\end{array}$$

(7)

5.2. Value at Risk

Another measure for risk is the so called value at risk (VaR) [23,37,46], which specifies with probability at most $\alpha$ that a loss greater then VaR will occur and with probability at least $1-\alpha$ that a loss smaller than VaR will occur. It is the smallest number y such that the probability $\left(p\right(Y=-X)\le y)\ge 1-\alpha$. Formally, the VaR is defined as

$$Va{R}_{\alpha }\left(X\right)=-inf\{x\in \mathbf{R}:F_X\left(x\right)>\alpha \}=F_Y^{-1}(1-\alpha )$$

(8)

where X is the random variable which describes the profit and losses, and $F_X$ is the cumulative distribution function. This can also be seen as the $(1-\alpha )$-quantile of Y. However, it implies that we assume a parametric form for the distribution of X, which is quite limiting in practice.

5.3. Conditional Value at Risk

This is also called expected shortfall [23,46] or average value at risk and is related to VaR, but it is more sensitive to the shape of the tail of the loss distribution. It is a more conservative way of measuring risk, since for the wins, the measure ignores the highest profits less likely to happen, and for the losses, it focuses on the worst ones. Formally, CVaR is defined as

$$\begin{array}{c}\hfill CVa{R}_{\alpha }\left(X\right)=-\frac{1}{\alpha }(E\left[X{\mathbf{1}}_{X<x_{\alpha }}\right]+x_{\alpha }(\alpha -P[X<x_{\alpha }]))\\ \hfill \mathrm{with}{x}_{\alpha }\mathrm{the}\mathrm{lower}\alpha -\mathrm{quantile}\mathrm{given}\mathrm{by}:\\ \hfill x_{\alpha }=inf\{x\in \mathbf{R}:P(X\le x)\ge \alpha \}\end{array}$$

(9)

and $\mathbf{1}$ is the indicator function. For more details and for other measures of risk, see https://en.wikipedia.org/wiki/Coherent_risk_measure, accessed on 15 October 2021.

5.4. Turbulence

In [47], the authors use the measure of turbulence to indicate if the market is in some extreme conditions. If this value is above some threshold, then the algorithm is stopped, and all assets are sold. This quantity is time-dependent and is defined as

$$turbulenc{e}_t=({\mathbf{x}}_t-\mu ){\Sigma }^{-1}{({\mathbf{x}}_t-\mu )}^T$$

(10)

where ${\mathbf{x}}_t\in {\mathbb{R}}^D$ is the return of the asset that we are computing the turbulence for at time-step t, and $\mu \in {\mathbf{R}}^D$ is the mean return of the asset. $\Sigma \in {\mathbf{R}}^{D\times D}$ is the covariance of the returns, with D being the number of assets.

5.5. Maximum Drawdown (MDD)

The maximum drawdown is a measure of downside risk. It represents the maximum loss from a peak to a trough in some period and is given by

$$MDD=\frac{TV-PV}{PV}$$

(11)

where $TV$ is the through value (the lowest drop an asset reaches before recovering), and $PV$ is the peak value of an asset in the period of interest. Obviously, this says nothing about the frequency of losses or about the magnitude or frequency of wins, but it is considered very informative, nonetheless (usually correlated with other complementary measures), with a small MDD indicating a robust strategy. The Calmar ratio is another measure which uses MDD as its only quantification of risk. For a comprehensive overview of the Calmar ratio, MDD, and related measures, see [48].

6. Deep Reinforcement Learning

Deep reinforcement learning is a recent field of ML that combines two powerful techniques, the data-hungry deep learning and the older process-oriented reinforcement learning (RL). Reinforcement learning took birth as a heuristic approach, a descendant of dynamic programming. Sutton and Barto in their seminal work [49] established the basis for a whole new field, which would turn out to be highly influential throughout the years, both in neuroscience and ML. In general, reinforcement learning was employed where the data were few and the behaviour was complex. However, recently, because of the advent of deep networks, reinforcement learning has received increased horsepower to tackle more challenging problems. For a comprehensive overview, see [50,51,52]. The general reinforcement learning problem is defined by an agent acting or making decisions in an environment, where the process is modelled as a Markov decision process (MDP).

The MDP is represented by a state space $\mathcal{S}$ comprising the states the agent can be in, defined by the environment, the action space $\mathcal{A}$, which is the set of actions an agent can take, a transition dynamics which gives the probabilities that the agent has of being in a state at time t; taking an action and being in another state at time $t+1$, i.e., $p\left(s_{t+1}\right|s_t,a_t)$; and a reward function $r:\mathcal{S}\times \mathcal{A}\to \mathcal{R}$. An MDP dynamics is assumed to satisfy the Markov property $p\left(s_{t+1}\right|s_1,a_1,...,s_t,a_t)=p\left(s_{t+1}\right|s_t,a_t)$, that is, the next state is dependent only on the previous state and action for any state-action space trajectory. This is generally verbalised as the future being independent of the past given the present. The general goal of the agent is to find a policy which maximizes the discounted future reward, given by $R={\sum }_{k=t}^T{\gamma }^{k-t}{r}_{s_k,a_k}$, with $\gamma$, a real number called the discount factor, with $0<\gamma <1$. The policy is modelled to be stochastic in general and is parameterized by a set of parameters $\theta$, i.e., ${\pi }_{\theta }:\mathcal{S}\to \mathcal{P}\left(\mathcal{A}\right)$, where $\mathcal{P}\left(\mathcal{A}\right)$ is the set of probability measures on the action space $\mathcal{A}$. The reward is assumed to be given by the environment, but we will see later that auxiliary reward functions (not given by the environment) can significantly improve an agent’s performance. We will state briefly the main paradigms used in finding a good policy, and then we will delve directly into the acclaimed DQN, the first DRL agent introduced in the machine learning community.

We consider the partitioning of the RL algorithms following Silver http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf, accessed on 2 October 2021. The three approaches include value-based RL, where the value in each state (and action) is a kind of prediction of the cumulative future reward. In short, each state has an associated real value number which defines how good it is to be in one state (or to take a specific action being in one state). This is given by the state-value function, usually denoted by $V^{\pi }\left(s\right)$, where $\pi$ is the policy which governs the agent and which has one identifier, namely the state, or the state-action value function given by $Q^{\pi }(s,a)$, which has two identifiers, namely the state and action. We will see later that new flavours of the value function exist, where additional identifiers are used: for example, the current goal of the agent (in the case where there are multiple goals). In DRL, the value functions are outputs of deep networks.

The second approach is to represent the actual policy as a deep network, and this is referred to as policy-based RL. In general, the policy networks can be directly optimized with stochastic gradient descent (SGD).

The third approach is model-based RL, where the agents specifically construct a transition model of the environment, which is generally modelled by a deep network. This is sometimes hard, depending on the complexity of the environment, but offers some advantages over the model-free approaches; for example, the model is generative, meaning that the agent can generate samples from its model of the environment and thus can avoid actually interacting with the environment. This can be highly beneficial when this interaction is expensive or risky. Having stated the main approaches, we proceed to the description of the first DRL agent, often referred to as DQN (Deep Q Network) [1]. We show in Figure 5 the much acclaimed architecture of the original DQN. In DQN, powerful deep networks were employed for approximating the optimal action-value function, i.e.,

$$Q^{*}(s,a)=\underset{\pi }{max}\mathbb{E}\left[r_t+\gamma r_{t+1}+{\gamma }^2{r}_{t+2}+...|s_t=s,a_t=a,\pi \right]$$

(12)

where $r_t,a_t$ are the reward and action at time-step t, each future reward is discounted by a factor of $\gamma$, and $\pi$ is the policy that governs the agent’s behaviour. For the rest of this document, if we do not specify with respect to what variables of the expectation operator are considered, then it is with respect to the state variable. The iterative version is then given by

$$Q^{*}(s,a)=\mathbb{E}\left[r_{t+1}+\gamma \underset{a^{\prime }}{max}{Q}^{*}(s_{t+1},a^{\prime })|s_t=s,a_t=a\right]$$

(13)

In general, in reinforcement learning, the optimal Bellman value or action-value function is denoted with an upper ${}^{*}$. DQN was able to play a few dozens of Atari games with no special tuning of hyperparameters or engineered features. Learning only from pixel data, the agent was capable of learning to play all games with human or almost human performance. This is remarkable for an artificial agent. For the first time, an agent combined the representational power of deep nets with the RL’s complex control with two simple tricks.

The first one is replay memory, which is a type of memory that stores the last million frames (this number can vary as a function of the task, and it is a hyperparameter of the model) from the experience of the agent and avoids the highly correlated consecutive samples arising from direct experience learning.

The second one was brought by replacing the supervised signal, i.e., the target values of the Q-network, or the optimal Q function from the Bellman equation with an approximation consisting of a previously saved version of the network. This is referred to as the target network. Using this approach defines a well-posed problem, avoiding the trouble of estimating the optimal value function. Then, the loss function is given by (we use the notation used in the original DQN paper [1])

$$L_i\left({\theta }_i\right)={\mathbb{E}}_{(s,a,r,s^{\prime })\sim U\left(D\right)}\left[{\left(r+\gamma \underset{a^{\prime }}{max}Q(s^{\prime },a^{\prime };{\theta }_i^{-})-Q(s,a;{\theta }_i)\right)}^2\right]$$

(14)

where $U\left(D\right)$ is the uniform distribution giving the sample from replay memory, $\gamma$ is the usual discount factor, and ${\theta }_i^{-}$ is the set of parameters from a previous iteration (this is the second enhancement we mentioned above). This gives the following gradient for the loss:

$${\nabla }_{{\theta }_i}{L}_i\left({\theta }_i\right)={\mathbb{E}}_{(s,a,r,s^{\prime })\sim U\left(D\right)}\left[\left(r+\gamma \underset{a^{\prime }}{max}Q(s^{\prime },a^{\prime };{\theta }_i^{-})-Q(s,a;{\theta }_i)\right){\nabla }_{{\theta }_i}Q(s,a;{\theta }_i)\right]$$

(15)

These two main additions enabled unprecedented performance and adaptability of the deep net and enabled the agent to deal with a relatively diverse set of task: different games but still the same domain, Atari 2600. The work that followed was immense; in Figure 6, we show the number of citations over the years.

We will talk about some of the ideas that extend the current approach, but functionally and theoretically. We will not be discussing practical concerns, even though we are aware of the critical importance of such works and the fact that they effectively enable these algorithms to run in an acceptable time. Deep learning is a relatively small subset of the whole machine learning community, even though it is progressing fast, at an unprecedented rate, due to the relative ease of use and ease of understanding, but also due to the higher computational resources available. Being a visual task, CNNs perform best at playing Atari games, but other types of networks have been used: for example, for text-based reinforcement learning as well [53,54]. More and more tasks will be amenable to DRL as the field embraces the new paradigm and the various existing approaches are adapted to fit these new understandings of what can be achieved. Trading has been of wide interest for a long time in the ML community, and DRL has been viewed as a new set of powerful tools to be used for modelling and analysis on the financial markets.

We now proceed to describe different aspects of training a DRL agent to make profitable trades. The first critical problem is how we present the data for an agent, such that it obtains all the information it needs in a compressed form for reduced dimensionality. This is referred to as the representation learning problem in RL [55], and recently, it has been shown that this can be decoupled from policy learning [56].

7. The Financial Market as an Environment for DRL

We now intend to find the specifics of the trading problem in the context of RL. What do the different modelling choices meant for the DRL agents dealing with trading or multiple asset trading, i.e., portfolio management. The main choices are:

• The data: this is how we sample them, concatenate them, preprocess them, and feed them. This amounts to defining the state space for the DRL agent: what is in a state and how it is represented.
• The actions: what can the agent do at any decision point? Can it choose from a continuous set of values, or does it have a limited choice? This amounts to what type of operations we allow for the agent. If they are continuous, this usually happens for portfolio management, where we have multiple assets to manage and we would like to assign a specific amount of the total wallet value to a specific asset. This then needs to translate into the actual trade-making as a function of the current portfolio holdings. This is generally more complicated, and not all RL algorithms can deal with continuous action spaces out of the box.
  In the case of a discrete action set for one asset, this amounts to the number of coins to buy/sell which can very from three to hundreds (where three is a simple buy/sell a fixed amount, say one, and hold). For more than three actions, this generally allows more quantities to be bought or sold, with each one representing an increasing number of coins to buy or sell.
• The reward is fully governing the agent’s behaviour; thus, a good choice of reward function is critical for the desired type of strategy and its performance. Since the reward will be maximized, it is natural to think of measures such as current balance or PnL as a possible reward function, and they have been used extensively in the community. However, these are often not sufficient to be able to customise a strategy according to different criteria; thus, measures of risk seem quite an obvious choice, and these also have been used with success in the DRL trading community.

8. Time-Series Encodings or State Representation for RL

There are many ways to encode a time-series, from a simple moving average to high-rank tensors. We will discuss in detail the most widely used encodings for the financial markets. We are interested solely in trading based on price and will not consider other alternatives, such as, for example, limit order books (LOB) https://www.investopedia.com/terms/l/limitorderbook.asp, accessed on 29 September 2021. In short, LOBs are lists of orders which actors on the market are willing to execute. They usually specify a price level and a quantity. Sometimes these are aggregated to give one single value for the price (for example, the first 10 buy orders and first 10 sell orders are weighted by their respective quantities to give a weighted average). We also skip the details of how the news/Twitter feed sentiments are encoded in the state.

8.1. Convolutions

As convolutional neural networks (CNNs) were initially successfully used in vision [57], they gained popularity throughout the wider ML field: for example, in natural language processing (NLP) [58], anomaly detection [59], drug discovery https://www.kqed.org/futureofyou/3461/startup-harnesses-supercomputers-to-seek-cures, accessed on 2 October 2021 and many others. Among these, there is also the application to time-series encoding and forecasting [60]. As we are interested in RL, one way to do it for the state representation of financial time-series is to concatenate the time-series of interest into a tensor, with one axis corresponding to time (usually there are multiple adjacent time-steps fed as a state, the last 5 or last 10 steps) and the other axis corresponding to the different quantities of interest (e.g., open, close, high, low price of a candlestick). We show the graphical representation from [27,40] in Figure 7.

8.2. Raw-Data OHLCV

Due to large variations in the price of different assets, we need to preprocess each one such that they have a standardized form (e.g., between 0 and 1). Using the return, $v_t/v_{t-1}$, to normalize prices is often used in the financial market literature. The OHLCV means the open, high, low, close, and volume of one time-point. Many works use this representation, concatenating the last few time-steps as a single state for each point. A variation on this is to obtain features from the raw data, which is often quite noisy, by preprocessing it with a Kalman filter and then extracting features with an autoencoder and/or SVD for time-series [61].

8.3. Technical Trading Indicators

These are analytical ways of transforming the time-series, developed by economists to summarize or highlight different properties of the time-series. A wide variety of technical indicators exist: see, for example, [62] or some of their uses combined with DRL in [63,64,65].

8.4. Correlated Pairs

In one work [66] dealing with the FX market, the authors use three related pairs to predict one. The same can be done for the crypto markets, where we use inputs of correlated pairs of cryptocurrencies (e.g., for predicting BTCUSD, we might use ETHUSD, BTCEUR, ETHBTC, etc.). This assumes that there is a lot of information in the related pairs such that we obtain a more informative dynamics than in the predicted time-series alone.

8.5. Historical Binning or Price-Events Environment

One interesting way to transform the continuous trading environment (i.e., price) is to discretize the price and transform the time-series such that it can take only a fixed number of values. If the price change is between the threshold values, then no effective change is seen by the DRL agent. This, in fact, compresses the time-series and transforms it into being price-event based instead of time-event based [35]. It is a way of simplifying the time-series, making it more relevant for trading and removing some of the noise existing in the original data. Due to the shorter time-series produced in this way, agents using this transformation should train faster. We can also combine the two approaches, where we keep the time component, but we still discretise the price value. We show both approaches in Figure 8. We embark on developing more types of environments (among them being the one in Figure 8b, where we modify the incoming data on the fly by, for example, artificially augmenting rises or drops). This should make agents more sensitive to one or the other and thus change their behaviour into being risk-seeking or risk-averse.

8.6. Adding Various Account Information

Some of the authors add to the state of the RL agent the price or quantity of the last buy, the current balance of the portfolio, or the specific stocks held and their respective quantities. There does not seem to be a general conclusion for what sort of information is useful in the state of the agent, as agents can be widely different, and the optimal choice is specific to each. For more information, see

Table 1. Table summarising the main modelling choices from a small subset of the papers which use DRL for trading.

Article	Data	State Space	Action Space	Reward	Performance
Deep Reinforcement Learning Agent for S&P 500 stock selection [67]	28 S&P500 5283 days for 415 stocks	total return, closing price, volumes, quarterly earnings, the out-paid dividends, the declaration dates for dividends, the publication dates for the quarterly reports (d, n, f), where d is the length of the observation in days, n is the number of stocks in the environment, f is the number of features for a single stock	Continuous 20 dims	Differential Sharpe	1255 days Total return: 328% Sharpe=0.91
Diversity-driven knowledge distillation for financial trading using Deep Reinforcement Learning [68]	28 FX, 1h	normalized OHLCV, current open position as one-hot	Discrete (exit, buy, sell)	PnL	2 years PnL ≈ 0.35
A Deep Reinforcement Learning Approach for Automated Cryptocurrency Trading [69]	Bitcoin, 3200 h	time stamp, OHLCV, USD volume weighted Bitcoin price	Discrete (buy, hold, sell)	Profit Sharpe	800h return ≈ 3%
An application of deep reinforcement learning to algorithmic trading [39]	30 stocks daily, 5 years	current trading position, OHLCV	Discrete (buy, sell)	Normalized price change	Sharpe = 0.4 AAPL annual return=0.32
Adaptive Stock Trading Strategies with Deep Reinforcement Learning Methods [63]	15 stocks daily, 8 years	OHLCV + Technical indicators (MA, EMA, MACD, BIAS, VR, and OBV)	Discrete (long, neutral, short)	Sortino ratio	3 years −6%−200%
Portfolio management system in equity market neutral using RL [70]	50 stocks daily, 2 years	normalized OHLC	Continuous 50 dims	Sharpe ratio	2 years profit ≈ 50%
Deep Reinforcement Learning for Automated Stock Trading: [47]	30 stocks daily, 7 years	balance, shares, price technical indicators (MACD, RSI, CCI, ADX)	Discrete ${(2k+1)}^{30}$	Portfolio value change Turbulence	Sharpe=1.3 Annual return=13%
Portfolio trading system of digital currencies: A deep reinforcement learning with multidimensional attention gating mechanism [71]	20 assets 30 min, 4 years	normalized HLC, shares	Continuous 20 dims	Log cumulative return	2 months Return = 22x
A framework of hierarchical deep q-network for portfolio management [26]	4 Chinese stocks 2 days, 2 years	OHLC, shares	Discretised × 4	Price change	1 year 44%
Attentive hierarchical reinforcement learning for stock order executions [22]	6 stocks 2 min, 2 months	OHLCV	Discrete (buy, sell)	Balance	1 month unclear%
Trader: Practical deep hierarchical reinforcement learning for trade execution [72]	35 stocks 1 min, 1 year	OHLCV	Discrete (buy, hold, sell) and Continuous	Auxiliary: Surprise minimisation	unclear
Commission fee is not enough: A hierarchical reinforced framework for portfolio management [25]	46 stocks daily, 14 years	LOB, OHLCV	Discrete (buy, sell) and Continuous	Price change, Cumulative trading costs	Wealth ≈ 200%
Model-based reinforcement learning for predictions and control for limit order books [73]	one stock 17s, 61 days	LOB, trade prints	Discrete 21 actions	Predicted price change	19 days cumulative PnL = 20
Model-based deep reinforcement learning for dynamic portfolio optimization [23]	7 stocks hourly, 14 years augmented	OHLC predicted next HLC market perf. index	Continuous	Predicted price change	2 years annual return = 8%

.

9. Action Space

The action space is generally the simple discrete $\{buy,hold,sell\}$, but there are some which use more actions to denote a different number of shares or quantity to buy or sell, as in [74]. If dealing with multiple assets, either a continuous multi-dimensional action space is used, or the problem is split into selecting the number of shares/quantity (for example, with a deep learning module [75,76]) and separately selecting the type of order (buy, sell).

10. Reward Shaping

The reward is the most important part when dealing with a DRL trading agent. Most works use the direct profit from a transaction or a longer term profit (even after a full episode, but episodes can vary in length from a few hundred time-steps to thousands). The other group of reward functions is based on various measures of risk (for a comprehensive overview see [35]). Even the number of transactions can be used [77].

We described in detail in Section 5 various measures of financial risk which can be used as a dense reward shaping mechanism. Generally, the rewards are sparse for complex environments, and we know from Section 11 that giving agents auxiliary rewards which are dense can help a great deal in improving performance and the speed of learning. Some measures do have limitations, in the sense that they can only be computed after a reasonable estimate of some expectation is computed, which can suffer from significant sampling bias; thus, the measures computed online are more suitable for DRL.

We have seen how computing turbulence can give a strong insight into the current regime of the market and how using VaR can directly give us a set of probabilities for our potential profits or losses, whereas CVar can be used to bias our strategy to focus more on the losses than on the profits. Choosing an appropriate measure of risk to work as a reward for DRL agents, combining more of them, or perhaps weighing them according to the current market development would provide additional benefits to the ability to customise a trading strategy based on the affinity towards risk. Thus, one solution to adapt to the changing market conditions or regimes could benefit from changing (some parameters of) the reward function based on some measure of these changing conditions.

10.1. Model-Based RL

In general, model-based methods are more data efficient, or have lower sample complexity, but are more computationally intensive. There are many approaches in the general RL literature, both parametric and non-parametric, most notably [78,79,80], but we are interested in describing, in this work, just a few approaches used in deep RL. Both model-free and model-based methods are known to exist in the brain [81], but the cognitive sciences differentiate between the types of tasks which employ each. Model-based decision making is shown to be used when the task is outcome-based, so one is interested in the unfolding in time of the sequence of states and actions (and sometimes rewards), while model-free methods are used when the decision making is mostly concerned with the value of an action (e.g., moral judgements; this action is good because it is moral). For DRL, however, model-based approaches are quite scarce, probably due to the high complexity of models that need to be learned and the variety of models needed for the tasks that DRL is usually applied to. For example, TRPO deals with a number of classical control tasks and Atari games; it would be quite hard to derive a model-building technique easily applicable to both of these domains. Even the Atari games are so different from one another that a general model-building algorithm for all of them (the 50 games used more often in the DRL literature) would not be trivial.

Model-Based RL in Trading

As we mentioned earlier, model-based RL implies the creation of a model or simulation of the environment. This simulation is then run to produce different (if it is run multiple times and it is a stochastic model) possible predictions for the future. One such example is the work in [73], where the authors learn a model of the time-series which is a summary of a limit order book with an MDN after encoding (or extracting features from) the LOB with an AE [82]. They then use this model to learn from it with an RL agent and without requiring interaction with the actual environment. In [23], the authors test two prediction models, the NDyBM and WaveNet adapted to multi-variate data. The predicted data are the close, high, and low percentage change in the asset price value. With a couple of additional modules (data-augmenting module and behavioural cloning module), the overall architecture is applied to a small number of US stocks data successfully.

10.2. Policy-Based RL

There are many approaches which use policy-based RL algorithms combined with other techniques.

For example, in [76], the authors use a supervised learning module (autoencoder) to predict the next time-step of the price, which was then used in signal representation for RL. This learns, through a combined reward function, both the prediction and the policy. They also use temporal attention and an interesting set of features in addition to the OHCLV price and a set of technical indicators, which is called cointegration and is a measure of the stationarity or linear stability of a sum of two different stocks. There is also a preprocessing step of gate-based feature selection among all these, where a gate similar to the ones in GRU and LSTM is present for each stock and is learned end-to-end.

There are many contributions and additions from this paper, training and testing on 2000 and 3000 steps, respectively, for 20 years with an annual rate of return of about $20\%$ in a bull and $10\%$ in a bear market. A larger scale test (55 stocks) shows a $26\%$ annual return. One of the most well-received papers is [27], where the authors apply their portfolio management DRL to 65 cryptocurrencies on Poloniex, on 30 min data, concatenating the last 50 time-steps and all assets into one large tensor $(50,65,3)$ with 3 being the number of features (high, low, and the so-called return $v_t/v_{t-1}$), where the $v_t$ is the open price for time t, and $v_{t-1}$ is the close price for time $t-1$. The algorithm used is DDPG [83] with the action-space $\in {\mathbb{R}}^{m+1}$, with m being the number of coins (11 plus cash). The architecture is called EIIE (Ensembles of Identically Independent Evaluators) because to some degree, each stream is independent, and they only connect at the output, even though they are sharing the weights among them. They also add the previous portfolio value before the softmax output as to minimise transaction costs. The results are remarkable, compared to the baseline strategies tested (up to almost 50× the initial investment). It is interesting to note the large training set (2 years) compared to the test set (50 days). They test 3 different topologies, a CNN, a simple RNN, and an LSTM, with all three performing significantly better than any other strategy tested.

11. Hierarchical Reinforcement Learning

DRL works great on a wide class of problems, where the state space and action space are reasonably small (there are flavours of DRL, such as deterministic policy gradient [83], which can deal with continuous action spaces as well, so in theory, an infinite number of actions). However, there are certain problems where DRL is insufficient or takes too much time to train. A divide-and-conquer extension for DRL is hierarchical reinforcement learning (HRL) where the overall problem is split into somewhat independent subproblems. Either the action space is divided into smaller partitions or the state space or both. The most appealing aspect of HRL is that the agent higher in the hierarchy gives goal information to the level below and thus rewards based on how well the agent is doing with respect to the goal being pursued. This means that auxiliary rewards (rewards which are given to the agent in addition to the environment rewards, usually defined by the programmer; see, for example, [84]) can be learned rather than hard-coded. This is a huge milestone in DRL since the rewards are fully governing an agent’s behaviour. There are many different architectures for HRL, each inheriting some of the traits of the original flat RL, some with engineered dynamics between the agents, while others learn this as well. HRL holds the promise of scaling DRL to large sets of different problems, because agents in lower levels learn how to solve specific narrow tasks, or skills, whereas going up the hierarchy, the agents learn more general behaviour and how to select between the agents in lower levels. Transfer learning also comes more naturally in HRL due to the advantages mentioned above.

HRL in Trading

There is little research of HRL in trading, but some works exist; see, for example, [22,25,26,72]. HRL is used in different ways in the mentioned works, each of which we will discuss in detail.

In one of the works, [26], the authors use HRL to manage a subset of the portfolio with an individual, low-level agent. Of course, the principle can be applied repeatedly, where a few flat agents (we call flat agents the lowest level agents which interact with the actual trading environment) are controlled by a higher level agent (level 1); then, a group of agents at level 1 is controlled by an agent at level 2, and so on. In this way, an arbitrary level hierarchical agent can be formed.

This type of partitioning deals with the state space (since each DRL agent will receive only the prices for the stocks that it is managing) and action space (will output only actions for specific stocks), but there is no shared dynamics between the levels of the hierarchy. However, there is information sharing in the form of the weight vector for each agent coming from the level above. Even though this approach is quite simplistic, it is also quite natural and partitions the overall problem into smaller subproblems which can be solved individually without loss of expressivity.

In a second line of work, [22] considers the problem of prediction and order execution as two separate problems and first uses one network to predict the price and another network which uses this prediction (concatenates it to the original state) to execute the orders. Even though this is not, strictly speaking, HRL, since the problem is not partitioned into smaller subproblems, and one cannot form an arbitrary level hierarchy (as in the previous work) based on these principles (there is no goal giving, etc.), there is information sharing between the agents. We would classify this work in the domain of multi-agent systems, where each agent is assigned one specific task, and they communicate to reach the overall goal. Unrelated, but worth mentioning, is the fact that this work also applies a simple version of causal attention which seems to improve the profitability of the strategy on stocks.

A somewhat related work (by the same principal author) uses HRL in [72] to now partition the trading problem into first estimating the quantity to be bought, held, or sold and then actually choosing one of the three actions. This is somehow unintuitive since the operation to be performed seems more important than the actual quantity, so first choosing a quantity and the choosing what to do with it seems quite far-fetched. Again, there is no partitioning of the state space (the quantity estimated is just concatenated to the original state) and no goal setting or reward giving by the high-level agent. However, there is an interesting contribution: the authors add a surprise term to the reward (coupled with the mellowmax operator) which makes the reward surprise agnostic, the authors state. One last work we found in this narrow HRL for trading subfield of RL is [25] which tackles a practical problem in trading, i.e., once we know what we want to invest in in the long run, how can we go about minimising the transaction cost in the short run? Even though this hierarchy only has two levels, it leverages the ability of HRL to deal with multiple time-scales to make decisions. The high-level agent decides which assets to hold for some larger period (they call it the holding period), and the low-level agent decides how to buy these assets (and/or sell them to reach the desired portfolio configuration) such that transaction costs are minimised through a shorter period (the trading period). The authors use limit order books for states, which are higher dimensional and require more resources (see more details in Section 8).

12. Conclusions

As a conclusion to this work, we mention some of the commonalities between papers and some methodological issues which arise, and if solved, could benefit the community significantly.

12.1. Common Ground

• Many papers use convolutional neural networks to (at least) preprocess the data.
• A significant number of papers concatenate the last few prices as a state for the DRL agent (this can vary from ten steps to hundreds).
• Many papers use the Sharpe ratio as a performance measure.
• Most of the papers deal with the portfolio management problem.
• Very few papers consider all three factors: transaction cost, slippage (when the market is moving so fast that the price is different at the time of the transaction than at the decision time), and spread (the difference between the buy price and sell price). The average slippage on Binance is around $0.1\%$ https://blog.bybit.com/en-us/insights/a-comparison-of-slippage-in-different-exchanges/, accessed on 5 Octobere 2021, while the spread is around $1\%$, with the maximum even being $8\%$ (see Figure 9).

12.2. Methodological Issues

• Inconsistency of data used: Almost all papers use different datasets, so it is hard to compare between them. Even if using the same assets (such as Bitcoin), the periods used are different, but in general, the assets are significantly different.
• Different preprocessing techniques: Many papers preprocess the data differently and thus feed different inputs to the algorithms. Even if algorithms are the same, the results might be different due to the nature of the data used. One could argue that this is part of research, and we agree, but the question remains: how can we know if the increased performance of one algorithm is due to the nature of the algorithm or the data fed?
• Very few works use multiple markets (stock market, FX, cryptocurrency markets, gold, oil). It would be interesting to see how some algorithms perform on different types of markets, if hyperparameters for optimal performance differ, and, if so, how they differ. Basically, nobody answers the question, how does this algorithm perceive and perform on different types of markets?
• Very few works have code available online. In general, the code is proprietary and hard to reproduce due to the missing details of the implementation, such as hyperparameters or the deep neural network architecture used.
• Many works mention how hard it is to train a DRL for good trading strategies and the sensitivity to hyperparameters that all DRL agents exhibit. This might also be due to the fact that they do not approach the issues systematically and build a common framework, such as, for example, automatic hyperparameter tuning using some form of meta-learning [85] or autoML [86]. This is an issue for DRL in general, but more pronounced in this application domain, where contributions are often minor in terms of solving more fundamental issues but are more oriented towards increasing performance.
• There is no consistency among the type of environment used. There are various constraints of the environment, with some being more restrictive (not allowing short-selling for example) or simplifying, assuming that the buy operation happens with a fixed number of lots or shares. A few works allow short-selling; among our corpus, 20 mention short-selling, while about 5 actually allow it [61,75,87,88,89].

In Table 1, we showed a list of the most representative modelling choices with respect to the data used, their sampling frequency, the state space, the action space, the reward functions, and the performance measures and values reported in the articles. This is not at all a comprehensive list, but a small set which seemed to the author to catch much of the diversity in the whole corpus. The table is intentionally heterogeneous to reflect the difference in modelling as well as in reporting the different choices. Therefore, one conclusion which comes from this study is that there is need for more homogeneity in the methodology, data used, sampling frequency, preprocessing choice, and reporting methods.

12.3. Limitations

The limitations of the current study depend on the tools we used, Google Scholar and recoll, and critically on the fact that we cannot really judge the use of some keywords as representatives for a paper; we would have to understand the context in which they are used. However, they do provide some sort of popularity measure of the specific concepts defined by the keywords, which is useful for knowing in which direction the community is heading. Another limitation comes from the independent search, which could surely be incomplete. Ideally, one would also conduct a thorough investigation of each set of articles cited in every paper found, which would probably reveal many related articles. However, this seems to be a task better suited to a machine than to a human. We leave this for further work.