Reinforcement Learning with Self-Attention Networks for Cryptocurrency Trading

Abstract

This work presents an application of self-attention networks for cryptocurrency trading. Cryptocurrencies are extremely volatile and unpredictable. Thus, cryptocurrency trading is challenging and involves higher risks than trading traditional financial assets such as stocks. To overcome the aforementioned problems, we propose a deep reinforcement learning (DRL) approach for cryptocurrency trading. The proposed trading system contains a self-attention network trained using an actor-critic DRL algorithm. Cryptocurrency markets contain hundreds of assets, allowing greater investment diversification, which can be accomplished if all the assets are analyzed against one another. Self-attention networks are suitable for dealing with the problem because the attention mechanism can process long sequences of data and focus on the most relevant parts of the inputs. Transaction fees are also considered in formulating the studied problem. Systems that perform trades in high frequencies cannot overlook this issue, since, after many trades, small fees can add up to significant expenses. To validate the proposed approach, a DRL environment is built using data from an important cryptocurrency market. We test our method against a state-of-the-art baseline in two different experiments. The experimental results show the proposed approach can obtain higher daily profits and has several advantages over existing methods.

1. Introduction

Asset exchange can be very profitable if it is done properly. The values of typically traded assets, such as cryptocurrencies, fluctuate constantly due to the interactions of the participants in markets. Therefore, if it can be predicted which assets have high chances of gaining value, they can be acquired and sold off in the future, generating profits to the investor. Profits can also be generated when the value of assets decreases; short selling and margin trading services work under this premise. Predicting which assets have the potential to increase their value is difficult. This is because trading is linked to human sentiment, which changes rapidly when important global events occur, for instance, political elections, international agreements, or natural disasters, and also it is easily manipulated using branding or marketing campaigns [1]. Nonetheless, asset trading is commonly practiced by both humans and algorithms throughout the world, where the most commonly traded assets are stocks, fiat money, and cryptocurrencies.

In the last decade, cryptocurrencies have gained worldwide relevance despite the initial skepticism of the people towards these assets. More and more countries are allowing the use of cryptocurrencies as a payment method, and in addition, in June of the present year (2021), the republic of El Salvador became the first country to adopt Bitcoin as legal tender. https://www.nytimes.com/2021/06/09/world/americas/salvador-bitcoin.html (accessed on 6 August 2021). However, the price of Bitcoin has been extremely volatile throughout its history, and widely different compared to traditional assets. Therefore, to overcome the shortcomings of this cryptocurrency, hundreds of alternative tokens, known as ‘altcoins’, have sprung up around the world [1], which in turn has made the number of platforms offering cryptocurrency services, such as trading, grow. These platforms have significant advantages against those for other types of assets, with the most important being access to the common public and a minimal amount of cash required to start trading.

We propose a DRL approach for cryptocurrency trading, where a self-attention (SA) network is the backbone of the system. Cryptocurrency markets contain hundreds of assets that can be exchanged. Therefore, being able to process massive amounts of data allows the system to gain a better understanding of the market state and the individual state of the assets, which in turn produces better diversification of investments and reduces risk [2].

This work also explores the problem of inter-asset transactions (IAT). Most markets require a fee for each transaction made on their platforms, and cryptocurrency markets are no exception. Therefore, a tool to compute the transactions that leads to the minimum possible fee expenditure is important. This issue is even more significant if the trading frequency is high, for example, if an investor trades every six hs a cryptocurrency volume worth 10,000 USD with a fixed transaction fee of 0.1%. Then, after only one month the total expenditure would be 1200 USD due to transaction costs, which is a non-trivial amount compared to the traded volume. We study this problem in-depth and propose three algorithms to perform optimal IAT for different applications.

The main contributions of this work are summarized below.

• A self-attention NN architecture for cryptocurrency trading is proposed. The NN is trained using DRL and is independent of the number of assets in the market.
• An analysis of the information flow inside the NN is given, explaining how and why the architecture works.
• The problem of IAT is studied in depth. This includes the description and theoretical justification of three algorithms, and the evaluation of the speed and accuracy of the algorithms.

The rest of the paper is organized as follows. Section 2 discusses the relevant points of related literature. Section 3 presents the mathematical formulation of the problem. Section 4 describes various algorithms that allow the computation of optimal transaction fees. Section 5 describes the deep neural network (NN) architecture used for cryptocurrency trading as well as the training method. Section 6 contains the setup of the experiments implemented to evaluate the performance of our method. Section 7 presents the results of the experiments along with a discussion of the most relevant findings. Finally, conclusions are drawn in Section 8.

2. Literature Review

One of the earliest studies on reinforcement learning applied to asset trading can be found in [3]. In that work, the authors proposed an algorithm named recurrent reinforcement learning (RRL) applied to stock market trading. The authors explored the use of well-known financial measures, such as the Sharpe ratio, as reward functions in the process. Those ideas were expanded in [4], where they compared their approach to other methods. Later on, Gold [5] used a similar approach for currency trading in the foreign exchange (FX) market and focused on comparing different neural networks (NNs) to obtain an optimal architecture for the problem. Dempster and Leemans [6] proposed adaptive reinforcement learning to the FX market. The method is a complex system, in which decisions are based on performance, risk measures, and a dynamic optimization process. Maringer and Ramtohul [7] proposed a threshold RRL method, which uses two networks trained in tandem to deal with two different market states, one for a volatile market and the other for a non-volatile one. They argued their approach outperforms the original RRL because it is difficult to model the entire behavior of a market with a single network. Zhang and Maringer [8] also based their approach on RRL and showed the results of technical and fundamental analysis can be used as inputs for the neural network along with price data. However, the approach requires human expertise to handcraft the indicators to be fed into the network.

More recently, due to the advancement of NN techniques, training deeper and more powerful NNs using reinforcement learning has become feasible [9], making architectures such as convolutional and recurrent networks popular in asset trading research. Deng et al. [10] proposed recurrent neural networks (RNNs) to trade stocks and used fuzzy logic to determine the trending state of the market. Jiang and Liang [11] used a policy gradient method to train a convolutional neural network (CNN) for cryptocurrency trading. Bu and Cho [12] proposed a two-step approach for cryptocurrency trading with a pre-trained deep RNN using a Boltzmann machine, and they trained it using the Double Q-learning algorithm, reporting positive returns even when the market state did not have an increasing tendency. Pendharkar and Cusatis [13] compared the performance of multiple DRL techniques for stock market training. Liang et al. [14] compared multiple DRL methods for asset trading as well, but they proposed adversarial training to all the studied methods to improve model performance. Jeong and Kim [15] used Q-learning to train a deep NN to trade stocks. In addition, they applied transfer learning to the case where a low amount of data is available. Wu et al. [2] studied the effects of gated recurrent units (GRUs) [16] as time series feature extractors for trading a single asset in a stock market. They compared two different approaches: deep Q-learning and policy gradient [17]. They found both methods are appropriate for asset trading and concluded trading a single asset is risky and diversifying investments should be preferred. Aboussalah and Lee [18] proposed a method named stacked deep dynamic reinforcement learning (SDDRL) for real-time stock trading, and argued the selection of the appropriate hyper-parameters is especially important in this type of problem. To deal with this issue, they proposed a Bayesian approach for hyper-parameter tuning. Park et al. [19] proposed an LSTM [20] network trained with deep Q-learning for stock market trading. Lei et al. [21] proposed a similar approach using a GRU network trained with Policy Gradient.

In a previous work [22], we proposed a DRL method for cryptocurrency trading, which can adapt to new assets introduced suddenly to the market. This work follows that line of study and tackles some important issues that were not covered in the mentioned study. One of the differences between this study and [22] is the architecture of the NN. In [22], the combination between the extracted features from the assets is a softmax layer. That layer simply normalizes the values proposed by the feature extractor to generate the final output. To allow a real exchange of information between the features of the assets, we integrate a series of self-attention layers into the new architecture. These layers take the information from each asset one by one, and once all the data is received, it generates the output corresponding to each asset. Therefore, the new blending mechanism has a general overview of the market as well as the individual information of the assets, allowing it to generate more reliable outputs. Another difference between this work and [22] is the inclusion of IAT in the asset exchange process. This allows the new method to spent the optimal amount of capital during the transactions, which is an important issue when assets are exchanged constantly. The proposed methods for IAT are covered in detail in Section 4.

Table 1. Literature review summary.

Work	Year	Main Contributions
Moody et al. [3]	1998	Proposed the RRL method.
Moody and Saffel [4]	2001	Expanded RRL and compared to other
		approaches.
Gold [5]	2003	Applied RRL to the FX market.
Dempster and Leemans [6]	2006	Proposed an adaptive reinforcement
		method for trading in the FX market.
Maringer and Ramtohul [7]	2010	Proposed a threshold RRL method.
Zhang and Maringer [8]	2013	Combined RRL with a technical and
		fundamental analysis for stock trading.
Deng et al. [10]	2016	Combined fuzzy logic and RNNs for
		stock trading.
Jiang and Liang [11]	2017	Trained CNNs using Policy Gradient
		for cryptocurrency trading.
Bu and Cho [12]	2018	Used RNNs and Boltzmann machines
		for cryptocurrency trading.
Pendharkar and Cusatis [13]	2018	Compared the performance of
		multiple DRL techniques for stock
		market trading.
Liang et al. [14]	2018	Proposed an adversarial training
		method to enhance DRL methods.
Jeong and Kim [15]	2019	Used Q-learning and transfer learning
		for stock trading.
Wu et al. [2]	2020	Used GRUs for stock market trading.
Aboussalah and Lee [18]	2020	Proposed the method SDDRL for
		stock market trading.
Park et al. [19]	2020	Trained an LSTM network using
		Q-learning for stock market trading.
Lei et al. [21]	2020	Trained a GRU network using Policy
		Gradient for stock market trading.
Betancourt and Chen [22]	2021	Proposed a NN architecture for markets
		with a dynamic number of assets.

summarizes the main contributions of the discussed works.

3. Problem Formulation

A trading session is the total amount of time in which an agent is allowed to conduct transactions to generate profits. The session is divided into equal-length steps, where at each of them, the following steps take place: selection, exchange, and waiting. The selection step is carried out by a neural network, which computes a non-negative scalar for each asset, representing the weight or percentage of that asset with respect to the entire set of investments to be held in the coming period. The design of the NN used in the selection step and its training procedure is described in detail in Section 5. After the selection process, some of the assets held by the agent have to be exchanged to obtain the assets chosen by the selection procedure. The transaction fees pre-established by the trading service provider are considered in computing the quantities to be exchanged. The algorithms developed for computing the transactions that lead to the lowest possible transaction fees are described in Section 4. After computing the transactions, these are executed in the market and a waiting period begins. This period allows the investments to mature and generate profits or losses to the agent. Once the waiting period finishes, the process moves to a new step, beginning with the selection of new assets for the new period. The process is repeated until the trading session is complete.

Let an asset in the market and a step in the trading session be i and k, respectively. The shares corresponding to each asset i held by the agent at the beginning of step k are denoted by $s_i^k$. If there are N assets in the market, then $i\in \{0,1,\dots ,N-1\}$. The index 0 is reserved for the cash asset, whose value is assumed to be constant throughout the trading process. Therefore, the asset is used as a reference, i.e., the shares of all the assets ($s_i^{k+1}$) are measured in units of this asset. In stock markets, the USD is used for this purpose, but in cryptocurrency markets, the most popular choice is the USDT. The sum of $s_i^{k+1}$ for all i, denoted $s^{k+1}$, is the total capital of the investor at the beginning of step k. The variables corresponding to the shares held by the agent at the end of period k are denoted similarly; ${\tilde{s}}_i^k$ represents the individual asset shares and ${\tilde{s}}^k$ is their sum. The trading process can be summarized as follows. The investor begins the trading session with a capital $s^0$ distributed among some assets ${\tilde{s}}_i^0$. Then, the neural network suggests some suitable assets for period $k=1$, and some assets are exchanged. The result of those transactions is the $s_i^1$. Then, a waiting period follows, allowing the values of the assets to evolve into ${\tilde{s}}_i^1$, reaching the end of the step, where a new set of assets is chosen by the NN for period $k=2$. This process is repeated until reaching the final step K, where the performance of the agent is computed using the formula: ${\tilde{s}}^K-{\tilde{s}}^0$, which is the difference between the final and initial capital.

The transition between period k and $k+1$ consists of a set of transactions summarized in Equation (1), where both i and j represent each integer from 0 to $N-1$. Thus, Equation (1) is in fact a system of N equations, one for each asset. Note, for simplicity, the upper limit in the summation symbols has been dropped; the upper limit in all expressions is $n-1$ unless otherwise specified. The amount transferred from asset i to asset j and the fee for a transaction is written as $t_{ij}^k$ and $f_{ij}$, respectively. Equation (1) can be interpreted as follows. The shares of asset i after executing the transactions ($s_i^{k+1}$) are equal to the shares before the transactions (${\tilde{s}}_i^k$), minus the shares transferred from that specific asset to each of the other asset ($t_{ij}^k$), minus the fees spent for transactions ($f_{ij}{t}_{ij}^k$), plus the shares received from the rest of assets ($t_{ji}^k$). Note, all the variables in Equation (1) are non-negative numbers.

$$s_i^{k+1}={\tilde{s}}_i^k-\sum _j{t}_{ij}^k-\sum _j{f}_{ij}{t}_{ij}^k+\sum _j{t}_{ji}^k,i,j\in \{0,1,\dots ,N-1\}$$

(1)

Then, since the output of the NN is the relative weights of the assets with respect to the total capital, Equation (1) has to be modified to contain the asset weights instead of the shares. This is accomplished by dividing Equation (1) by ${\tilde{s}}^k$, as in [22], resulting in Equation (2), where the old and new weights are denoted by ${\tilde{w}}_i^k$ and $w_i^{k+1}$, respectively, ${\tau }_{ij}^k$ are the transfers with respect to the weights, and ${\rho }^{k+1}$ is a rate representing the capital lost due to the transactions. The definition of these variables is shown in Equation (3).

$${\rho }^{k+1}{w}_i^{k+1}={\tilde{w}}_i^k-\sum _j(1+f_{ij}){\tau }_{ij}^k+\sum _j{\tau }_{ji}^k$$

(2)

$$w_i^{k+1}\stackrel{\mathrm{def}}{=}\frac{s_i^{k+1}}{s^{k+1}},{\tilde{w}}_i^k\stackrel{\mathrm{def}}{=}\frac{{\tilde{s}}_i^k}{{\tilde{s}}^k},{\tau }_{ij}^k\stackrel{\mathrm{def}}{=}\frac{t_{ij}^k}{{\tilde{s}}^k}\mathrm{and}{\rho }^{k+1}\stackrel{\mathrm{def}}{=}\frac{s^{k+1}}{{\tilde{s}}^k}$$

(3)

Note, the sum of all the weights $w_i^{k+1}$ equals one, and the same is true for the weights ${\tilde{w}}_i^k$. Therefore, by adding the N expressions represented by Equation (2), i.e., from i equals 0 to $N-1$, and simplifying the result, we arrive at Equation (4), providing an easy way to evaluate the discount rate ${\rho }^{k+1}$ that becomes one minus the fees due to all the transactions. Therefore, it can be used as the objective function of an optimization process to compute the optimal exchange amounts. Note, since it is not possible to transfer any amount from an asset to itself, both ${\tau }_{ii}^k$ and $f_{ii}$ are assumed to be zero, leading to Equation (4).

$${\rho }^{k+1}=1-\sum _i\sum _j{f}_{ij}{\tau }_{ij}^k$$

(4)

The optimal solution for the exchange fee minimization problem must satisfy Equation (2) for all $i\in \{0,1,\dots ,N-1\}$ and maximize ${\rho }^{k+1}$ in Equation (4). In addition, the solution has to satisfy Equation (5) for all i as well. The set of inequalities in Equation (5) simply states the sum of the transferred amounts from one asset to the rest cannot exceed the amount held of that specific asset at that moment. The process described in this section is summarized in Figure 1.

$$\sum _j(1+f_{ij}){\tau }_{ij}^k\le {\tilde{w}}_i^k$$

(5)

4. Exchange Fee Minimization

This section describes the process for computing the quantities that need to be exchanged to satisfy the asset weights suggested by the NN leading to the least amount of spent cash in transaction fees.

4.1. Approximate Solution for IAT

Given a certain value of ${\rho }^k$, the transfer direction of any asset (buy or sell) is determined by the difference between ${\tilde{w}}_i^k$ and ${\rho }^k{w}_i^{k+1}$ (i.e., ${\tilde{w}}_i^k-{\rho }^k{w}_i^{k+1}$), denoted by $r_i$. If this residual is positive, then the asset has an excess in value that has to be sold to balance Equation (2); on the contrary, if $r_i$ is negative, shares of that asset need to be bought to balance the equation. To reduce expenses, the transfer fees ($f_{ij}$) are sorted; by doing this, the transfer values are determined one by one from the pairs of assets ($ij$) corresponding to the lowest fees. First, the smallest fee is selected and the corresponding asset indices are retrieved. With them, two conditions are checked: $r_i>0$ and $r_j<0$. If both of the constraints are satisfied, ${\tau }_{ij}^k$ is non-negative, and its value is determined by Equation (6). Then, $r_i$ and $r_j$ are updated using Equation (7). In this way, the value of ${\tau }_{ij}^k$ becomes uniquely determined by ${\rho }^k$. This procedure is repeated for all pairs of assets corresponding to the sorted transaction fees from lowest to highest. When the routine is finished, a new value for ${\rho }^k$ is computed using Equation (4), which is used to start a new iteration of the process. The procedure is repeated until ${\rho }^k$ converges. The values of the variables corresponding to the final value of ${\rho }^k$ are the solution to the problem. This procedure is summarized in Algorithm 1.

$${\tau }_{ij}^k=\min (r_i/(1+f_{ij}),-r_j)$$

(6)

$$\begin{array}{cc}\hfill r_i& \leftarrow r_i-(1+f_{ij}){\tau }_{ij}^k\hfill \\ \hfill r_j& \leftarrow r_j+{\tau }_{ij}^k\hfill \end{array}$$

(7)

The solutions of Algorithm 1 are generally better than those of Algorithm 2 because Algorithm 1 uses IAT. Plus, it is much faster than Algorithm 3 because it does not use convex optimization to compute the solution. Therefore, Algorithm 1 is adopted and integrated into the cryptocurrency trading environment because it best balances the tradeoff between speed and optimality. A comparison of performances of these methods is shown in Section 6.

Algorithm 1 Approximate solution for the general fee minimization problem

Inputs:$f_{ij},w_i^{k+1}\mathrm{and}{\tilde{w}}_i^k\mathrm{for}i,j\in \{0,1,\dots ,N-1\}$        ▹ Fees and weights

Outputs:${\rho }^k\mathrm{and}{\tau }_{ij}^k\mathrm{for}i,j\in \{0,1,\dots ,N-1\}$    ▹ Discount rate and transactions

1: ${\rho }^k\leftarrow 1-\max \left(f_{ij}\right)/2$                         ▹ Initialize ${\rho }^k$

2: repeat

3:  for $i=0\mathrm{to}N-1$ do

4:   $r_i\leftarrow {\tilde{w}}_i^k-\rho w_i^{k+1}$                      ▹ Compute residuals

5:  for $i,j$ in $\mathrm{argsort}\left(f_{ij}\right)$ do                ▹ Sort in increasing order of f

6:   if $r_i>0\mathrm{and}{r}_j<0$ then

7:    ${\tau }_{ij}^k\leftarrow \mathrm{Equation}(6)$                  ▹ Assign value to ${\tau }_{ij}^k$

8:    $r_i,r_j\leftarrow \mathrm{Equation}\left(7\right)$                   ▹ Update residuals

9:  ${\displaystyle {\rho }^k\leftarrow \mathrm{Equation}(4)}$                          ▹ Update ${\rho }^k$

10: until ${\rho }^k$ converges

11: return ${\rho }^k\mathrm{and}{\tau }_{ij}^k\mathrm{for}i,j\in \{0,1,\dots ,N-1\}$

Algorithm 2 Solution for the constrained case of the fee minimization problem

Inputs:${\rho }_0,f_{i0},f_{0i},w_i^{k+1}\mathrm{and}{\tilde{w}}_i^k\mathrm{for}i\in \{0,1,\dots ,N-1\}$   ▹ Fees and weights

Outputs:${\rho }^k,{\tau }_{i0}^k\mathrm{and}{\tau }_{0i}^k\mathrm{for}i\in \{0,1,\dots ,N-1\}$  ▹ Discount rate and transactions

1: ${\rho }^k\leftarrow {\rho }_0$                               ▹ Initialize ${\rho }^k$

2: repeat ${\rho }^k\leftarrow$ Equation (A7)                        ▹ Update ${\rho }^k$

3: until ${\rho }^k$ converges

4: for $i=0\mathrm{to}N-1$do

5:  ${\tau }_{i0}^k\leftarrow {({\tilde{w}}_i^k-\rho w_i^{k+1})}^{+}/(1+f_{i0})$            ▹ Compute ${\tau }_{i0}\mathrm{and}{\tau }_{0i}^k$

6:  ${\tau }_{0i}^k\leftarrow {(\rho w_i^{k+1}-{\tilde{w}}_i^k)}^{+}$

7: return ${\rho }^k,{\tau }_{i0}^k\mathrm{and}{\tau }_{0i}^k\mathrm{for}i\in \{0,1,\dots ,N-1\}$

Algorithm 3 Exact solution for the general fee minimization problem

Inputs:$f_{ij},w_i^{k+1}\mathrm{and}{\tilde{w}}_i^k\mathrm{for}i,j\in \{0,1,...,N-1\}$       ▹ Fees and weights

Outputs:${\rho }^k\mathrm{and}{\tau }_{ij}^k\mathrm{for}i,j\in \{0,1,\dots ,N-1\}$    ▹ Discount rate and transactions

1: ${\rho }^k\mathrm{and}{\tau }_{ij}^k\leftarrow$ Solve the Linear Program of Equation (A9)    ▹$i,j\in \{0,1,\dots ,N-1\}$

2: return ${\rho }^k\mathrm{and}{\tau }_{ij}^k\mathrm{for}i,j\in \{0,1,\dots ,N-1\}$

In the following subsections, we evaluate the efficiency of Algorithm 1 by comparing its performance against the exact general solution for the IAT problem, and the solution for the case where IAT is not allowed. These algorithms are described in Appendix A.

4.2. Evaluation of the Fee Minimization Algorithms

The properties of the fee minimization algorithms are compared in these experiments to determine the real applicability of each algorithm. Algorithms 1–3 are tested along the method introduced in [22], which is used as a baseline and abbreviated as BCH-21 in the rest of the paper. These algorithms are compared in two aspects: computation time and error with respect to the optimal value. Note, the computation times of these algorithms change depending on the number of assets in the market; the larger the number of assets, the greater the number of mathematical operations needed to solve the problem, which in turn implies a longer computation time. This is important because the fee minimization algorithm needs to be executed constantly during the training process. The function describing the growth in computing time for an algorithm given the size of the input is known as the complexity of the algorithm. Computing this function exactly can be difficult [23]. Therefore, instead, we follow an empirical approach. For each integer value from one to 200, 1000 pairs of vectors are randomly generated and fed to all the studied algorithms. In each generated pair, one of the vectors represents the asset quantities held by the agent at a certain time, and the other vector represents the quantities proposed by the NN to be held in the next step of the process. Thus, by tabulating the computation times obtained by each algorithm, we can construct graphs showing how the computation time increases with respect to the number of available assets in the market.

The same experiments allow us to measure how close the values computed by the studied algorithms are with respect to the optimal value. Algorithm 3 always computes the optimal solution for the problem, as proven in Appendix B. Thus, the error is simply the difference between the value computed by an algorithm and the value computed by Algorithm 3. The computation times of these algorithms grow with respect to the number of inputs, but for Algorithm 3, the growth is too fast to be practical for markets with a large number of assets. For this reason, the best algorithm for this application is not necessarily the one that always retrieves the optimal values or the one with the smallest computation time, but it is the one with the best tradeoff between these two measures.

4.3. Performance of the Fee Minimization Algorithms

The computation time obtained in our experiments is shown in Figure 2. As expected, Algorithm 3 needs the longest time to produce the solutions. Further, for 125 assets or more, it requires more than a second to find the solution. This disproportionate growth in computation time, compared to the other methods, is due to the elements added to the formulation of the problem for each new asset, including two new variables: one equality and one inequality. The results of the other analyzed algorithms are significantly better. Algorithm 1 and BCH-21 are faster than Algorithm 3 by 10 to 100 times, respectively. As can be seen in Figure 2, the graphs of the algorithms are highly correlated, and the highest registered time for both of them is around 0.1 s. Nonetheless, since Algorithm 1 allows IAT, the transactions computed by Algorithm 1 are in general better than those computed by BCH-21. Yet, the fastest method by a significant margin is Algorithm 2. The speed of Algorithm 2 is orders of magnitude faster than any other tested algorithm. Further, the increments in computation time are barely noticeable along with the tested range.

The results of the optimality tests are shown in Figure 3 and Figure 4. The first figure shows the error of the solutions computed by both Algorithm 2 and BCH-21. Note, given the same inputs, both of the algorithms always compute the same solution. According to Figure 3, the error oscillates around $5\times {10}^{-4}$ and is consistent throughout the tested range. However, Algorithm 1 obtained much better results in this experiment, as shown in Figure 4. The solutions computed by this method are a million times closer to the optimal value than those computed by Algorithm 2 and BCH-21.

Using the two measures, computation time and optimality, we can clearly see the advantages and disadvantages of the tested algorithms. Algorithm 3 produces the optimal solution for any input. Therefore, if IAT is available in the market, Algorithm 3 should be preferred over any other studied algorithm. However, it is only practical if the number of assets is sufficiently small or the waiting periods are sufficiently large. Otherwise, Algorithm 1 should be used instead. However, if the market does not have the option for IAT, Algorithm 2 is the best choice because its solutions are optimal in that specific case. Algorithm 2 is also appropriate for long iterative routines such as a DRL process because the extra time it adds to the process is minimal. However, we opted to use Algorithm 1 because it has the best tradeoff between speed and optimality. Among all the studied algorithms, BCH-21 is the least advantageous one. However, it is important from a theoretical view because the feasibility of BCH-21 is used to prove the optimality of Algorithm 3.

5. DRL for Asset Trading

DRL is a type of machine learning combining deep learning with a DRL design to enable agents in a given state to take actions that maximize their rewards in a virtual world. State, action, and reward are three key concepts of reinforcement learning. At each state, the agent observes its surroundings and takes an action based on the observation. Then, the state of the environment changes to a new state and the agent receives a numerical reward based on its actions. The changes between states are driven by both the environment dynamics and the agent’s actions. Thus, good actions are those producing high rewards or driving the environment into states where the expected reward can be maximized. The process is complete when a maximum number of states has been visited or a stop condition is met. The period from the initial state to the terminal state is called an episode. The sum of all the rewards collected by the agent during an entire episode is the total reward. Therefore, the purpose of reinforcement learning is to train agents to learn a policy that can make decisions to produce the maximum possible total reward in a specific environment.

In our case, the environment is the cryptocurrency market, the agent is the investor and the rewards are the profits generated in the process. An action in our environment is the execution of the transactions needed to acquire the assets suggested by the NN. The NN observes the environment state, i.e., market prices and volumes, and outputs a set of suggested initial asset weights for the subsequent period of the process. Then, the fee minimization algorithm computes the necessary transactions to acquire the assets suggested by the NN with the minimal possible transaction fees. Next, the transactions are executed, allowing the market to evolve into a new state. This process is depicted in Figure 1. The rewards received by the agents are the earnings or losses received at the end of each step due to the chosen transactions.

Our NN is not limited by the number of assets in the market. The initial layers of the NN process the data of each asset individually. Then, the generated feature vectors are fed one by one to a series of self-attention layers which produce the initial asset weights for the subsequent step of the process. Self-attention networks receive and output sequences of data. This characteristic allows our NN to process a large number of assets and continue working even if new assets are added to the market, which happens often in cryptocurrency markets.

Our agent is allowed to perform daily transactions. Thus, an episode in the environment consists of a specific number of days, in which the agent is allowed to buy and sell assets at every 24-h time interval. The state of the environment consists of the current prices, capitalizations, and volumes of the assets in the market. Of which, the agent is allowed to observe the data corresponding to the latest 20 days. The action consists of the set of transactions made by the agent before a 24-h waiting period, and it is executed in two steps: asset selection (done by the NN) and transaction fee minimization (done using Algorithm 1). The reward of the step is the capital gained or lost by the agent after the waiting period.

5.1. Implementation Details

The environment was built using data from the cryptocurrency market: Binance www.binance.com (accessed on 3 November 2020), and it corresponds to the period: August 2017 to November 2020. The features of the dataset are summarized in

Table 2. Dataset properties.

Feature	Value
Initial date	17 August 2017
Final date	3 November 2020
# of days (training)	809
# of days (test)	365
# of assets	3 to 227
# of indicators	9
Sampling time	30 min
# of entries (training)	38,832
# of entries (test)	17,520
Fees (except BNB)	0.1%
Fees BNB	0.05%

. The dataset was divided into two parts: training and test, where the test dataset is the last year in the dataset. Note, the total number of financial indicators in the dataset is nine, but only six of them were used: open, close, high, and low prices, plus volume, and the number of transactions per sampling period.

5.2. The Deep NN for Asset Trading

An NN processes numeric data of the market’s assets, such as prices and volumes, and outputs the distribution of assets to be held in the subsequent periods of the process. The initial layers of the network extract independent high-level features for each asset, and a set of self-attention layers [24] blend that information to produce the output. The components of our NN are shown in Figure 5. The data fed to the NN is an arrangement of financial indicators with dimensions $F\times N\times K$, where F is the number of financial indicators, N is the number of assets, and K is the number of past steps the agent is allowed to observe. The number of financial indicators equals six, as mentioned in Section 5.1. F is considered to be the number of channels of the input data. The number of assets in the dataset increases over time from 3 to 227 (see Table 2); therefore, N is not a constant. The number of past steps (K) depends on two factors: the number of days and sampling time. The data used in our experiments has a sampling time of 30 min, and the number of days our agent is allowed to observe is 20; therefore, K equals 960.

Since the channels of the data have a very different range of values, normalization was applied before feeding to the NN. The normalization consists of subtracting the mean from each channel and dividing the results by the standard deviation of the channel, resulting in a normal distribution with zero mean and unit standard deviation. Then, for each asset, the normalized inputs are independently passed through a sequence of two convolutional layers, which are the primal feature extraction mechanism of the NN. The extracted features are flattened and passed through a fully connected (FC) layer to produce a set of feature vectors. The feature vectors are directed one by one to the blending mechanism of the network, consisting of a set of three self-attention encoders [24].

The design of the self-attention encoders used in our NN is shown in Figure 6a. The self-attention encoders consist of two different sub-layers. The first sub-layer is a multi-head self-attention block, meaning it contains parallel feature extractors (heads), which generate independent feature vectors from the inputs. The number of heads in our self-attention blocks is eight. The distinct feature vectors generated by the heads are combined into a single feature vector using a linear layer, as shown in Figure 6b. Then, these results are fed to a normalization layer [25], which has a residual connection [25] to generate the output of the first sub-layer.

Each head in a self-attention encoder has three matrices: query ($W_Q$), key ($W_K$), and value ($W_V$), which are the parameters to be trained. When a self-attention encoder receives the feature vectors corresponding to the assets, the feature vectors are concatenated and multiplied by the matrices $W_Q$, $W_K$, and $W_V$ to obtain the matrices Q, K and V, which are used to compute the output of the self-attention head using the Equation (1) of [24], which computes the amounts of attention having to be given to all the assets when generating the output of each asset.

The second sub-layer of the self-attention encoder also has two blocks. The first block is a feed-forward NN with two layers. The number of nodes in each layer is 400. The second block is a normalization layer with a residual connection, which produces the final output of the self-attention encoder. We use three identical self-attention encoders placed in sequential order as the blending mechanism of our design. The output of the blending mechanism is a high-level feature vector for each asset, containing information about every asset in the market. The generated feature vectors individually go through an FC layer to output a single scalar for each asset. These scalar values are normalized by a softmax layer to obtain the asset weights to be held in the next period.

We use an actor-critic design for the agent to learn an optimal policy that maximizes cumulative rewards. Therefore, the NN has two outputs: actor and critic. The actor determines which action to take and the critic evaluates how good the action was. The value function consists of another FC layer that takes the output of the self-attention layers and produces a scalar for each asset. Then, the output of the value function is simply the average of the scalar values.

5.3. Training

Our agent was trained using the ‘synchronous’ version of the Asynchronous Advantage Actor-Critic method (A3C) [26], introduced by Open AI, known as A2C. https://openai.com/blog/baselines-acktr-a2c/ (accessed on 2 August 2021). Therefore, the NN has two parts: actor and critic. The actor, also known as the policy, receives environment states and outputs actions. The critic, on the other hand, predicts the total reward the agent will receive at the end of the process by following the actions given by that specific policy. The policy is represented by ${\pi }_{\theta }$ and the value function is represented by $v_{\theta }$. In our implementation, both the actor and critic are combined into a single NN with two independent output layers. Hence, they are trained simultaneously applying Adam optimizer [27] to the cost function shown in Equation (8), where $s_t$ is the state, $a_t$ is the action taken in that state, $r_t$ is the reward received, A is a variable named the advantage, and R is the total discounted reward (see [26]). The values of the parameters used in the optimization process are listed in

Table 3. List of training parameters.

Parameter	Value
# of workers	16
# of step per episode	4
# of step per rollout	4
# of optimization steps	150,000
Learning rate ($\alpha$)	7 × 10${}^{-7}$
Adam opt. param. (${\beta }_1$)	0.9
Adam opt. param. (${\beta }_2$)	0.999
Decay factor ($\gamma$)	0.99
A2C parameter ($\eta$)	0.5
Advantage parameter ($\lambda$)	0.95

. The values were chosen by comparing the performance of training sessions, which used hyperparameter values in the ranges shown in

Table A1. Fine tuning parameters.

Parameter	Range	Increment
# of optimization steps	100,000–200,000	10,000
Learning rate ($\alpha$)	1 × 10${}^{-7}$–1 × 10${}^{-6}$	1 × 10${}^{-7}$
A2C parameter ($\eta$)	0.1–0.7	0.1
# of masks in CNN layers	8–32	8
# of SA layers	1–8	1
# of heads SA layers	8–32	8

. We used 16 workers for data collection. Thus, in each iteration, we randomly draw 16 days from the training dataset to be the initial days of each trading session, and the workers execute trades for four days. Then, this data is used to compute the cost function, which in turn is used to improve both the policy and value function together.

$$\begin{array}{c}{L}_{A2C}\left(\theta \right)={\mathrm{L}}_{\pi }\left(\theta \right)+\eta L_v\left(\theta \right)\hfill \\ {\mathrm{L}}_{\pi }\left(\theta \right)=\mathbb{E}\left[{\displaystyle \sum _{t=0}^{T-1}}\log {\pi }_{\theta }\left(a_t\right|s_t)A(s_t,a_t)\right]\hfill \\ {\mathrm{L}}_v\left(\theta \right)=\mathbb{E}\left[{\left(v_{\pi }\left(s_t\right)-R_t\right)}^2\right]\hfill \end{array}$$

(8)

5.4. Layer Analysis

In this subsection, we show how the information flows through the NN layers and give an interpretation of the meaning of the features computed by some of the layers of the NN. Recently, Makridakis et al. [28] made a critic about the performance of machine learning approaches on time-series forecasting tasks, which are closely related to our application. They state that ML researchers tend to over-utilize resources on tasks where simpler statistical methods can produce better results. Therefore, to show the relevance of each of the components of our NN, we fed our NN with the data of four popular cryptocurrencies drawn randomly from the dataset and analyzed the results computed by different layers.

The input layer of the NN is a normalization layer. The result of the normalization operation is shown in Figure 7. Both graphs are practically the same. However, an important difference is that, in the normalized data, most of the values corresponding to LINK are negative. This is useful as those values can be ruled out by the ReLU operation that takes place after the convolutional layers. Additionally, it has been shown that normalizing the inputs makes the training process faster and more stable [29].

The next group of layers is a series of convolutional layers. These layers have two functions: feature extraction and data size reduction. Figure 8 shows the features extracted by one of the convolutional masks of the first convolutional layer. As shown in the figure, this layer highlights assets that have significant positive changes in value, which is a typical convolution operation. We can see, as expected, that the features of the asset in green have been flattened out almost completely. These simple features are combined in deeper layers into abstract complex features which are given to the self-attention layers.

The final part is a sequence of self-attention layers, which choose the assets to be acquired in the subsequent step of the process. The results of the heads of the last self-attention layer are shown in Figure 9. The figure shows that most of the heads in the last layer focused their attention on Bitcoin, Ethereum, and Litecoin, which are the cryptocurrencies that increased the most their value in the tested sample. This shows that the layer can process the features extracted by the convolutional layers and determine the assets with higher chances to continue being in an increasing tendency.

6. Experiments

The experiments presented in this section evaluate the performance of the proposed NN architecture. These experiments are performed in an environment built using real data from a well-known cryptocurrency trading platform, allowing us to point out issues and make predictions about the performance of this system when implemented in a real market. The effects of including transactions fees in the training process are evaluated as well. We trained our model in an environment with transaction fees, and then we trained a clone of the model without fees. Next, both models were tested in the environment with transaction fees to observe the need of including the subprocess during the training stage. If a significant difference in performance cannot be observed during that experiment, this subprocess could be omitted in future training runs and only be used during deployment to save an important amount of time. We also validated the use of self-attention networks by comparing the performance of our design against a similar approach [22]. This architecture was also designed for markets with a large and non-constant number of assets, but it does not include self-attention networks in its design.

The agents are tested in multiple setups. In the first setup, each day in the test dataset was selected as an initial point, and each of the models performs a four-day trading session. The profits obtained by the models during the short experiments are recorded, and their average is taken as the overall performance. In the second setup, we run the agents throughout the test dataset in a single session (for 365 days), and the total profit is recorded as the overall performance. This allows us to analyze the robustness of the models to changes in the setup. Additionally, the algorithms were tested in the same setups of [22] and their performances were analyzed and compared. This includes a two-day trading period with transactions every 30 min, and a two-week trading period with transactions every six hours.

7. Results and Discussion

Performance of the Models

The performance of the tested models in the four-day experiment is listed in

Table 4. Comparison of the performance of the algorithms in a short and long trading episode.

Model	Four-Day Episode		One-Year Episode
	Mean	Standard Deviation	Total Profit
SA-NET	0.043	0.11	4.66
SA-NET-NF	0.034	0.10	3.80
Betancourt and Chen [22]	0.030	0.07	2.24

. Both networks using self-attention networks (SA-NET) introduced in this study outperform the baseline, showing the benefits of using self-attention networks to process data from a large pool of assets and extract high-level features. SA-NET must incur transaction fees during training, and it obtained average daily profits of 4.3%, while the model without fees (SA-NET-NF) obtained 3.4%. Therefore, the model trained in an environment with fees can make better decisions. However, the difference is not significant, meaning the most important factor to determine the performance of the two models is the self-attention networks.

In the setup with an episode length of two days, the results are similar among the competing algorithms. The algorithm that performed best is [22]. However, the difference with the respect to the rest of the tested algorithms is 0.1%. Additionally, the standard deviation of the experiments is large compare to the average performance. Therefore, we can conclude that the algorithms have similar performances when implemented in short trading episodes. These results are shown in

Table 5. Comparison of the performance of the algorithms in the two-day and two-week trading episode.

Model	Two-Day Episode		Two-Week Episode
	Mean	Standard Deviation	Mean	Standard Deviation
SA-NET	0.028	0.07	0.33	0.30
SA-NET-NF	0.027	0.06	0.31	0.26
Betancourt and Chen [22]	0.029	0.06	0.28	0.33

. In the two-week setup, the results have a bigger spread. The algorithm that performed best in that setup is SA-NET. Again, the standard deviation of the performance is large. This is a characteristic of an extremely volatile market. However, the difference between the proposed approach and that of [22] is around 5%, showing that in longer episodes, the proposed NN outperforms the NN of our previous work. Figure 10a,b show examples of the performance of the tested algorithms in the two-day and two-week setup.

The results of the one-year setup, shown in Figure 10c, also demonstrate the self-attention design as the best alternative. However, despite obtaining a 5-fold return in one year, SA-NET is also the model suffering the worst losses during the run, which is evidence of taking riskier actions than the competing models. Nonetheless, the order of performance of the tested methods is consistent with both experiments. Therefore, the range of applicability of our system is flexible, showing the robustness of the presented approach.

8. Conclusions

We introduced a DRL system for cryptocurrency trading. Our system uses self-attention networks to simultaneously process data of a market with a large number of assets. Our deep NN has been trained using data from a real cryptocurrency market to execute daily trades. The experimental results show the methodology presented in this work outperforms the baseline, obtaining higher profits in two different setups, illustrating self-attention networks can extract meaningful relations from chaotic cryptocurrency data. We also studied the impact of transaction fees in cryptocurrency trading and presented three optimization solutions to deal with this important issue. The best solution has been integrating into the system to create a robust tool for cryptocurrency trading. Future work includes the optimization of the layers of the NN architecture to boost the performance and the application of the proposed system in stock markets and the FX market.