Optimal Reinforcement Learning-Based Control Algorithm for a Class of Nonlinear Macroeconomic Systems

Abstract

Due to the vital role of financial systems in today’s sophisticated world, applying intelligent controllers through management strategies is of crucial importance. We propose to formulate the control problem of the macroeconomic system as an optimization problem and find optimal actions using a reinforcement learning algorithm. Using the Q-learning algorithm, the best optimal action for the system is obtained, and the behavior of the system is controlled. We illustrate that it is possible to control the nonlinear dynamics of the macroeconomic systems using restricted actuation. The highly effective performance of the proposed controller for uncertain systems is demonstrated. The simulation results evidently confirm that the proposed controller satisfies the expected performance. In addition, the numerical simulations clearly confirm that even when we confined the control actions, the proposed controller effectively finds optimal actions for the nonlinear macroeconomic system.

1. Introduction

Nowadays, the control and synchronization of nonlinear systems have attracted attention due to their appreciable usage in a wide variety of fields [1,2,3,4,5,6,7]. Up to now, intelligent control systems have effectively solved some problems in the control of modern tools and methods, such as self-organizing maps, neural networks, fuzzy logic, expert systems and various nature-inspired algorithms [8,9]. Several research studies have applied neural networks as disturbance estimators to observe the disturbances and uncertainties in nonlinear systems [10,11,12]. On the other hand, the robustness of the control strategies plays an inevitable role in the performance of the systems. Hence, researchers also propound robust controllers for complex systems [13,14,15].

A macroeconomic model describes the dynamics of an economic system over long- and short-run periods. The macroeconomic model can consist of difference or differential equations. Usually, the variables in the macroeconomic model are categorized into the following three main classes: (1) variables that represent the state of the system, (2) variables that indicate an uncontrollable environment and (3) control variables that denote leverage, which can be employed as economic policies to push the system towards a target [16]. It is noteworthy that the existence of chaos in the macroeconomic time series has not been conclusively confirmed so far. This issue belongs to the unsolved scientific problems. In [17], Barnett and He have thoroughly explained this matter in their study.

Though studies have made impressive progress in economic systems, the investigation of macroeconomic systems and control methods for these systems have room for improvement [18,19]. For instance, in the literature, a few studies exist that have investigated the control of macroeconomic systems [20]. In this study, a fuzzy controller has been designed for the macroeconomic system. Nevertheless, this control scheme has some disadvantages. For instance, the fuzzy control may not give assurance that the system reaches the desired value in the presence of uncertainties. Actually, controllers that have been proposed for macroeconomic systems are neither intelligent nor robust enough for many uncertain situations. Due to the system uncertainty, complexity, and nonlinearity in economic systems, mathematical models may not be able to consider all the variations in the system [21]. Therefore, further studies on robust and intelligent control techniques are required to achieve appropriate performance for the stabilization and control of macroeconomic systems.

These issues motivated this study. In the current paper, a macroeconomic system will be considered in the presence of dynamic uncertainties, and we aim to enhance the performance of the system by taking advantage of artificial intelligence techniques. Since applying optimal polices for economic systems is crucial, we propose to formulate the control of the macroeconomic system as an optimization problem and find optimal actions based on a reinforcement learning algorithm. The significant advantage of the proposed reinforcement learning-based method is its optimal performance. In addition, using the proposed method, we can control the system’s behavior with only restricted and specific actions. In the current study, we use Goodwin’s non-linear model and generate data from it. Actually, for real-world application, we need a model to train the Q-learning algorithm and extract the control laws. After that, the proposed controller can be applied to the real-world economic system. This is a theoretical study that aims to show the application of the Q-learning algorithm in economic systems. Hence, we use a well-known mathematical model in this study.

2. Literature Review

Up to now, various control techniques, such as predictive control [22,23,24], sliding mode control [25,26,27,28,29,30,31,32], fuzzy control [33,34,35], adaptive control [36,37,38,39], and so on, have been proposed for nonlinear systems. Among the stated control methods for various systems, the methods that can act intelligently in unpredictable conditions have gained significant attention because of their unique properties [40,41]. Artificial intelligence does combine a wide variety of state-of-the-art technologies to give systems an ability to adaptively make decisions in new and unknown situations [42,43]. In some applications, due to the high value of their tasks, and their considerable risks, implementing a robust and reliable control scheme is a significant concern. In the presence of high disturbances and uncertainties, classical control methods may fail [44]. Moreover, today, technologies have reached a point where machine intelligence can be utilized for many systems. Intelligent controllers can make systems smarter and able to find more effective ways, hopefully for better actions in real-world applications [45,46].

Reinforcement learning is one of the earliest learning techniques that emerged in the field of machine learning, and soon became a popular method [47,48]. In [49,50], the authors have used reinforcement learning methods to explore the system’s response topossible actions, and in this way, they found optimal actions by calculating how the previous actions move the system to a favorable state. After the learning process, the controller uses the optimal learned policies. It is shown in [51,52] that reinforcement learning can be utilized for the control of complex and unknown systems, as it does not require a precise mathematical model of systems. Hence, as is stated in [53,54,55], reinforcement learning-based control methods have been used in various fields of study, including the speed control of turbines, drug dosing, image evaluation and robotics.

3. Methodology

In the current study, we propose a reinforcement learning-based method for optimal control of the macroeconomic system. Traditionally, designing optimal control for nonlinear systems is challenging [56]. When the model of a system is known, using the algebraic Riccati equation, we can design the optimal control law for linear systems. However, when designing an optimal control law for nonlinear systems, the Hamilton–Jacobi–Bellman equation must be solved [57].

Watkin’s Q-learning is a reinforcement learning-based method that has attracted significant attention in recent years. This method does not need a priori information about systems and can effectively be employed online when the model of the system changes during the learning process [58]. Figure 1 illustrates the structure of reinforcement learning. In this process, agents or control signals apply an action on the system and observe the corresponding reward to find effective control policies or action plans.

Reinforcement learning starts by choosing an initial arbitrary policy and then, by interacting with the system, it learns the optimal policy. In general, a policy can be a rule base, such as “if in this state, then do this” [59]. Reinforcement learning progresses iteratively by interacting with the system. Hence, the agent’s decision sets an optimal control policy based on the rewards it obtains [60,61]. In the Q-learning algorithm, each batch of information, including action, reward and state, is utilized to update the Q table. In the Q table, the entry $Q_k\left(s_k, a_k\right)$ denotes the desirability of actions in the finite sequence ${\left(A_j \right)}_{j\in J^{+}}$ with respect to states of the finite sequence ${\left(S_i\right)}_{i\in I^{+}}.$ As is demonstrated in Figure 1, the central part of the reinforcement learning includes a system and an agent [62].

At time step $k$, firstly, the agent observes the current state $s_k$; after that it selects action $a_k$ from the sequence of actions (A). The results of the selected action $a_k$ are scored based on an appropriate reward ($r_{k+1}$ $\in R$). Based on the value of the reward, the agent realizes whether the last action that had been selected was “bad” or “good.” The agent uses the Q-learning algorithm to obtain an optimal policy that maximizes the expected value $\mathbb{E}$ [·] of the discounted reward, which is given by:

$$J\left(r_k\right)=\mathbb{E}\left[{\displaystyle \sum }_{k=1}^{\infty }{\theta }^{k-1}{r}_k\right]$$

(1)

In Equation (1), the importance of future and immediate rewards is represented by $\theta \in \left[0, 1\right],$ where for $\theta =0,$ the agent only considers the current reward, and for θ approaching 1, the agent takes into account current and future rewards. In this regard, when the agent calculates action $a_k$ and reward $r_{k+1}$, with respect to the state transition $s_k \to s_{k+1}$, the Q table will be updated based on the Q-learning algorithm, which is given by the following equation:

$$Q_k\left(s_k, a_k\right)=Q_{k-1}\left(s_k, a_k\right)+{\eta }_k\left(s_k, a_k\right)\times \left[r_{k+1}+\theta \underset{a_{k+1}}{\max }{Q}_{k-1}\left(s_{k+1}, a_{k+1}\right)-Q_{k-1}\left(s_k, a_k\right)\right]$$

(2)

where the learning rate that impacts the size of the correction after each iteration is represented by ${\eta }_k\left(s_k, a_k\right)$ $\in \left[0, 1\right), k=1, 2,\dots$. It is noteworthy that the Q-learning algorithm begins with an initial $Q_1\left(s_1, a_1\right).$ Then, the Q table will be updated based on the observations.

To assign the minimum threshold for convergence, it is common to use a tolerance parameter δ with condition $Q_k \left|Q_k-Q_k-1\right|\le \delta .$ Several sources, such as references, can be consulted for more details on the conditions required and the proof of convergence of the Q-learning algorithm [63,64]. The algorithm used in this paper has been delineated in Algorithm 1, in which episode is defined as the process of reaching the final state from the initial state.

Algorithm 1. Q-learning algorithm.

1: Initialize Q-table. 2: Loop {for all of episodes}. 3: Initialize state s. 4: Repeat {for each step-in episode}. 5: Calculate firing strength of $A \mathrm{at} \mathrm{state} s_i.$ 6: Choose action $(a_i$$) \mathrm{for} \mathrm{each} \mathrm{of} \mathrm{the} \mathrm{rules} \mathrm{at} \mathrm{state} s_i.$ 7: Take action $a_{i+1}$$, \mathrm{observe} \mathrm{reward} r_i$. 8: Calculate state value of state $s_{i+1}$. 9: Update Q-table. 10: Until $s$ is terminal state. 11: End loop.

Figure 2 demonstrates the structure of the training sequence that has been used to obtain the optimal Q table for the macroeconomic system. In the current study, we consider the state $s_k$ of the macroeconomic model in terms of the available output of tracking error. In the macroeconomic system, the target is to obtain the optimal sequence of actions, which results in a minimum value of error.

As was previously mentioned, the reward function is defined to guide the agent in recognizing whether the last action was appropriate or not. Actually, this information is utilized to reinforce the decision making of the agent. At every time step, the controller will choose the action $a_k$ as follows:

$$a_k={\left(A_j\right)}_{j\in {\mathbb{J}}^{+}}\hspace{1em}j=\arg \max \left(Q_k\left(s_k,.\right)\right)$$

(3)

The function that is utilized for calculating the reward of the agent for the transition from state $s_k$ to state $s_{k+1}$ is as follows:

$$r_{k+1}=\{\begin{array}{l}\left|\frac{e\left(kT\right)-e\left(\left(k+1\right)T\right)}{e\left(kT\right)}\right|, \left| e\left(\left(k+1\right)T\right)\right|<\left|e\left(kT\right)\right|\\ -\xi \left| e\left(\left(k+1\right)T\right)\right|<\left|e\left(kT\right)\right|\end{array}$$

(4)

where $\xi \ge 0$ and $e\left(t\right)$ represent the error of the system. The more the agent explores the system, the more it learns. When $k \to \infty$, the algorithm is able to converge to the optimal Q table. In addition, in most cases, for a finite value of $k$, systems converge to their optimal solution with an acceptable tolerance $\delta$ [51].

4. Macroeconomic System

Puu [65] proposed a modification of the Goodwin model by allowing the accelerator function to be non-monotonic. He has supported the idea that when the income sharply drops, the investment might rise. Moreover, when the income is rising, especially sharply, the investment might fall. This model is described as follows [65]:

$$Z\left(t\right)=C\left(t\right)+I\left(t\right)+G\left(t\right),$$

(5)

$$C\left(t\right)=cY\left(t\right)-u\left(t\right),$$

(6)

$$\frac{dI\left(t\right)}{dt}=-\beta \left(I\left(t\right)-B\left(t\right)\right),$$

(7)

$$\frac{dY\left(t\right)}{dt}=-\alpha \left(Y\left(t\right)-Z\left(t\right)\right).$$

(8)

where the aggregate demand $\left(Z\right)$ consists of consumption $\left(C\right)$, autonomous expenditures of the government ($G)$ and investment (I). Consumption $\left(C\right)$ depends on income ($Y)$ and is disturbed by a spontaneous change in the $u$ coefficient, in which $u$ is defined by a step function. In addition, parameter $c$ denotes the marginal propensity to consume. Equation (7) represents the rate of investment, where $\beta$ is the speed of the response of the investment to changes in production, and the speed of the response of production to changes in demand is represented by $\alpha .$ In addition, $B\left(t\right)=\mathsf{\Phi }\left(v\frac{dY\left(t\right)}{dt}\right)$. In this nonlinear macroeconomic model, the rate of investment ($\frac{dI\left(t\right)}{dt}$) depends on investment $I$ and the amount of the investment decision $B$. Moreover, as is described by this model, the decision to invest $B$ depends nonlinearly, by $\mathsf{\Phi },$ on the rate of change in production ($\frac{dY\left(t\right)}{dt}$). The nonlinear accelerator $\mathsf{\Phi }$ is given by:

$$\mathsf{\Phi }\left(v\frac{dY\left(t\right)}{dt}\right)=M\left(\frac{L+M}{L{e}^{v\frac{dY\left(t\right)}{dt}}+M}-1\right)$$

(9)

where $M$ indicates the scrapping rate of capital equipment and $L$ is the net capacity of the capital goods trades. Substituting Equation (9) into Equation (7) results in the final shape of the governing equation of the Goodwin’s model, which is given by:

$$\frac{dI\left(t\right)}{dt}=-\beta \left(I\left(t\right)-M\left(\frac{L+M}{L{e}^{v\frac{dY\left(t\right)}{dt}}+M}-1\right)\right),$$

(10)

5. Results

5.1. Dynamical investigation

The time history of the macroeconomic system is shown in Figure 3. It illustrates the time history of the system with different initial conditions. It is noteworthy that since we did not examine the absolute values of the system’s variables, and only their rates of change have been examined, they can be negative and positive. For this simulation, $c=0.25, \beta =3,v=3, \alpha =4, M=-1$ and $L=2$. Additionally, for case 1 and case 2, the initial conditions are considered to be [−2, 0, 1, −1] and [−8, 1, 1$\frac{1}{4}$, 6$\frac{3}{4}$], respectively. As is depicted in Figure 3, the behavior of the system depends on the initial condition [66].

5.2. Control Results

The proposed controller learns the optimal strategies based on the response of the system to different actions. For control of the system, the consumption of Equation (6) in the macroeconomic system, including the disturbance and control input, is given by:

$$C\left(t\right)=cY\left(t\right)-u\left(t\right)+U_c+D$$

(11)

where $U_c$ is the control action and $D$ represents disturbances that are imposed on the system. The time response of the macroeconomic system in the presence of uncertainties is simulated to test the effectiveness of the proposed control scheme. In the simulations, the number of iterations for the learning algorithm is set on 500,000 scenarios. In each scenario, the macroeconomic system is simulated (see Figure 2), and the scenarios indicate the series of transitions from arbitrary initial states to the terminal state $s_k$. In addition, ${\left(A_j\right)}_{j\in {\mathbb{J}}^{+}}$ denotes the action $a_k,$ where ${\mathbb{J}}^{+}$ $=\left\{1, 2, . . . , 24\right\}$. Actually, we consider 24 possible actions for the control of the system, and the learning algorithm learns how and when to apply each control action. 

Table 1. The norms of the regulation errors and control input.

$\mathbf{State} \left({\mathit{s}}_{\mathit{k}}\right)$	$\mathit{e}\left(\mathit{t}\right)$	$\mathbf{State} \left({\mathit{s}}_{\mathit{k}}\right)$	$\mathit{e}\left(\mathit{t}\right)$
1	$\left[-\infty ,-15\right]$	13	$\left[-0.15,-0.1\right]$
2	$\left[-15,-9\right]$	14	$\left[0.1,0.15\right]$
3	$\left[-9,-8\right]$	15	$\left[0.15,0.2\right]$
4	$\left[-8,-6.5\right]$	16	$\left[0.2,0.5\right]$
5	$\left[-6.5,-5\right]$	17	$\left[0.5,1\right]$
6	$\left[-5,-4\right]$	18	$\left[1,1.5\right]$
7	$\left[-4,-3\right]$	19	$\left[1.5,3\right]$
8	$\left[-3,-1.5\right]$	20	$\left[3,4\right]$
9	$\left[-1.5,-1\right]$	21	$\left[4,5\right]$
10	$\left[-1,-0.5\right]$	22	$\left[5,8\right]$
11	$\left[-0.5,-0.2\right]$	23	$\left[8,15\right]$
12	$\left[-0.2,-0.15\right]$	24	$\left[15,\infty \right]$

lists the norms of regulation errors and control inputs.

These intervals are user-defined and could be changed. In the current study, we have classified these intervals into 24 levels, and the controller will try to find an appropriate control action for each of them. After convergence of the $Q$ table, the agent will select an action $a_k={\left(A_j\right)}_{j\in {\mathbb{J}}^{+}}$, where $j=\arg \max Q_k\left(s_k, ·\right).$

Furthermore, the learning rates that influence the value of the correction after each iteration are considered as follows:

$${\eta }_k\left(s_k, a_k\right)=\zeta \frac{E{p}_i}{E{p}_f }$$

(12)

where $\zeta =0.5$ and $E{p}_i$ denote the number of the current episode, and $E{p}_f$ is the maximum value of the episode that we want the simulation to run. To control the system, two cases have been considered. Namely, in the first simulation, we consider the maximum action (maximum value for the controller) to be 40, and the minimum is equal to –40. In the second simulation, the maximum and minimum values of actions are considered to be 30 and –30, respectively. Note that two different reinforcement learning agents have been trained for each of the aforementioned cases.

For both cases, the desired value of income is considered to be two $\left(Y_d=2\right),$ and the aim is to reduce the absolute value of error $(\left|e\left(t\right)\right|=\left|Y-Y_d\right|$) until it becomes equal to zero. The reward is calculated based on Equation (4), in which $\xi =-0.1$. The discount factor $\theta =0.7$. In order to reduce the convergence time, we set the control policy as: when $\left|e\left(t\right)\right|<0.1$ $U_c=K e\left(t\right)$ and parameter $K$ is equal to ten, and other actions are considered as:

$$\begin{array}{l}A=U_{max}[0, 0.001, 0.002,-0.002, 0.005,-0.005, 0.1\\ ,-0.1, 0.2,-0.2, 0.3,-0.3, 0.4,-0.4, 0.6,-0.6, 0.7,\\ -0.7,-0.8, 0.8,-0.9, 0.9,-1, 1].\end{array}$$

(13)

At first, we consider $U_{max}=40.$ The stabilized states of the macrosystem are depicted in Figure 4. As is shown in this figure, the states of the system are stabilized in finite time using the proposed controller.

Additionally, Figure 5 depicts the control input based on the applied optimal reinforcement learning-based controller. Based on the numerical simulation, using the proposed controller, after a short period of time, the system is completely stabilized, and the states of the system reach their desired value.

Here, we consider $U_{max}=30$. Actually, to investigate the proposed control in another condition, we confine the controller with actions that have fewer values. Figure 6 demonstrates the time history of the macroeconomic system by applying the designed control when $U_{max}=30$. In addition, the control input is displayed in Figure 7. The simulation results evidently corroborate that the proposed controller satisfies the expected performances. Moreover, the numerical simulations clearly corroborate that the proposed controller effectively finds optimal actions for the control of the nonlinear macroeconomic system, even when we confined the control actions. It is noteworthy that the results of the examined model show that it is possible to control and stabilize the variables at constant values, e.g., Y (national income) = 2 and I (investments) = 0. This means simple reproduction, which is normal when investments are zero.

In [20], a fuzzy controller has been presented for macroeconomic models. Although a fuzzy controller is a good candidate for economic systems, in that study, only some simple fuzzy rules are considered, and the controller is not robust against uncertainties and disturbances. In addition, that controller is only suitable for continuous systems, which is not the case in most studies on macroeconomic models.

Remark 1.

The numerical results show that the proposed controller can be applied to nonlinear economic systems. Since the controller can be used for tracking control and stabilization, from an economic point of view, this method is useful to remove business cycles, even when these systems are not chaotic and have deterministic results.

Remark 2.

The nonlinear Goodwin model and other models that are similar to it are the most plausible in science, but they have serious limitations that the researcher must be aware of. There are differences between the data generated by Goodwin’s simple nonlinear model and actual macroeconomic data. The former is purely deterministic, while the latter is burdened with significant noise and may be complex. In the current study, we only use this model to investigate the developed control approach. Consequently, applying the developed method to a real-world economic system will provide better insight into the control of the macroeconomic model, and it is the next step for this study. Hence, we will consider it as a future direction for our study.

6. Conclusions

The control and stabilization of macroeconomic systems using the reinforcement learning-based controller were studied. The proposed control scheme uses a reinforcement learning algorithm to find optimal control actions. Based on the Q-learning algorithm, the best optimal action for the system was obtained. The proposed method can control the system using restricted actuation, and, this way, it provides highly effective performance and optimal strategies using possible actions. Through numerical simulation, the adequate performance of the proposed reinforcement learning-based controllers was demonstrated by using different numerical simulations. The possibility of controlling the behavior of the macroeconomic system using the proposed reinforcement learning-based controller suggests the extension of the application of the reinforcement learning algorithms to the control of other nonlinear economic systems. Future studies will also be devoted to a complete comparison with other learning-based strategies, as well as alternative algorithms from the Q-learning method.