Spiking Neural-Networks-Based Data-Driven Control

Abstract

Machine learning can be effectively applied in control loops to make optimal control decisions robustly. There is increasing interest in using spiking neural networks (SNNs) as the apparatus for machine learning in control engineering because SNNs can potentially offer high energy efficiency, and new SNN-enabling neuromorphic hardware is being rapidly developed. A defining characteristic of control problems is that environmental reactions and delayed rewards must be considered. Although reinforcement learning (RL) provides the fundamental mechanisms to address such problems, implementing these mechanisms in SNN learning has been underexplored. Previously, spike-timing-dependent plasticity learning schemes (STDP) modulated by factors of temporal difference (TD-STDP) or reward (R-STDP) have been proposed for RL with SNN. Here, we designed and implemented an SNN controller to explore and compare these two schemes by considering cart-pole balancing as a representative example. Although the TD-based learning rules are very general, the resulting model exhibits rather slow convergence, producing noisy and imperfect results even after prolonged training. We show that by integrating the understanding of the dynamics of the environment into the reward function of R-STDP, a robust SNN-based controller can be learned much more efficiently than TD-STDP.

1. Introduction

A traditional control loop can be conceptualized into several essential blocks [1], including the internal and external observation blocks, the control block, and the (optional) dynamic model block. The internal and external observation blocks describe the (past and) current states of the system and the environment. These states are used by the control block to determine which future states the system should reach to achieve certain control goals and what actions should be taken to bring the system into these future states. In model-based control [2], the decision making of the control block is assisted by the dynamic model block that predicts the system’s behavior (e.g., through simulations of the physical system). In contras, in model-free control [3], no dynamic model of the system is required.

For many control systems, blocks that perform one or more of the above functions can be advantageously implemented with machine learning (ML). This field has experienced rapid developments in recent decades and is having substantial impact in various disciplines of science and engineering [4]. In control engineering, ML approaches have been extensively investigated for tasks, such as industrial process control [5], autonomous vehicle maneuvering [6], and robotics operation [7].

ML can be applied to preprocess complex sensory data before these data are used to make control decisions [8]. Moreover, supervised learning (one type of ML) [9,10] can be applied to produce a dynamic model [11], which can be used in place of a physics-based model if the physics model is overly simplified, difficult to derive, or too time-consuming to simulate. At the control loop’s core, ML can achieve robust nonlinear mapping from high-dimensional state variables to optimal control decisions. Reinforcement learning (RL) [12], a subfield of machine learning, has been widely recognized as targeting tasks that generally fall within the scope of control engineering [13].

Most commonly, the ML approaches used for control tasks have been based on artificial neural networks (ANNs), especially deep ANNs (i.e., ANNs with many hidden layers) [14,15]. To a large extent, the versatility and effectiveness of ANNs can be attributed to their hierarchical organization of relatively simple computational units into dedicated network structures, which gained inspiration from intricately connected neuron networks in biological brains [16]. However, despite such a structural similarity between ANNs and biological neural networks, the computations performed by the nodes of ANNs (the artificial “neurons”) are fundamentally different from the “computations” performed by the biological neurons. While the nodes in ANNs usually use numerical values as inputs and outputs, biological neurons use time-dependent signal series (i.e., spike trains) as inputs and outputs [17].

A typical neuron cell has many dendrites and an axon connected to neurons upstream and downstream through microstructures called synapses [18]. Interconnected neurons use spikes to communicate with each other [17]. Over a while, a series of spikes (a spike train) generated by a pre-synaptic neuron can modulate the voltage level of a postsynaptic neuron. When the latter voltage is above a threshold value, the postsynaptic neuron generates its spike (that is, it is firing), and its voltage level drops. In a spiking neuron network (SNN), each neuron will receive spike trains from the upstream neurons connecting to its input synapses, produce its spike train, and pass its spike train to the downstream neurons connected with its output synapses [19].

Vreeken et al. proposed that computational neural networks can be divided into three generations [20]: the first and second generations use artificial neurons that accept real-valued inputs and generate outputs of binary (first generation) or continuous (second generation) values, while the third generation uses neurons that consider time-dependent signal series (i.e., spike trains) as input and output, mimicking the signaling between actual neuron cells in biological brains [17].

One of the most attractive characteristics of SNNs is that they can potentially offer much higher energy efficiency than conventional ANNs [21]. The neural activity in an SNN is event-driven because a neuron is only active when it receives or fires a spike, while it can be idle when there is no event. This is different from an artificial neuron in an ANN, which must always be active. Energy-efficient SNN computations are actively pursued through the development of SNN-based ML algorithms [22] and neuromorphic hardware that allows on-chip SNNs [23,24].

This paper investigates the use of SNN instead of ANN in control problems. In particular, we are interested in designing SNNs to carry out the task of making optimal control decisions. Since the most widely used ML approach for such tasks is reinforcement learning (RL), in what follows, we will first present a brief introduction to the basic techniques of RL, followed by an overview of existing learning methods for SNN.

Using the classical cart-pole game as an example, we constructed a simple yet effective SNN to solve this control problem with delayed rewards. We examine the performance of two learning methods adapted from spike-timing-dependent plasticity (STDP). The learning method that introduces the temporal difference signal into the training process presents a general framework for tackling various control problems. The other method that employs the reward signal in the training process exploits our prior knowledge of a specific control problem and therefore trades its generality for the efficiency of solving that specific control problem.

2. Preliminary

2.1. A Brief Introduction to Reinforcement Learning (RL)

In a typical RL scheme, an agent (a learner is often noted as an agent in RL) is in one of a range of possible states, noted as $s\in S$, where S is the state space. In each state, the agent can take an action a from a predefined set A, which denotes the action space. The agent’s action can potentially change the agent’s current state. At each state, the agent may receive a certain amount of reward. The agent’s goal is to choose a proper action at each given state so that the total reward the agent receives from the states it goes through will eventually be maximized.

The rule by which the agent chooses its action can be noted as the policy $\pi$. For example, $\pi$ can correspond to the probability that the agent will take action a when in state s, that is, $\pi (s,a)=P\left(a\right|s)$ for $s\in S$ and $a\in A$. The goal of RL is to find the optimal policy for every state.

Assuming that the environment evolves in consecutive steps during learning, at a certain step n, the agent detects that the environment is at a state $s_n$ and chooses an action $a_n$ (according to its current policy). We define the Q value function as the averaged future total reward the agent can receive by following the policy $\pi$ at all future states, namely,

$$Q_{\pi }(s_n,a_n)=E_{\pi }\left(G\right|s=s_n,a=a_n),$$

(1)

where G represents the total future rewards or gains, and $E_{\pi }$ represents averaging over all possible evolutionary trajectories of future state sequences given the policy $\pi$.

Now, we consider the Q value function that corresponds to the optimal policy ${\pi }_{opt}$ (i.e., the policy that maximizes the Q values) and note this function as $Q_{opt}$. Because the total future reward of step n is the sum of the reward for the state at step $n+1$ and the total future reward of step $n+1$, and also because the optimal policy at step $n+1$ would be to take action with the maximum future gain after step $n+1$, $Q_{opt}$ should satisfy the following Bellman optimal equations, which Bellman proposed as the basis for solving the RL problem via a dynamic programming algorithm [25],

$$Q_{opt}(s,a)=R\left(s\right)+{\Sigma }_{s^{\prime }\in S}{P}_{s{s}^{\prime }}^{a}ma{x}_{a^{\prime }\in A}{Q}_{opt}(s^{\prime },a^{\prime }),$$

(2)

where $P_{s{s}^{\prime }}^a$ refers to the transition probability $P(s_{n+1}=s^{\prime }|s_n=s,a_n=a)$.

To solve the Bellman equations, one can start with some initial Q value functions and update iteratively. In this process, we can define the deviations of the current Q value function from the Bellman equations as temporal differences, that is,

$$\begin{array}{c}\hfill T{D}_Q=R\left(s\right)+{\Sigma }_{s^{\prime }\in S}{P}_{s{s}^{\prime }}^{a}ma{x}_{a^{\prime }\in A}Q(s^{\prime },a^{\prime })-Q(s,a)\end{array}$$

(3)

The optimal Q value function can be obtained by iteratively using the temporal differences to update the Q value function,

$$Q_{new}(s,a)=Q_{old}(s,a)+\gamma T{D}_Q,$$

(4)

in which $\gamma$ is the learning rate.

The RL algorithm that derives the optimal policy by learning the Q value function is known as Q-Learning. When the state space and the action space are discrete, the Q value function is a table (i.e., Q-table) of Q values for all state–action pairs. Guided by the Q-table, the agent can choose the appropriate action based on its state. Its inventor Watkins proved that the Q learning algorithm based on Equation (4) can converge if the Q values are represented in a lookup table [26].

2.2. An Overview of Learning Methods for SNN

While the temporal dynamics of spiking neurons can be well-simulated by models such as the Leaky Integrate and Fire (LIF) Model [27,28,29], it is not immediately clear how to train networks composed of such neurons for diverse ML tasks [30], including RL. Unlike ANNs that use differentiable real-valued signals and can thus be trained with the well-known backpropagation algorithm [31], SNNs use discrete and nondifferentiable spike train signals, making training based on straightforward backpropagation unfeasible.

It is an active field of research to develop appropriate algorithms to train SNNs for different applications [19,32]. One way to train an SNN is first to train an ANN with the desired functionality and then convert the ANN into an SNN. This is also known as shadow training [19]. The rules for the conversion or replacement of ANN neurons by spiking neurons are to match the activation functions of the ANN nodes with the firing rates of the spiking neurons [33] or the timing of the spikes emitted by the spiking neurons [34]. Although the conversion approach has been shown to produce SNNs with an accuracy comparable to deep ANNs for certain image recognition tasks, the approach has several drawbacks. Training a complex ANN is time-consuming and converting an ANN to an SNN requires a rather long simulation time. Moreover, the approach usually leads to large and deep SNNs, with performances worse than the source ANNs since the conversion processes lead to no exact replications of but only approximations to the original ANNs [32].

Another type of SNN training algorithm enables backpropagation or gradient descent training by using differentiable approximations of the spike function for derivative estimation [35,36]. The Back Propagation Through Time or BPTT method [35] converts the neural network to an equivalent sequence of computations and applies the backpropagation framework to it. The SpikeProp method uses the fact that time is continuous and calculates the derivative of the spike timing concerning the weight [36]. Backpropagation-based training has produced SNNs that can solve small-scale image recognition problems well.

The above learning algorithms for SNN echo the learning methods for ANN. They do not correspond to how learning occurs in a biological brain. Neuroscience studies have long revealed synaptic plasticity, i.e., changes in the connection strengths of synapses, as a fundamental mechanism underlying biological learning. A highly interesting SNN learning algorithm uses a biological learning mechanism called spike-timing-dependent plasticity (STDP) of synapses [37].

STDP is a form of Hebbian plasticity that follows the rule first postulated by Hebb as “cells that fire together wire together” [38]. In other words, it is based on the correlated firing activity of presynaptic and postsynaptic neurons to decide whether to change the strength of connectivity of a synapse. The core idea behind these learning rules is to build synaptic connections to represent causal relationships. As exhibited by its name, STDP has precisely defined relationships between spike timing and changes in connection strengths [39,40]. Biologically, STDP is supported by the experimental finding that the co-activation of the pre-and post-synaptic neurons can lead to two types of plasticity or synaptic changes: long-term potentiation (LTP) and long-term depression (LTD) [41]. LTP refers to strengthening the synaptic connection when the pre-synaptic neuron fires within a short period before the firing of the post-synaptic neuron. In contrast, LTD refers to weakening the connection when firing events occur in reverse order. Equation (5) presents a simple mathematical model for STDP with LTP and LTD,

$$\Delta w=\left\{\begin{array}{cc}{A}_{+}exp(-\frac{s}{{\tau }_{+}}),\hfill & s>0\hfill \\ A_{-}exp\left(\frac{s}{{\tau }_{-}}\right),\hfill & s<=0,\hfill \end{array}\right.$$

(5)

where $\Delta w$ is the change in the strength of connection or synaptic weight, s is the time difference between postsynaptic and presynaptic spikes, and $A_{+}$ and $A_{-}$ are, respectively, the coefficients for potentiation (weight increase) and depression (weight decrease). The time windows for LTP and LTD are given by ${\tau }_{+}$ and ${\tau }_{-}$, respectively [42].

The STDP learning mechanism provides a possible basis for training SNNs to make control decisions in a control loop. This can be achieved by introducing an additional modulating factor for the SNN synaptic plasticity process [43,44]. The modulating factor can be determined from the environment’s reactions to the control decisions made according to the output of the SNN so that the finally trained SNN can produce control decisions that lead to optimal outcomes. Neuroscience studies have shown that in biological brains, dopamine is one of the most important neuromodulators in such feedback- or reward-based learning processes [45]. That is, the dopamine level represents the amount of rewards that guide the learning process by modulating STDP [46]. This biological phenomenon can be simulated by extending the STDP model in formula 5 to the following modulated STDP formula,

$$\Delta w=M\ast w_{eligibility},$$

(6)

where M represents the effects of neuromodulators, while $w_{eligibility}$ selects synapses for reward-modulated plasticity according to conventional STDP rules [43].

Several previous studies have investigated the application of the above idea of modulated STDP learning rules to various ML problems, including the making of control decisions [44,47]. Florian et al. introduced the reward-modulated STDP, or R-STDP algorithm [44], in which the modulating factor came from the deviations of the actual and expected SNN output. In that study, the algorithm’s effectiveness was illustrated with the example of using SNN to solve the XOR problem and to output predefined spike trains. We note that these examples were essentially regression problems. The so-called “rewards” were derived from the regression errors, but not from a reaction of an environment impacted by a control decision (determined from the outcome of the SNN). In other words, the rewards for these problems were instantaneous but not delayed, as in typical control problems.

Frémaux et al. proposed using temporal difference (TD) in reinforcement learning (RL) as the modulating factor for STDP [47]. This TD-STDP learning rule was applied to train actor-critic-type reinforcement learning (RL) controllers [12,13]. In such a controller, an SNN subnetwork serving as the critic provides estimations of the “values” of the environmental states. During training, the temporal change of the values estimated by the critic was used to derive the modulating factor for STDP. The analysis in [47] illustrated the potential of using SNN to implement general RL strategies for control tasks.

Despite these previous studies, STDP-based SNN learning for control tasks remains an underexplored research problem. As mentioned above, the examples did not include typical control tasks with delayed rewards in the original study that examined R-STDP [44]. Moreover, there is a lack of studies that compare the relative advantages and drawbacks of different schemes, such as R-STDP and TD-STDP, for the same specific control task. In this paper, we explore SNN with STDP learning modulated with the reward factor for control tasks by considering the cart-pole balancing problem [48] as a representative example. We have designed and implemented a novel SNN controller trained by the TD-STDP rules similar to those proposed in ref. [47]. Based on the RL Q learning strategy [26], the SNN presented here has a much simpler general network architecture than the actor–critic architecture of ref. [47]. Although the TD-STDP learning rule is very general and does not require any specific assumption about the dynamics of the environment (i.e., the target to be controlled), it leads to rather slow convergence, producing somewhat imperfect controllers even after prolonged training. We have also examined R-STDP learning. We show that by integrating the understanding of the specific dynamics of the environment into the design of the reward function, a more robust SNN-based controller can be learned much more efficiently by R-STDP than by TD-STDP.

2.3. The Cart-Pole Environment

The cart-pole environment is one of the classic control problems that has been widely applied for the performance evaluation of control algorithms. Several studies on SNN have also chosen the cart-pole control problem as their test bench [49,50,51]. This problem could be stated as the following: given a cart with a pole attached to it through a fixed joint (i.e., the pole can only rotate in a plane perpendicular to the horizontal surface), the cart can move on a flat surface with no friction. At each step of the game, the control agent should choose one of two actions, pull right or pull left. The action will affect the state of the cart pole. The controller’s goal is to keep the pole in approximately upright positions without falling for as long as possible. The values that the agent can observe are:

• The position of the cart: x,
• The velocity of the cart: $v=\frac{dx}{dt}$,
• The pole angle: $\theta$,
• The angular velocity of the pole: $\omega =\frac{d\theta }{dt}$.

Here, right is defined as the positive direction for these values.

In this paper, the cart-pole model environment implemented in the gym library [52] is applied. This environment terminates its simulation if one of these three situations occurs:

• The pole drops when the absolute value of the angle of the pole is greater than ${12}^{\circ }$.
• The cart slides out of the edge of the animation display (i.e., the cart position is out of the range between −2.4 to 2.4).
• The simulation steps are larger than 200 (or other customized values).

By definition, the first two termination conditions correspond to the reach of some failure states of the environment, while the third termination condition indicates the successful completion of the task.

3. The TD-STDP SNN

3.1. The Overall Workflow of the Program

The overall framework of our program that uses TD-STDP learning to solve the cart-pole control problem is presented in Figure 1. The entire program can be divided into two main parts; one is the cart-pole environment and the other is the SNN-based controller. Each training epoch comprises a consecutive simulation of the cart-pole steps from the start until one of the finishing conditions is met. At each cart-pole step, the environment simulator and the SNN simulator are run sequentially. After a cart-pole simulation step, the SNN simulator takes the current state of the cart pole as its input (see below), runs for a fixed period, and produces its output, based on which an action to be applied to the cart-pole environment will be chosen. After taking action, the cart-pole environment evolves to the next step. The whole process is iterated until the simulation of the environment ends. Then, the cart-pole environment is reset and started from a new initial state to perform the next training epoch.

3.2. The Architecture of the SNN

The overall structure of the SNN is illustrated in Figure 2. There are only two layers of neurons. The first is the input layer, which comprises some input neurons. The second is the output layer, which comprises two groups of neurons, each corresponding to one possible choice of action. Every input neuron is connected to every output neuron through a synapse.

3.3. The Input Neurons

The spikes generated by the input neurons code the observed state of the cart pole. First, an observed variable of the cart pole is mapped to an integer index using the following formulae.

$$i{d}_{obs}=\left\{\begin{array}{cc}0,\hfill & obs\le ob{s}_{min}\hfill \\ floor\left(\frac{x-x_{min}}{\Delta x}\right),\hfill & ob{s}_{min}<obs<ob{s}_{max}\hfill \\ N_{states,obs}-1,\hfill & obs\ge ob{s}_{max}\hfill \end{array}\right.$$

(7)

$$N_{states,obs}=ceil\left(\frac{x_{max}-x_{min}}{\Delta x}\right),$$

(8)

where the variable $obs$ corresponds to one of the observed variables x, v, $\theta$, or $\omega$; $[ob{s}_{min},ob{s}_{max}]$ defines the region of values for evenly divided bins; $\Delta x$ is the width of the bins; and $N_{states,obs}$ is the total number of bins or discrete states for the variable $obs$.

Any possible state of the cart pole is described by a unique combination of the four integers ($i{d}_x,i{d}_v,i{d}_{\theta },i{d}_{\omega }$). The total number of possible states is $N_{states,total}=N_{states,x}\ast N_{states,v}\ast N_{states,\theta }\ast N_{states,\omega }$.

The total number of SNN input neurons is defined to be $N_{states,total}\ast n_{input}$, so that a group of $n_{input}$ neurons represents each possible state. When a given state of the cart pole is passed to the SNN, only the $n_{input}$ neurons representing that particular state will fire spikes. Meanwhile, all other input neurons will remain inactive. This encoding technique is inspired by methods for encoding categorical variables for ML problems and is generally known as one-hot encoding [53].

3.4. The Output Neurons

An LIF model describes the dynamics of each output neuron with adaptation in Equation (9) [54].

$${\tau }_m\frac{dV}{dt}=g_e(E_e-V)+E_l-V,$$

(9)

in which V is the membrane potential, $E_l$ is the resting potential, $E_e$ is a high value voltage, ${\tau }_m$ is a time constant for the membrane potential, and $g_e$ is a dimensionless quantity representing the effects of the upstream spikes received through the input synapses. Without receiving spikes from the input synapses, the dynamics of $g_e$ is described by

$${\tau }_g\frac{d{g}_e}{dt}=-g_e,$$

(10)

in which ${\tau }_e$ is a time constant for the lasting effect of the input spikes. When an input spike from a synapse i connects to the neuron, $g_e$ changes instantly according to the number of parameters.

$$g_e=g_e+w_i,$$

(11)

in which $w_i$ is the weight of the synapse i. The effects of spikes received by different synapses are simply accumulated.

3.5. Q-Learning by SNN

The output of the SNN is interpreted as the (scaled) Q values of corresponding actions given the particular input state, namely,

$$\begin{array}{c}\hfill Q(s,a)=scale\ast n_{spikes}(s,a),\end{array}$$

(12)

in which $n_{spikes}(s,a)$ refers to the number of spikes fired by neurons in the output group associated with action a when only the input neurons that encode the state s have been firing during a period of SNN simulation.

Note that this definition of the output of the SNN as the (scaled) Q value that depends both on the state and the action is somewhat different from the SNN value network of ref. [47]. There, the output from a single group of neurons provided the values of only the states without further discrimination of different actions.

Using separated groups of output neurons for different actions, we do not need an extra actor module as in ref. [47] to evaluate the actions. Instead, the action is determined by the same groups of neurons that estimate the Q values.

To determine the TD error of the current SNN for the cart-pole step n, we run the SNN with the input state $s_n$ for a while, obtain the values of $Q(s_n,a)$, choose and carry out the action $a_n$, and obtain the next state of the cart pole $s_{n+1}$. We then run the SNN with the input state $s_{n+1}$ to obtain $Q(s_{n+1},a)$. Following Equation (3), the TD error for step n is computed as

$$T{D}_n=TD(s_n,a_n)=\gamma ma{x}_{a}Q(s_{n+1},a)+R_{n+1}-Q(s_n,a_n).$$

(13)

Here, we have included a “discount factor” $\gamma$ so that the rewards of only a finite number of future steps effectively contribute to the current Q value.

The formula above with $R_{n+1}=1$ is applied only when the state at step $n+1$ does not correspond to a failure state of the cart pole. When $s_{n+1}$ corresponds to a failure state (and the simulation of the environment will terminate), both $Q(s_{n+1},a)$ and $R_{n+1}$ should be zero (because there will be no current or future reward). Then, the $TD$ error for step n is computed as

$$T{D}_n=-Q(s_n,a_n).$$

(14)

As shown in Figure 1, the TD error is used to update the weights of the synapses connecting the input state neurons to the output neurons.

3.6. Determining Eligibility of the Synapses

Given the current state of the cart pole, the SNN is simulated for a fixed number of SNN time steps (note that these are not the cart-pole steps). The SNN simulation produces two outcomes. The first is the estimated Q values described above. The second outcome includes the eligibility traces of the synapses for updating the synapses’ weights. To determine the eligibility for each synapse (for simplicity, we will omit the index of the synapse from the following formulations) based on the relative timing of pre-and post-synaptic spikes, we first define the following pre-synaptic activity and post-synaptic activity traces,

$$A_{pre}\left(t\right)=\sum _{k\in inputspikes}\xi (t-t_k)e^{-\frac{t-t_k}{{\tau }_{pre}}},$$

(15)

$$A_{post}\left(t\right)=\sum _{k\in outputspikes}\xi (t-t_k)e^{-\frac{t-t_k}{{\tau }_{post}}},$$

(16)

in which $t_k$ represents the time of spike k, and the Heaviside step function $\xi (t-t_k)$ is applied to select spikes that were received (for $A_{pre}$) or fired (for $A_{post}$) before the time t, i.e.,

$$\xi (t-t_k)=\left\{\begin{array}{cc}1,\hfill & t-t_k\ge 0\hfill \\ 0,\hfill & otherwise.\hfill \end{array}\right.$$

(17)

$A_{pre}$ is not negligible only for a short time (compared to the time constant $t_{pre}$) after receiving a presynaptic spike. Similarly, $A_{post}$ is not negligible, only not long after the firing of a post-synaptic spike. Then, the eligibility of the synapse according to the STDP rule can be determined as

$$w_{trace}={\Delta }_{pre}\int O\left(t\right)A_{pre}\left(t\right)dt-{\Delta }_{post}\int I\left(t\right)A_{post}\left(t\right)dt,$$

(18)

where ${\Delta }_{pre}$ and ${\Delta }_{post}$ are parameters that affect learning rates. In the above equation, the function $I\left(t\right)$ represents the time series of input spikes on that synapse, while the function $O\left(t\right)$ represents the time series of the output spikes, that is,

$$I\left(t\right)=\sum _{k\in inputspikes}\delta (t-t_k),$$

(19)

$$O\left(t\right)=\sum _{k\in outputspikes}\delta (t-t_k).$$

(20)

In Equation (18), the first term corresponds to long-term potentiation (LTP) of the synapses because it makes a positive contribution to $w_{trace}$ only when an output spike is generated shortly after receiving a presynaptic spike (thus $A_{pre}$ is non-negligible); the second term corresponds to long-term depression (LTD), as it makes a negative contribution to $w_{trace}$ if an input spike is received at the synapses shortly after the post-synaptic neuron has fired (thus $A_{post}$ is not negligible).

3.7. Learning the Synaptic Weights by TD-STDP

After the determination of $T{D}_n$, the weights of the synapses that connect to the output groups corresponding to an action $a_n$ are updated using

$$w_{new}=w_{old}+\beta \ast T{D}_n\ast w_{trace,n},$$

(21)

where $\beta$ is a constant that adjusts the rate of the weight change, and $w_{trace,n}$ is the eligibility trace calculated using Equation (18) at the nth cart-pole step.

3.8. Exploration and Exploitation in Training

For the agent to accumulate experience in various cart-pole states, the agent should be encouraged to explore various possible states during the initial training phase of the SNN. This is achieved by adding a stochastic mechanism to choose the action. Specifically, for the first 100 epochs, the agent chooses an action randomly at every cart-pole step. After that, the agent randomly chooses an action with a probability of $P_{explore}$, the value of which starts from 1.0 and is scaled down at the beginning of each new epoch by a factor of $\alpha =0.99$. With a probability of $1-P_{explore}$, the agent chooses the action according to

$$P(a_n=a)\propto e^{(Q_a/{\delta }_{Q_0})}.$$

(22)

With the above definition, the agent’s action would be deterministic only when the difference between the Q values of the two actions provided by the SNN is significant (relative to ${\delta }_{Q_0}$, which is chosen to be 0.1).

4. The R-STDP SNN

4.1. The Differences between the R-STDP and the TD-STDP Programs

The overall framework of the R-STDP program shown in Figure 3 is similar to the overall framework of the TD-STDP program. The main differences between the two programs come from the different interpretations of the output of the SNN (i.e., the number of spikes fired by each of the two groups of neurons in the output layer). In the TD-STDP SNN, the output represents the scaled Q values for different actions of the corresponding input state. Thus, the relative output changes upon changes in the input state have meaning. In the R-STDP SNN, the output is (by definition) interpreted or used as simple numerical metrics for choosing the action (here, the action associated with the output group that fires the most spikes is preferred). Thus, when there are changes in the input state, the corresponding changes in the output of the R-STDP-trained SNN are not concerned. This leads to a different weight-updating scheme in the R-STDP program.

4.2. Updating Synapse Weights with Delayed Reward

After the action (chosen on the basis of the output of the SNN) has been taken, a new state of the cart pole is returned. A reward value of R is calculated from both the previous state and the new state of the cart pole (see below). The weights of the SNN synapses are updated using the reward and eligibility factors computed from the previous SNN run. When the action leads to a positive reward, the synapses connections that would increase the probability of the SNN output of the action taken should be strengthened. In contrast, the synapse connections that would increase the probability of the SNN producing the opposite action should be weakened. The opposite should apply when the action taken leads to a negative reward. This reasoning leads to the following updating rules. For every synapse connected to the output neuron associated with the action taken, the weights increase according to the action.

$$w_{new}=w_{old}+R(s_n,s_{n+1})\ast w_{trace,n},$$

(23)

while for every synapse connected to the output neuron associated with the opposite of the taken action,

$$w_{new}=w_{old}-R(s_n,s_{n+1})\ast w_{trace,n}.$$

(24)

4.3. Reward Function Designs

Intuitively, the reward function should reflect the objective of the control task, which is to keep the pole in the upward direction. As this goal should be realized by the actions that are chosen according to and carried out upon the given states of the cart pole, an action should be rewarded positively if it causes the state of the cart pole to change towards a new state that favors achieving the control goal. Based on this intuition, we designed and tested the three reward functions of the SNN controller. From Reward Function 1 to Reward Function 3, more sensory information is encoded into the reward function; therefore, the agent’s actual tendency to keep the pole upright is more accurately pictured. In the following formulations, the subscript $old$ corresponds to the cart-pole step n in Equations (23) and (24), while the subscript $new$ denotes the cart-pole step $n+1$ in the same equations.

Reward Function 1: This reward function is defined to return one when the angle of the pole is in the range where the pole does not fall, and the cart is still within the range of the animation display and otherwise returns zero. That is,

$$R_1=\left\{\begin{array}{cc}1,\hfill & Thesimulationisnotterminated\hfill \\ 0,\hfill & otherwise.\hfill \end{array}\right.$$

(25)

Reward Function 2: This reward function is defined as

$$R_2({\omega }_{old},{\omega }_{new})=\left\{\begin{array}{cc}1,\hfill & {\omega }_{old}\ast {\omega }_{new}<0or\hfill \\ & |{\omega }_{old}|>|{\omega }_{new}|\hfill \\ -1,\hfill & otherwise.\hfill \end{array}\right.$$

(26)

As shown in Figure 4a, this reward is determined by comparing the current angular velocity ${\omega }_{new}$ with the angular velocity observed in the previous cart-pole step ${\omega }_{old}$. If a significant change in angular velocity has been detected, the action has reversed the previous moving trend of the pole, and thus a reward of one is assigned to this kind of action. Suppose that there is no significant change in angular velocity. In that case, the shrinking of the absolute value of the angular velocity itself also indicates that the pole is slowing down its rotation. Therefore, the action is also encouraged with a reward of one. In other situations, the reward value of −1 is used to punish the action taken for not contributing to reverse or reduce the rotation of the pole.

Reward Function 3: This reward function is defined as

$$\begin{array}{c}\hfill R_3({\omega }_{old},{\omega }_{new},{\theta }_{old},{\theta }_{new})=\\ \hfill \left\{\begin{array}{cc}{R}_2({\omega }_{old},{\omega }_{new}),\hfill & {\theta }_{new}\ast {\omega }_{old}>0\hfill \\ 1,\hfill & {\theta }_{new}\ast {\omega }_{old}\le 0and{\theta }_{new}\ast {\omega }_{new}<0\hfill \\ -1,\hfill & otherwise.\hfill \end{array}\right.\end{array}$$

(27)

Compared to Reward Function 2, Reward Function 3 considers not only the angular velocity but also the angular position of the pole. In Reward Function 3, Reward Function 2 is only used when ${\omega }_{old}$ and ${\theta }_{new}$ are of the same sign. This is based on the following reasoning. If ${\omega }_{old}$ and ${\theta }_{new}$ are not of the same sign but of opposite signs, ${\theta }_{old}$ must be of the same sign as ${\theta }_{new}$ because

$${\theta }_{old}={\theta }_{new}-{\omega }_{old}\ast \Delta t.$$

(28)

Then, ${\theta }_{old}$ and ${\omega }_{old}$ must be of opposite signs. This means that ${\omega }_{old}$ causes a reduction in the angle of tilt of the pole. Therefore, Reward Function 2, which rewards the reversal or reduction of ${\omega }_{old}$, is no longer suitable. In this case, a comparison of the sign of ${\theta }_{new}$ with that of ${\omega }_{new}$ remains to be further considered. If the signs are opposite, the new state has retained the desired direction of ${\omega }_{new}$ to correct the tilting angle ${\theta }_{new}$. A positive value then rewards the action taken. Otherwise, the action is punished with a negative value of −1. This reward function is illustrated by Figure 4b.

4.4. Exploitation and Exploration

With the above R-STDP learning rules and the cart-pole state encoding scheme, the SNN will only be able to learn to choose a good action for states that the agent has already visited. It would not provide a good policy for the states it has not seen yet. To encourage the exploration of various possible states during the initial training phase, we define a probability $P_{explore}$ for the agent to randomly take one of the two possible actions (exploration) instead of taking action determined from the SNN’s outputs (exploitation). The value of $P_{explore}$ is set to one at the beginning of the learning. It is downscaled by a value of ${\gamma }_{explore}$ = 0.9 at the beginning of each new epoch so that as the training process progresses, fewer random actions will be taken, and the learning process can eventually converge.

5. The SNN Simulator

This work uses the highly flexible and easily extensible SNN simulator $Brian2$ developed by the computational neuroscience community. Brian2 is a clock-driven simulator (also supports event-driven simulation mode) implemented purely in Python and supports a range of different platforms [55]. It provides a variety of libraries to generate spike signals, create neuron groups, define connecting synapses, monitor spike signals, and control overall simulations. In this study, the input neurons have been defined using the Brian2 function $SpikeGeneratorGroup\left(\right)$, and the output neuron groups have been defined using the Brian2 function $NeuronGroup\left(\right)$. Synapses have been created using the Brian2 function $Synapses\left(\right)$ and connected using the function $Synapse.connect\left(\right)$. For the TD-STDP SNN, $n_{output}$=10. For the SD-STDP SNN, $n_{output}$=1. The counting of spikes generated by output neuron groups has been monitored with the Brian2 function $SpikeMonitor\left(\right)$.

6. Results

6.1. TD-STDP Learning

During one training session, the duration starting from the initializing/resetting of the gym cart-pole environment to the termination of the cart-pole environment is considered as one epoch. As the goal of the cart-pole problem is to keep the pole upright as long as possible, we define a trial of the cart-pole balancing problem to be successful if the number of time steps by which the pole does not fall is more significant than a threshold of $n_{Threshold}=200$ time steps. At a particular training stage, we estimate the performance of the SNN at that training stage by considering the success rate over 20 consecutive epochs centered around that stage. Figure 5a shows the success rates as functions of the number of training epochs in three independent runs.

As the training process evolves, the success rate of the SNN controller generally increases, but with several drops. These drops could result from the agent encountering states for which it does not yet have enough experience to form a good policy. This change in success rate indicates that the training process is indeed converging, although somewhat noisy. This phenomenon can also be seen in Figure 5b, which plots the root mean square change of the Q values between two consecutive training epochs against the number of training epochs. The curves exhibit fluctuations within a specific range even after prolonged training.

6.2. R-STDP Learning

We tested the three different reward functions described above. We only show the results for Reward Function 3, in which the rewards depend jointly on angular position and velocity changes. This function led to much higher success rates after training than the other two reward functions.

Using the success rate over 20 epochs as the metric for the performance of the SNN at different training stages, the results of R-STDP are plotted in Figure 6a. Figure 6a, compared with Figure 5a, shows a more satisfactory result for the use of R-STDP. As the training progresses, the success rate increases relatively smoothly. They gradually reached the value of 1 with only tiny fluctuations in less than 50 epochs. In comparison, the TD-STDP training did not show a trend to convergence until more than 300 epochs and remained imperfect and noisy until more than 400 epochs.

The accumulated reward in each training epoch is shown in Figure 6b. As intended, the R-STDP learning process led to a gradual increase in the total rewards.

7. Discussion

TD-STDP is built on the theoretical basis of classical reinforcement learning algorithms. Although for the problem examined here, the TD-STDP approach exhibits lower learning efficiency than R-STDP (R-STDP produces consistently good results after less than 50 epochs, while TD-STDP still delivers noisy results after 300 training epochs), TD-STDP does not use any assumption about the dynamics of the cart-pole system for learning. Thus, it provides a more general solution for RL problems using SNN. The shared basic concepts between traditional RL and TD-STDP-based SNNs also imply that future optimization of SNN for RL may benefit from ideas developed in previous studies on RL.

The interaction between the agent and the environment in which it roams is often crucial in RL control applications. Due to the nature of this application, the reward received by the agent for taking action is usually delayed relative to the timing of the action because the feedback of the environment is obtainable, as the sensory data are usually reactions to previously taken actions. The R-STDP learning rule employed in the cart-pole controller training successfully used the delayed reward signals to extract useful information to help the SNN generate a stable policy. This was achieved by storing the eligibility traces of the synapses and using a reward function jointly determined by current and past observations.

The solutions we presented here for the cart-pole example employ a rather simple but effective SNN structure at the expense of using discrete variables to represent the states of the cart pole. The network has only two layers without using any hidden layers. However, the goal of balancing the pole is still efficiently achieved after a reasonable amount of training. That said, we note that the success of the R-STDP SNN critically relied on the definition of a reward function (here, Reward Function 3) that utilized substantial preestablished knowledge or understanding of the dynamics of the problem. This restricts the model’s applicability to general control problems for which learning has to be based on experiences, not on assumptions about the actual dynamics of the system to be the controller. In this sense, TD-STDP presents a more general framework for the SNN to solve control problems, as no pre-established knowledge about the dynamics of the problem is needed. However, for particular problems, such generality may come with the expenses of lower performance relative to R-STDP learning with problem-specific rewards.

The one-hot coding of the state space also proves its usefulness in this application. However, the resolution for discretizing the state space may affect the control system’s performance. Moreover, as the dimension of the state space increases, the required number of discretized states (and thus the number of input neurons to cover the state space) will increase exponentially. Implementing other coding schemes for input states will be an interesting problem for future investigations.

This work has only provided a glimpse into the feasibility of using SNN to solve RL problems. The cart-pole game acts as a good starting point for more complicated control problems that could be solved using RL techniques, where SNN has shown its interesting properties that the formation of the synapse connection is heavily affected by the reward function and coding scheme design. These properties could be better explained by a more in-depth study of the observations of neuroscience experiments.

Although RL has been a well-studied field, as a wide range of RL algorithm-based applications have shown their power in various fields, the training and application of SNN in solving RL problems still remain an active field for ML researchers to explore, since these two techniques share the methodology inspired from the learning process of a biological neural system. The long-term goal of this project is to help develop control software in neuromorphic computational units. The current implementation uses an SNN software simulator. Thus, the computational cost of the current implementation cannot reflect the gain in energy efficiency of the SNN. An important follow-up work would be to implement similar SNNs on physical hardware beyond software simulators. The potential of implementing SNN at the hardware level to exploit its power efficiency and biological precision of SNN compared to former ANNs also makes SNN a promising approach to further enhance the overall performance of neural network models.