3.1. The Structure of the OFESN
The subreservoirs of the OFESN may be composed of different types of neurons or the same types of neurons. Here, we assume that the reservoir of the OFESN is composed of m subreservoirs, and each subreservoir is composed of the same types of neurons, i.e., the neuron state update model of each subreservoir is the same. The enrichment of the dynamics of the reservoir can be achieved by constructing an echo state network with the following idea:
- (1)
First, m neurons with different initial states are generated, and the Euclidean distance between any two initial states is greater than or equal to a certain number that can be either specified or generated randomly. If this number is randomly generated, the state of neurons of the subreservoirs subsequently generated will be guaranteed to have more complex differences. The m neurons generated above are referred to as master neurons, similar to the master neurons of typical neural circuits, such as olfactory cortex, cerebellar cortex, and hippocampal structures, that are responsible for the input and output of the circuit. Each master neuron becomes the core of each subreservoir and thus becomes the representative of each subreservoir. Let , , ⋯ denote these m neurons, respectively. Their initial states need to meet the inequality . Here, may be either specified or randomly generated.
- (2)
Next, with each master neuron as the center, a subreservoir is constructed around the master neuron. In each subreservoir, the other neurons except the master neuron are called sister neurons. The master neuron and the sister neurons of a subreservoir need to ensure high similarity and correlation. Thus, the master neuron can represent its own subreservoir. The communication between subreservoirs can be realized by the communication between master neurons. The sister neurons belonging to the same subreservoir can communicate with each other, but the sister neurons belonging to a different subreservoir cannot communicate with each other. The m master neurons can construct an m subreservoir, called the actual subreservoir. Let the neurons of the subreservoir satisfy . Here, denotes the sister neuron of the subreservoir, and denotes the number of neurons of the subreservoir.
- (3)
The OFESN model can provide a new connection mode as follows: (i) The connections between subreservoirs are transformed into the connections between their respective master neurons, which can be determined by the small-world network method or sparse connection. The connections between these master neurons actually generate a virtual and flexible subreservoir. The biggest difference between the virtual reservoir and the actual subreservoir is that the virtual reservoirs are only composed of the master neurons, and thus their neuron states have a bigger difference and less redundant information than those of the actual subreservoir. Therefore, the OFESN is equivalent to having a flexible virtual subreservoir and m actual subreservoirs. (ii) The sister neurons within each subreservoir are sparsely connected. (iii) The sister neurons in different subreservoirs cannot communicate with each other, and there are no connections among them. Such a new connection mode greatly simplifies the coupling connection between neurons in different reservoirs and then reduces the information redundancy. The sparse connection between the master neurons actually creates a virtual subreservoir, which makes the reservoir add a new subreservoir composed of master neurons with large diversity and thus is equivalent to increasing the number of neurons. Therefore, the OFESN model can be more suitable for the situation where the network approximation ability is poor due to the small number of neurons in the whole network reservoir, especially the situation where the number of neurons in each subreservoir is small and the number of subreservoirs is large.
The structure of the OFESN is shown in
Figure 2. In
Figure 2, the dashed lines from the output layer to the reservoir represent
, the solid lines from the reservoir to the output layer represent
, and the colored dotted line means that
can be adjusted online such that the network output
follows
, in addition to being calculated by Equation (
4). The ellipses of the black line in the reservoir represent actual subreservoirs, respectively. In each subreservoir, a circle filled with red denotes its master neuron. All master neurons construct a virtual subreservoir denoted by the ellipses of the green dotted line. The neurons of each actual subreservoir, including its master neuron and the sister neurons, use the sparse connection. Each subreservoir can be represented by its master neuron, and then the connections between actual subreservoirs can be determined by the connections between master neurons. The master neurons, i.e., the neurons of the virtual subreservoir, may use sparse connection or the small-world network method.
The state update equation of the OFESN is as follows:
where
denotes the state vector of the reservoir and
denotes the state of the subreservoir.
indicates the number of neurons of the
ith subreservoir.
,
W,
are the input connection weight matrix, the reservoir neuron connection weight matrix, and the output feedback connection weight matrix, respectively. The reservoir neuron connection weight matrix takes the following form:
where
is the connection weight matrix between the
ith subreservoir and the
jth subreservoir,
,
, ⋯,
denote the internal connection weight matrix of
m actual subreservoirs, which can be generated randomly with a certain sparsity. The dimensions of
W,
, and
are
,
, and
, respectively.
The following is an example of how to generate , , ⋯, to explain how to generate . denotes the connection weight matrix between the 1st subreservoir and the 2nd subreservoir. To reduce information redundancy and simplify the connections between neurons, the OFESN uses the connections between the master neuron of the 1st subreservoir and the master neuron of the 2nd subreservoir to represent the connections between the 1st subreservoir and the 2nd subreservoir. We assume that the first neuron of the first subreservoir is the principal neuron and the second neuron of the second subreservoir is the master neuron. Thus, the element in the 1st row and the 2nd column of denotes the connection weights between the master neuron of the 1st subreservoir and the master neuron of the 2nd subreservoir, which can be expressed as . may be a zero or nonzero value. represents that there is no connection between the master neuron of the 1st subreservoir and the master neuron of the 2nd subreservoir. represents that there is a connection between the master neuron of the 1st subreservoir and the master neuron of the 2nd subreservoir. In other words, only of may be a nonzero element, and the rest of the elements are zero. Similarly, the element in the 1st row and 3rd column of , denoted by , represents the connection weights between the master neuron of the 1st subreservoir and the master neuron of the 3rd subreservoir. Only the of may be a nonzero element, and the rest of the elements are zero. So, the element in the 1st row and mth column of is denoted by . Only the of may be a nonzero element, and the rest of the elements are zero. The values of , , ⋯, are determined by the connection weights between the neurons of the virtual subreservoir, and , , ⋯, is actually the corresponding element of the connection weight matrix of the virtual subreservoir.
From the above, the element in the ith row and the jth column of , denoted by , represents the connection weights between the master neuron of the ith subreservoir and the master neuron of the jth subreservoir. Only one element of is possibly nonzero, and the rest of the elements of are zero. The possibly nonzero element characterizes the master neuron of the ith subreservoir possibly connected with the master neuron of the jth subreservoir, and its value is determined by the random sparse connection of the virtual subreservoir. represents the connection of the master neuron itself in the ith subreservoir. So, is made up of . Let , which is the equivalent of added to . can be generated randomly with a spectral radius less than 1 and a certain sparsity. It is especially worth noting that the connection weight matrix of reservoir W is asymmetric; that is, the connection between reservoir neurons is duplex.
In addition, in Equation (
7), the input matrix
and feedback connection weight matrix
are in the following form:
After the matrix is normalized, Equation (
6) is rewritten as
where
is the input scaling factor,
is the spectral radius, and
is the output feedback scaling factor.
After normalization,
W,
,
can be, respectively, rewritten as
among them,
,
,
is the normalized matrix.
3.3. Optimizing the Global Parameters of OFESN
The models to be optimized here are Equations (
7) and (
8). The parameters to be optimized are
a,
,
,
,
(spectral radius of matrix), and
.
can be solved by linear regression method, such as pseudoinverse method. In order to simplify the operation,
,
,
is not optimized and is given in advance. Only the
parameters are optimized, and echo state property conditions are satisfied. In this paper, stochastic gradient descent method is used to optimize these parameters. Here,
.
When
, invoke the chain rule and observe (
8); we obtain
:
where
denotes the element-wise product of two vectors.
When containing
, let
. Here, we use a simple symbol
to represent the input vector whose entries are all zeros; the
, and we obtain
:
In other words, to train the output weight matrix
, the output
should be as close as possible to the teacher output
during the training process. The error
expression is as follows:
we define the squared error
as follows:
to
, there is
the global parameter update expression is as follows:
K represents the learning rate of the global parameter
q. The parameters modified in this process must ensure that the OFESN model has the echo state property in practical applications.