Recurrent Neural Network Models Based on Optimization Methods

Abstract

Many researchers have addressed problems involving time-varying (TV) general linear matrix equations (GLMEs) because of their importance in science and engineering. This research discusses and solves the topic of solving TV GLME using the zeroing neural network (ZNN) design. Five new ZNN models based on novel error functions arising from gradient-descent and Newton optimization methods are presented and compared to each other and to the standard ZNN design. Pseudoinversion is involved in four proposed ZNN models, while three of them are related to Newton’s optimization method. Heterogeneous numerical examples show that all models successfully solve TV GLMEs, although their effectiveness varies and depends on the input matrix.

1. Introduction and Preliminaries

Novel zeroing neural network (ZNN) dynamical systems are presented using various error functions that arise from gradient-descent and Newton optimization methods. Introduced models are theoretically investigated and compared to each other and to the standard ZNN design. The proposed ZNN models are applied in solving time-varying (TV) general linear matrix equations (GLMEs) with arbitrary TV real input matrices. The TV GLME problem is expressed as the matrix equation

$$\begin{gathered} A\left(t\right)Y\left(t\right)C\left(t\right)=B\left(t\right),A\left(t\right)\in {\mathbb{R}}^{m\times n},Y\left(t\right)\in {\mathbb{R}}^{n\times k},C\left(t\right)\in {\mathbb{R}}^{k\times h},B\left(t\right)\in {\mathbb{R}}^{m\times h}, \\ m\ge n,k\le h, \end{gathered}$$

(1)

in which $Y\left(t\right)$ is unknown matrix.

The solutions $Y\left(t\right)$ to the Equation (1) are extremely useful in science as well as in a wide range of engineering applications, including optimizing the manipulability of a robotic arm’s joint angles [1], making highly accurate predictions for missing quality of service data [2], tracking robotic motion and mobile objects [3], obtaining weights of feed-forward neural networks [4]. This paper presents and contrasts six ZNN models based on two new error functions for solving TV GLMEs that involve arbitrary matrices $A\left(t\right),B\left(t\right)$, and $C\left(t\right)$. It is worth mentioning that the matrix pseudoinversion is involved in four of the proposed ZNN models, while three are related to Newton’s optimization method.

The ZNN concept was coined by Zhang et al. in [5] for determining online solutions to TV matrix inversion problems. Most ZNN-based dynamical systems belong to classes of recurrent neural networks (RNNs) designed to approximate zeros of matrix, vector, or scalar equations. Besides ZNN, there is another type of RBBs, which is known as Gradient neural networks (GNN). As an outcome, numerous significant research results have been revealed in the scientific literature. ZNN and GNN dynamics for solving linear matrix equations of the form (1) were described and compared in [6]. Two Zhang functions and initiated ZNN dynamics for solving the TV GLME $A\left(t\right)Y\left(t\right)=B\left(t\right)$ were presented and investigated theoretically and numerically in [7]. A gradient-based neural dynamics for solving the matrix equation $AXB=D$ was investigated in [8]. Gradient and zeroing dynamical systems aimed to solve the system of linear equations in both the TV and constant environment are investigated in [9,10,11]. GNN neural-dynamic schemes are substantially designed for solving equations with time-invariant coefficient matrices [11]. A global overview of Zeroing dynamical systems (ZND), involving ZND aimed to solving linear matrix equations, was presented in [12]. Nonlinear Zeroing neural dynamical systems with finite-time convergence and noise-tolerant for solving linear matrix equations in time-varying scenarios were proposed and investigated in [13,14]. A ZNN design whose dynamics are based on varying gain parameter and which are suitable for solving TV GLME was proposed in [15]. A very popular class of ZNN dynamical systems is represented by ZNN models with a finite-time or pre-defined convergence time. A finite-time convergent ZNN dynamics with a robust activation was presented in [14] and applied in some engineering fields. Xiao in [16] discovered and investigated finite-time dynamical systems for solving time-dependent complex matrix equation $AX=B$. A desirable feature of ZNN dynamics is its ability to neutralize different types of noises that appear during the evolution of dynamic systems. A noise-tolerant ZNN design with nonlinear activation that involves appropriate integral term was proposed and investigated in [17]. Finite-convergent zeroing neural design in the complex domain based on the Tikhonov regularization on the matrix A and aimed at solving time-dependent matrix equations $AX=D$ was considered in [18]. Various and numerous areas of numerical linear algebra are studied by applying ZNN dynamics in the time-varying case. Solving tensor and matrix inversion on TV arrays [19], generalized eigenvalues problems [20], generalized inversion [21], matrix equations and matrix decompositions [3], quadratic optimization problems [22], and approximations of various matrix functions [23,24] are predominant utilizations of ZNNs. ZNN models based on two fuzzy-adaptive activations for resolving specific linear TV linear matrix equations (LME) were introduced in [25,26]. The hybrid models introduced in [27] are defined using suitable combinations of GNN and ZNN neural dynamics for solving matrix equations $BX=D$ or $CX=D$. Apart from different types of activation in dynamic systems, numerous studies have been devoted to the modification of the dynamic evolution of ZNN. High-order error function designs and high-order ZNNs which are designed using powers of standard Zhang functions and adopted for solving LME were proposed in [28].

Dtermining an error function (or Zhang function) $E\left(t\right)$, based on a suitable expression that reflects the problem being solved, is the first step in developing ZNN dynamics [29]. The second stage takes the dynamical development

$$\dot{E}\left(t\right)=\frac{\mathrm{d}E\left(t\right)}{\mathrm{d}t}=-\lambda \mathcal{F}\left(E\left(t\right)\right),\lambda >0$$

(2)

in which $\dot{E}\left(t\right)\in {\mathbb{R}}^{m\times n}$ means the time derivative of $E\left(t\right)\in {\mathbb{R}}^{m\times n}$, a real number $\lambda >0$ represents the scaling parameter required for accelerating the convergence speed, and $\mathcal{F}(·):{\mathbb{R}}^{m\times n}\to {\mathbb{R}}^{m\times n}$ denotes elementwise usage of appropriately defined odd-increasing activation function on $E\left(t\right)$. Target of our research is the linear ZNN design

$$\dot{E}\left(t\right)=-\lambda E\left(t\right),\lambda >0.$$

(3)

The system of first-order differential Equation (3), which defines the ZNN evolution law, has the analytical solution $E\left(t\right)=E\left(0\right){\mathrm{e}}^{-\lambda t}$ [12]. Thus, $E\left(t\right)$ in the standard ZNN is exponentially convergent to the zero 0 with the exponential convergence rate $\lambda$ as $t\to +\infty$.

The ZNN model for solving the GLME (1) is defined using the matrix-valued Zhangian

$$E\left(t\right)=A\left(t\right)Y\left(t\right)C\left(t\right)-B\left(t\right),A\left(t\right)\in {\mathbb{R}}^{m\times n},Y\left(t\right)\in {\mathbb{R}}^{n\times k},C\left(t\right)\in {\mathbb{R}}^{k\times h},B\left(t\right)\in {\mathbb{R}}^{m\times h}.$$

(4)

The main drawback of the basic ZNN design (3) and (4), based on the standard error function $E\left(t\right)$, is its limited application to cases where the input matrices meet certain requirements. Motivated by that shortcoming, in this research we propose two new error functions derived from optimization methods. The error function $E_G\left(t\right)$ is defined by zeroing the gradient of the Frobenius matrix norm of $E\left(t\right)$, while the error function $E_N\left(t\right)$ is defined by the Newton step corresponding to the Frobenius matrix norm of $E\left(t\right)$. That is why we use the terms Gradient zeroing neural network (GZNN) and Newton zeroing neural network (NZNN) for dynamic systems that are defined based on the new error functions $E_G$ and $E_N$, respectively.

Moreover, we noticed that the analysis of the complete set of solutions to the Equation (1) has not been investigated so far. In previous research, the solution based on invertible matrices A and C has been analyzed, which corresponds to the solution $Y\left(t\right)=A^{-1}\left(t\right)B\left(t\right)C^{-1}\left(t\right)$ [6,26]. Our idea is to look for least-squares solutions, under weaker assumptions that allow the coefficient-matrices to be singular.

The following highlights are key results of this work:

• A new and unique ZNN development, based on two novel error functions $E_G\left(t\right)$ and $E_N\left(t\right)$, is used to address the problem of solving TV GLMEs (1) with arbitrary TV real input matrices.
• Six ZNN models based on different error functions are presented and compared, five of which are new. Pseudoinversion is involved in three new ZNN models, while two are related to Newton’s optimization method.
• A precise convergence analysis of the proposed ZNNs is presented and the set of solutions is defined using known results from linear algebra.
• Six numerical examples confirm that all considered models can solve TV GLMEs, although their effectiveness varies depending on input matrices.

The following is the structure of the presentation. Section 2 presents the motivation and describes the main results. Section 3 defines six ZNN dynamical systems and analyses their solvability, general solution, sets of all solutions, and convergence properties. Section 4 introduces and analyses simulation results obtained through six numerical examples that involve input matrices of different types. Finally, in Section 5, there are closing remarks.

The paper follows classical notations: $I_n\in {\mathbb{R}}^{n\times n}$ signifies the identity matrix of the order n; $I_{m,n}\in {\mathbb{R}}^{m\times n}$ signifies a $m\times n$ matrix with ones on the main diagonal and zeros elsewhere; ${\mathbf{1}}_n$ and and ${\mathbf{1}}_{m,n}$, respectively, signify a vector in ${\mathbb{R}}^n$ and a matrix in ${\mathbb{R}}^{m\times n}$ consisting of ones; $O_{m,n}\in {\mathbb{R}}^{m\times n}$ signifies the zero $m\times n$ array; ⊗ designates the Kronecker product; $\mathrm{vec}(·)$ stands for the vectorization; ⊙ means the Hadamard product; ${∥·∥}_F$ signifies the Frobenius norm; the operators ${(·)}^{\mathrm{T}}$, ${(·)}^{-1}$ and ${(·)}^{†}$, respectively, signify the matrix transposition, inversion and the pseudoinversion.

2. Motivation and Description of Main Results

The traditional ZNN dynamics for solving (1) is based on the error function (Zhangian) $E\left(t\right)$ defined in (4), and follows the dynamic flow

$$\dot{E}\left(t\right)=-\lambda E\left(t\right)=-\lambda \left(A\left(t\right)Y\left(t\right)C\left(t\right)-B\left(t\right)\right).$$

(5)

The standard ZNN design (5) developed to solve the GLME (1) requires invertible matrices A and C to force $E\left(t\right)$ to zero, which the unique solution $Y\left(t\right)=A^{-1}\left(t\right)B\left(t\right)C^{-1}\left(t\right)$ [26,30]. Our goal is to resolve this restriction and propose dynamical evolutions based on error functions that tend to zero in more general cases, which include the singularity of matrix-coefficients $A\left(t\right)$ and $C\left(t\right)$. This research aims to find new neurodynamical models based on different error functions. Our motivation in defining new Zhang functions has its origin in gradient-descent methods for solving the unconstrained minimization problem

$$minf\left(\mathbf{x}\right),\mathbf{x}\in {\mathbb{R}}^n,$$

(6)

where $f:{\mathbb{R}}^n\to \mathbb{R}$ is a continuously-differentiable function bounded from below. The gradient descent (GD) iterative scheme for solving (6) is defined by

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k-{\alpha }_k{\mathbf{g}}_k,$$

(7)

whereby ${\mathbf{g}}_k=\nabla f\left({\mathbf{x}}_k\right)$ is the gradient vector and ${\alpha }_k$ is appropriately defined step size. The convergence result

$$\underset{k\to \infty }{lim}\parallel {\mathbf{g}}_k\parallel =0$$

(8)

for iterations (7) is widely known in the literature [31].

The Newton method with line search for solving (6) is defined by

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k-{\alpha }_k{G}_k^{-1}{\mathbf{g}}_k,$$

(9)

where $G_k^{-1}$ denotes the inverse of the Hessian matrix $G_k={▽}^{2}f\left({\mathbf{x}}_k\right)$.

New error functions $E_G\left(t\right)$ and $E_N\left(t\right)$ are defined using analogies with gradient-descent and Newton method for nonlinear optimization. The residual matrix $E\left(t\right)=A\left(t\right)Y\left(t\right)C\left(t\right)-B\left(t\right)$ is the desired goal function which is forced to the zero matrix. The gradient of the Frobenius norm

$${\epsilon }_Y=\frac{{\parallel E\left(t\right)\parallel }_F^2}{2}=\frac{{\parallel A\left(t\right)Y\left(t\right)C\left(t\right)-B\left(t\right)\parallel }_F^2}{2}$$

is equal to [6]

$$\frac{\partial {\epsilon }_Y}{\partial Y}=\nabla {\epsilon }_Y=A^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}=E_G\left(t\right).$$

Follwing iterations defined in [32], the gradient-descent iterations for minimizing the goal function ${\epsilon }_Y$ are defined by

$$Y_{k+1}=Y_k-{\alpha }_k{A}^{\mathrm{T}}(A{Y}_{k}C-B)C^{\mathrm{T}},{\alpha }_k>0,$$

taking an arbitrary initial matrix $Y_0\in {\mathbb{R}}^{n\times k}$. The Gradient neural network (GNN) dynamic evolution minimizes ${\parallel AY\left(t\right)C-B\parallel }_F^2$ and it is based on the direct proportionality between the time derivative $\dot{Y}\left(t\right)$ and the negative gradient of the goal function ${\epsilon }_Y\left(t\right)$ [33,34]:

$$\dot{Y}\left(t\right)=-\lambda \frac{\partial {\epsilon }_Y\left(t\right)}{\partial Y},Y\left(0\right)=Y_0.$$

(10)

Our goal is to define the ZNN design for solving the TV GLME (1) based on the error function

$$E_G\left(t\right):=\nabla {\epsilon }_Y\left(t\right)=A^{\mathrm{T}}\left(AYC-B\right)C^{\mathrm{T}}=A^{\mathrm{T}}E\left(t\right)C^{\mathrm{T}},$$

(11)

which initiates the following dynamics, termed GZNN dynamics:

$${\dot{E}}_G\left(t\right)=-\lambda E_G\left(t\right)=-\lambda \left(A^{\mathrm{T}}\left(AYC-B\right)C^{\mathrm{T}}\right).$$

(12)

Further, Hessian of ${\epsilon }_Y$ in the case $C:=I$ is equal to ${\nabla }^2{\epsilon }_Y=\frac{{\partial }^2{\epsilon }_Y}{\partial Y^2}=A^{\mathrm{T}}A.$ In the case $A:=I$, Hessian of ${\epsilon }_Y$ is equal to ${\nabla }^2{\epsilon }_Y=\frac{{\partial }^2{\epsilon }_Y}{\partial Y^2}=C{C}^{\mathrm{T}}.$ Following the results from [32], the Newton iterations with line search in the time-invariant case can be defined by

$$Y_{k+1}=Y_k-{\alpha }_k{\left(A^{\mathrm{T}}A\right)}^{-1}{A}^{\mathrm{T}}(A{Y}_{k}C-B)C^{\mathrm{T}}{\left(C{C}^{\mathrm{T}}\right)}^{-1},{\alpha }_k>0.$$

Consider the real numbers ${\delta }_{{}_{A^{\mathrm{T}}A}}$ and ${\delta }_{{}_{C{C}^{\mathrm{T}}}}$ defined by

$${\delta }_{{}_{A^{\mathrm{T}}A}}=\left\{\begin{array}{cc}\epsilon ,\hfill & \mathrm{rank}\left(A^{\mathrm{T}}A\right)<n,\hfill \\ 0,\hfill & \mathrm{rank}\left(A^{\mathrm{T}}A\right)=n,\hfill \end{array}\right.{\delta }_{{}_{C{C}^{\mathrm{T}}}}=\left\{\begin{array}{cc}\epsilon ,\hfill & \mathrm{rank}\left(C{C}^{\mathrm{T}}\right)<k,\hfill \\ 0,\hfill & \mathrm{rank}\left(C{C}^{\mathrm{T}}\right)=k,\hfill \end{array}\right.$$

where $\epsilon >0$ is a small real regularization parameter. So, we define the Zhangian in the case of $\mathrm{rank}\left(A\right)\le n\le m$, $\mathrm{rank}\left(C\right)\le k\le h$ as

$$E_N\left(t\right):={\left(A^{\mathrm{T}}\left(t\right)A\left(t\right)+{\delta }_{{}_{A^{\mathrm{T}}A}}I\right)}^{-1}{A}^{\mathrm{T}}\left(A\left(t\right)Y\left(t\right)C\left(t\right)-B\left(t\right)\right)C^{\mathrm{T}}\left(t\right){\left(C\left(t\right)C^{\mathrm{T}}\left(t\right)+{\delta }_{{}_{C{C}^{\mathrm{T}}}}I\right)}^{-1}.$$

(13)

Given the well-known limit representation of the Moore-Penorse inverse, restated in [35], it can be obtained

$$\underset{\epsilon \to 0}{lim}{\left(A^{\mathrm{T}}A+{\delta }_{{}_{A^{\mathrm{T}}A}}I\right)}^{-1}{A}^{\mathrm{T}}=A^{†},\underset{\epsilon \to 0}{lim}{C}^{\mathrm{T}}{\left(C{C}^{\mathrm{T}}+{\delta }_{{}_{C{C}^{\mathrm{T}}}}I\right)}^{-1}=C^{†},$$

which implies

$$E_N\left(t\right)\to A^{†}\left(t\right)(A\left(t\right)Y\left(t\right)C\left(t\right)-B\left(t\right))C^{†}\left(t\right)=A^{†}\left(t\right)E\left(t\right)C^{†}\left(t\right)$$

as $\epsilon \to 0$. The ZNN design for solving $E_N\left(t\right)=0$ based on the error function $E_N\left(t\right)$, and termed NZNN, is defined by

$${\dot{E}}_N\left(t\right)=-\lambda E_N\left(t\right).$$

(14)

Dynamics (14) forces $E_N\left(t\right)$ to zero, which (in view of (13)) coincides with the zero of $E_G\left(t\right)$.

Solvability and Solutions of Proposed Dynamical Systems

In this subsection we investigate conditions for solvability of the matrix equations $E\left(t\right)=0$, $E_G\left(t\right)=0$ and $E_N\left(t\right)=0$ and their sets of solutions.

Lemma 1 defines conditions for the solvability and defines solutions of the linear matrix equation $AYC=B$. The results is obtained as a consequence of [35] (p. 52, Theorem 1) in the time-varying case.

Lemma 1

([35] p. 52, Theorem 1). For arbitrary $A\left(t\right)\in {\mathbb{C}}^{m\times n}$, $B\left(t\right)\in {\mathbb{C}}^{p\times q}$, $D\left(t\right)\in {\mathbb{C}}^{m\times q}$, the general linear matrix equation

$$A\left(t\right)Y\left(t\right)C\left(t\right)=B\left(t\right)$$

(15)

is solvable if and only if

$$A\left(t\right)A^{†}\left(t\right)B\left(t\right)C^{†}\left(t\right)C\left(t\right)=B\left(t\right),$$

(16)

in which case its general solution is given by

$$A^{†}\left(t\right)B\left(t\right)C^{†}\left(t\right)+Q\left(t\right)-A^{†}\left(t\right)A\left(t\right)Q\left(t\right)C\left(t\right)C^{†}\left(t\right),$$

(17)

for arbitrary $Q\left(t\right)\in {\mathbb{C}}^{n\times p}$.

Corollary 1.

If (16) is satisfied, the set of all solutions to (15) is defined by

$${\mathsf{\Theta }}_Q=\left\{A^{†}\left(t\right)B\left(t\right)C^{†}\left(t\right)+Q\left(t\right)-A^{†}\left(t\right)A\left(t\right)Q\left(t\right)C\left(t\right)C^{†}\left(t\right)\mid Q\left(t\right)\in {\mathbb{C}}^{n\times p}\right\}.$$

(18)

The set ${\mathsf{\Theta }}_Q$ is termed as solutions set in rest of the paper.

Lemma 2 restates results from [8] (Theorem 2.2). These results are used in Remark 1 to define the solution which will be used in numerical comparison of defined models.

Lemma 2

([8] Theorem 2.2). Assume that constant matrices $A\in {\mathbb{R}}^{m\times n}$, $C\in {\mathbb{R}}^{k\times h}$ and $B\in {\mathbb{R}}^{m\times h}$ satisfy

$$A{A}^{†}B{C}^{†}C=B.$$

(19)

Then the activation state variables matrix $Y\left(t\right)$ of the dynamical system

$$\dot{Y}\left(t\right)=-\gamma A^{\mathrm{T}}(AV\left(t\right)C-B)C^{\mathrm{T}}$$

(20)

is convergent as $t\to +\infty$ and has the equilibrium state

$$Y\left(t\right)\to \tilde{Y}=A^{†}C{B}^{†}+Y\left(0\right)-A^{†}AY\left(0\right)C{C}^{†}$$

(21)

for every initial state matrix $Y\left(0\right)\in {\mathbb{R}}^{n\times p}$.

Remark 1.

Theorem 2.2 in [8] claims that the activation state variables matrix $Y\left(t\right)$ of the GNN dynamics (20) for solving (15) in the time-invariant case converges as $t\to +\infty$ to the equilibrium state defined in (21).

Accordingly, we will use the equilibrium matrix

$${\tilde{Y}}_{Y\left(0\right)}\left(t\right)=A^{†}\left(t\right)B\left(t\right)C^{†}\left(t\right)+Y\left(0\right)-A^{†}\left(t\right)A\left(t\right)Y\left(0\right)C\left(t\right)C^{†}\left(t\right)\in {\mathsf{\Theta }}_Q,$$

(22)

for the comparison of the proposed models in performed numerical experiments.

Theorem 1 investigates conditions for the solvability of the equations $E\left(t\right)=0$, $E_G\left(t\right)=0$, $E_N\left(t\right)=0$, and solutions sets to these equations.

Theorem 1.

Consider arbitrary smooth TV matrices $A\left(t\right)\in {\mathbb{R}}^{m\times n}$, $C\left(t\right)\in {\mathbb{R}}^{k\times h}$ and $B\left(t\right)\in {\mathbb{R}}^{m\times h}$. The following statements are valid.

(a)

The equation $E\left(t\right)=0$ is solvable if and only if (16) holds, and the solutions set to $E\left(t\right)=0$ is ${\mathsf{\Theta }}_Q$ defined in (18).

(b)

If $\mathrm{rank}\left(A\right)\le n\le m$, the equation $E_G\left(t\right)=0$ is always solvable and the solutions set to $E_G\left(t\right)=0$ is ${\mathsf{\Theta }}_Q$ defined by (18).

(c)

If $\mathrm{rank}\left(A\right)\le n\le m$, the equation $E_N\left(t\right)=0$ is always solvable and the solutions set to $E_N\left(t\right)=0$ is ${\mathsf{\Theta }}_Q$ defined by (18).

Proof. 

(a)

The error function $E\left(t\right)$ in the traditional ZNN (TZNN) model is defined by (4) and solves the equation $E\left(t\right)=0$. This part of the proof follows from known results on the solvability and the general solution to the matrix Equation (15), given in Lemma 1, and its application to the matrix equation $E\left(t\right)=0⟺A\left(t\right)Y\left(t\right)C\left(t\right)=B\left(t\right)$.

(b)

The dynamics (12) forces $E_G\left(t\right)=\nabla {\epsilon }_Y\left(t\right)=A^{\mathrm{T}}\left(AYC-B\right)C^{\mathrm{T}}$ to zero. On the other hand, according to Lemma 1, the error function $E_G\left(t\right)$ vanishes in the case.

$$E_G\left(t\right)=0⟺A^{\mathrm{T}}AYC{C}^{\mathrm{T}}=A^{\mathrm{T}}B{C}^{\mathrm{T}}.$$

As a consequence, the matrix equation $E_G\left(t\right)=0$ is consistent, its general solution is is given by (17), and its solutions set is defined in (18). The consistency is confirmed by the identity

$$A^{\mathrm{T}}A{\left(A^{\mathrm{T}}A\right)}^{†}{A}^{\mathrm{T}}B{C}^{\mathrm{T}}{\left(C{C}^{\mathrm{T}}\right)}^{†}C{C}^{\mathrm{T}}=A^{\mathrm{T}}B{C}^{\mathrm{T}}.$$

In addition, the general solution to $E_G\left(t\right)=0$ is given by (17). Indeed, using Lemma 1, the general solution to $E_G\left(t\right)=0$ is equal to

$$\begin{array}{ccc}Y& =& {\left(A^{\mathrm{T}}A\right)}^{†}{A}^{\mathrm{T}}B{C}^{\mathrm{T}}{\left(C{C}^{\mathrm{T}}\right)}^{†}+Q-{\left(A^{\mathrm{T}}A\right)}^{†}{A}^{\mathrm{T}}AQC{C}^{\mathrm{T}}{\left(C{C}^{\mathrm{T}}\right)}^{†}\\ & =& A^{†}B{C}^{†}+Q+A^{†}AQC{C}^{†}.\end{array}$$

(23)

(c)

In the case $\mathrm{rank}\left(A\right)\le n\le m$, $\mathrm{rank}\left(C\right)\le k\le h$, the Zhangian $E_N\left(t\right)$ becomes the zero matrix in the case

$$E_N\left(t\right)=0⟺{\left(A^{\mathrm{T}}A+{\delta }_{{}_{A^{\mathrm{T}}A}}I\right)}^{-1}{A}^{\mathrm{T}}\left(AYC-B\right)C^{\mathrm{T}}{\left(C{C}^{\mathrm{T}}+{\delta }_{{}_{C{C}^{\mathrm{T}}}}I\right)}^{-1}=0.$$

(24)

The GLME $E_N\left(t\right)=0$ is always solvable and its solutions set is equal to (18). This statement follows from

$$\begin{array}{cc}{E}_N\left(t\right)=0& ⟺A^{\mathrm{T}}\left(AYC-B\right)C^{\mathrm{T}}=0\\ & ⟺E_G\left(t\right)=0.\end{array}$$

Following general results from [35] (p. 52, Theorem 1) it is straightforward to verify that (18) is the general set of solutions to $E_N\left(t\right)=0$.

   □

Remark 2.

The general conclusion is that the neural dynamics (12) forces the residual matrix $E_G\left(t\right)=\nabla {\epsilon }_Y\left(t\right)=A^{\mathrm{T}}\left(AYC-B\right)C^{\mathrm{T}}$ to zero. According to (23), the ZNN dynamics (12) reaches the solution $Y\left(t\right)$ of the form (23) which coincides with the solution (18) to $E\left(t\right)=A\left(t\right)Y\left(t\right)C\left(t\right)-B\left(t\right)\to 0$. Finally, $E_N\left(t\right)\to 0$ is equivalent to $E_G\left(t\right)\to 0$.

In this way, we explain one of the main advantages of the proposed models: the matrix equation $E\left(t\right)=0$ is solvable under the condition (16), while zeroing $E_G\left(t\right)=0$ and $E_N\left(t\right)=0$ are always consistent. In view of (18) and (23), the general solutions to all three equations $E\left(t\right)=0$, $E_G\left(t\right)=0$ and $E_N\left(t\right)=0$ are identical.

Corollary 2.

Let us consider the solutions set

$$\mathsf{\Lambda }=\left\{{\tilde{Y}}_{Y\left(0\right)}\left(t\right)\right|Y\left(0\right)\in {\mathbb{R}}^{n\times k}\},$$

generated by all possible initial states $Y\left(0\right)$ of the form (22). The following statements are valid.

(a)

$A\mathsf{\Lambda }C=\left\{B\right\}$.

(b)

An arbitrary solution ${\tilde{Y}}_{Y\left(0\right)}\left(t\right)\in \mathsf{\Lambda }$, defined in (22), is a least squares solution to (1).

(c)

The unique solution ${\tilde{Y}}_0\left(t\right)\in \mathsf{\Lambda }$ produces the best-approximate (i.e., the minimal-norm least-squares) solution $A^{†}B{C}^{†}$ to (1).

Proof. 

Part (a) follows from Lemma 1.

(b) Following the results from [36,37],

$${\parallel AYC-B\parallel }_2\ge {\parallel A{A}^{†}B{C}^{†}C-B\parallel }_2,$$

for arbitrary solution Y to (1), where the equality is valid if $Y\in {\tilde{Y}}_{Y\left(0\right)}\left(t\right)$.

(c) Follows from the known result originated in [36,37]

$$\parallel A^{†}B{C}^{†}{\parallel }_2\le {\parallel A^{†}B{C}^{†}+Y-A^{†}AY{C}^{†}C\parallel }_2.$$

The proof is complete.    □

3. Various ZNN Models for Solving TV GLMEs

The current section defines and analyses six ZNN expansions for calculating online solutions of TV GLMEs, including TZNN which is based on the traditional ZNN approach, GZNN based on the gradient ZNN approach, NZNNV1, NZNNV2 and NZNNV3 which are based on the Newton’s optimization method, and DZNN based on the ZNN approach for finding the direct solution produced with the use of pseudoinverse. The TZNN and GZNN models provided here cover all potential scenarios for arbitrary TV real input matrices $A\left(t\right),B\left(t\right),C\left(t\right)$ for solving the TV GLME (1).

3.1. Traditional ZNN Model

Considering smooth arbitrary TV matrices $A\left(t\right)\in {\mathbb{R}}^{m\times n}$, $C\left(t\right)\in {\mathbb{R}}^{k\times h}$ and $B\left(t\right)\in {\mathbb{R}}^{m\times h}$, the error function in the traditional ZNN (TZNN) model is established as

$$E\left(t\right)=A\left(t\right)Y\left(t\right)C\left(t\right)-B\left(t\right),$$

(25)

under conditions

$$\{A,B,C\}\in \mathcal{D}=\left\{A\in {\mathbb{R}}^{m\times n},C\in {\mathbb{R}}^{k\times h},B\in {\mathbb{R}}^{m\times h},m=n=\mathrm{rank}\left(A\right),k=h=\mathrm{rank}\left(C\right)\right\}.$$

(26)

It is important to note that the desired solution is $Y\left(t\right)\in {\mathbb{R}}^{n\times k}$. The time derivative of $E\left(t\right)$ defined in (25) is equal to

$$\dot{E}\left(t\right)=\dot{A}\left(t\right)Y\left(t\right)C\left(t\right)+A\left(t\right)\dot{Y}\left(t\right)C\left(t\right)+A\left(t\right)Y\left(t\right)\dot{C}\left(t\right)-\dot{B}\left(t\right).$$

(27)

Therefore, the following implicit dynamics can be obtained considering the linear ZNN flow (2)

$$\begin{gathered} A\left(t\right)\dot{Y}\left(t\right)C\left(t\right)=-\lambda (A\left(t\right)Y\left(t\right)C\left(t\right)-B\left(t\right))-\dot{A}\left(t\right)Y\left(t\right)C\left(t\right)-A\left(t\right)Y\left(t\right)\dot{C}\left(t\right)+\dot{B}\left(t\right), \\ \{A,B,C\}\in \mathcal{D}. \end{gathered}$$

(28)

which is just the linear ZNN model (8) in [6]. Using the Kronecker product in conjunction with the vectorization, the dynamics (28) is modified and, as a result, we obtain the following ZNN model in vector form

$$\begin{gathered} (C^{\mathrm{T}}\left(t\right)\otimes A\left(t\right))\mathrm{vec}\left(\dot{Y}\left(t\right)\right)=\mathrm{vec}\left(-\lambda (A\left(t\right)Y\left(t\right)C\left(t\right)-B\left(t\right))-\dot{A}\left(t\right)Y\left(t\right)C\left(t\right)-A\left(t\right)Y\left(t\right)\dot{C}\left(t\right)+\dot{B}\left(t\right)\right), \\ \{A,B,C\}\in \mathcal{D}. \end{gathered}$$

(29)

The ZNN model (29) will be termed as TZNN model. The solution to the dynamical system (29) can be approximated using the standard Matlab ode solver. If $C^{\mathrm{T}}\left(t\right)\otimes A\left(t\right)$ is a nonsingular mass matrix, the exponential convergence speed achieved by the TZNN design (29) to the exact TV solution of the TV GLME (1) is proved in [6] (Theorem 1).

In [6], the authors vectorized the TV GLME $A\left(t\right)Y\left(t\right)C\left(t\right)=B\left(t\right)$ into its equivalent form $(C^{\mathrm{T}}\left(t\right)\otimes A\left(t\right))\mathrm{vec}\left(Y\left(t\right)\right)=\mathrm{vec}\left(B\left(t\right)\right).$ We restate the main result from [6] (Theorem 1) in the form adopted for (1) to improve readability of the paper. This result defines Unique-solution condition that requires invertibility of the mass matrix $(C^{\mathrm{T}}\left(t\right)\otimes A\left(t\right))$.

Lemma 3

([6] Theorem 1). Assume that time-varying coefficient matrices in (1) are smooth. If the following Unique-solution condition

$${(C^{\mathrm{T}}\left(t\right)\otimes A\left(t\right))}^{\mathrm{T}}(C^{\mathrm{T}}\left(t\right)\otimes A\left(t\right))\ge \delta I,\forall t\ge 0,$$

(30)

is satisfied for a real number $\delta >0$, then the matrix $Y\left(t\right)$ of the Zhang neural network (28), starting from arbitrary initial state $Y\left(0\right)$, converges exponentially to the theoretical solution of (1) with the exponential convergence rate λ.

Vectorized ZNN design (29) exploits the same mass matrix $C^{\mathrm{T}}\left(t\right)\otimes A\left(t\right)$ and requires its invertibility. Using ${(C^{\mathrm{T}}\left(t\right)\otimes A\left(t\right))}^{-1}=C^{\mathrm{T}}{\left(t\right)}^{-1}A{\left(t\right)}^{-1}$, we conclude that the Unique-solution condition requires invertibility of $A\left(t\right)$ and $C\left(t\right)$. Therefore, if $C^{\mathrm{T}}\left(t\right)\otimes A\left(t\right)$ is nonsingular then (16) is satisfied.

Corollary 3.

Let $A\left(t\right)\in {\mathbb{R}}^{m\times n}$, $C\left(t\right)\in {\mathbb{R}}^{k\times h}$ and $B\left(t\right)\in {\mathbb{R}}^{m\times h}$ be differentiable. If the condition (26) is satisfied, starting from arbitrary initial value $Y\left(0\right)$, the TZNN model (40) converges exponentially and globally to $A^{-1}\left(t\right)B\left(t\right)C^{-1}\left(t\right)$.

3.2. Gradient ZNN (GZNN) Model

The gradient-based ZNN (GZNN) approach considers arbitrary smooth TV matrices $A\left(t\right)\in {\mathbb{R}}^{m\times n}$, $C\left(t\right)\in {\mathbb{R}}^{k\times h}$ and $B\left(t\right)\in {\mathbb{R}}^{m\times h}$ for solving

$$A^{\mathrm{T}}\left(t\right)A\left(t\right)Y\left(t\right)C\left(t\right)C^{\mathrm{T}}\left(t\right)=A^{\mathrm{T}}\left(t\right)B\left(t\right)C^{\mathrm{T}}\left(t\right),$$

(31)

where $Y\left(t\right)\in {\mathbb{R}}^{n\times k}$ is the desired solution. According to general requirements of the ZNN design (2) [6,30], we consider the constraints

$$\{A,B,C\}\in {\mathcal{D}}_G=\left\{A\in {\mathbb{R}}^{m\times n},C\in {\mathbb{R}}^{k\times h},B\in {\mathbb{R}}^{m\times h},m\ge n\ge \mathrm{rank}\left(A\left(t\right)\right),h\ge k\ge \mathrm{rank}\left(C\left(t\right)\right)\right\}.$$

(32)

The error function in GZNN design will be defined by

$$E_G\left(t\right)=A^{\mathrm{T}}\left(t\right)\left(A\left(t\right)Y\left(t\right)C\left(t\right)-B\left(t\right)\right)C^{\mathrm{T}}\left(t\right)=A^{\mathrm{T}}\left(t\right)E\left(t\right)C^{\mathrm{T}}\left(t\right),\{A,B,C\}\in {\mathcal{D}}_G.$$

(33)

The time derivative of (33) is

$${\dot{E}}_G\left(t\right)={\dot{A}}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}+A^{\mathrm{T}}(\dot{A}YC+A\dot{Y}C+AY\dot{C}-\dot{B})C^{\mathrm{T}}+A^{\mathrm{T}}(AYC-B){\dot{C}}^{\mathrm{T}}.$$

(34)

Then, considering (2), it can be obtained:

$$\begin{array}{cc}{A}^{\mathrm{T}}A\dot{Y}C{C}^{\mathrm{T}}=& -\lambda A^{\mathrm{T}}(AYC-B)-{\dot{A}}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}-A^{\mathrm{T}}(\dot{A}YC+AY\dot{C}-\dot{B})C^{\mathrm{T}}\\ & -A^{\mathrm{T}}(AYC-B){\dot{C}}^{\mathrm{T}},\{A,B,C\}\in {\mathcal{D}}_G.\end{array}$$

(35)

Using the vectorization in conjunction with the Kronecker product, the dynamics (35) is modified as

$$\begin{gathered} (C{C}^{\mathrm{T}}\otimes A^{\mathrm{T}}A)\mathrm{vec}\left(\dot{Y}\right)=\mathrm{vec}\left(-\lambda A^{\mathrm{T}}(AYC-B)-{\dot{A}}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}-A^{\mathrm{T}}(\dot{A}YC+AY\dot{C}-\dot{B})C^{\mathrm{T}}-A^{\mathrm{T}}(AYC-B){\dot{C}}^{\mathrm{T}}\right), \\ \{A,B,C\}\in {\mathcal{D}}_G. \end{gathered}$$

(36)

Replacements

$$\begin{array}{cc}\hfill W\left(t\right)=& C{C}^{\mathrm{T}}\otimes A^{\mathrm{T}}A\hfill \\ \hfill r\left(t\right)=& \mathrm{vec}\left(-\lambda A^{\mathrm{T}}(AYC-B)-{\dot{A}}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}-A^{\mathrm{T}}(\dot{A}YC+AY\dot{C}-\dot{B})C^{\mathrm{T}}-A^{\mathrm{T}}(AYC-B){\dot{C}}^{\mathrm{T}}\right)\hfill \\ \hfill \dot{\mathbf{y}}\left(t\right)=& \mathrm{vec}\left(\dot{Y}\left(t\right)\right),\mathbf{y}=\mathrm{vec}\left(Y\left(t\right)\right)\hfill \end{array}$$

(37)

lead to the dynamical evolution

$$W\left(t\right)\dot{\mathbf{y}}\left(t\right)=r\left(t\right).$$

(38)

The mass matrix $W\left(t\right)$ in (38) is singular in rank conditions $\mathrm{rank}\left(A\right)<min\{m,n\}$ or $\mathrm{rank}\left(C\right)<min\{k,h\}$. The Tikhonov regularization is the principle widely exploited to solve such singularity problems. As a result, $W\left(t\right)$ is regularized by the matrix

$$M\left(t\right)=\left\{\begin{array}{cc}W\left(t\right),\hfill & \mathrm{rank}\left(A\right)=min\{m,n\}\&\mathrm{rank}\left(C\right)=min\{k,h\}\hfill \\ W\left(t\right)+\beta I_{kn},\hfill & \mathrm{rank}\left(A\right)<min\{m,n\}\mathrm{or}\mathrm{rank}\left(C\right)<min\{k,h\}\hfill \end{array}\right.$$

(39)

where $\beta \ge 0$ is a small regularization quantity. Then the following ZNN model is used instead of (38):

$$M\left(t\right)\dot{\mathbf{y}}\left(t\right)=r\left(t\right),$$

(40)

where $M\left(t\right)$ is a nonsingular mass matrix. The ZNN (40) is named GZNN, and it is solvable using a proper ode Matlab solver. A global convergence with exponential speed of the GZNN design (40) to the exact TV solution of the TV GLME (31) is certified in Theorem 2.

Theorem 2.

Let $A\left(t\right)\in {\mathbb{R}}^{m\times n}$, $C\left(t\right)\in {\mathbb{R}}^{k\times h}$ and $B\left(t\right)\in {\mathbb{R}}^{m\times h}$ be differentiable. If the condition (32) is satisfied, starting from arbitrary initial value $\mathbf{y}\left(0\right)$, the vectorized GZNN model (40) converges exponentially and globally to one element from the set ${\mathsf{\Theta }}_Q$ defined in (18), which solves the TV GLME (31).

Proof. 

To obtain the solution $\mathbf{y}\left(t\right)$ to the TV GLMEs (31), the Zhangian matrix is defined by (33) inline with its time derivative (34). The ZNN expansion based on (33) leads to the model (35). Furthermore, from the derivation procedure, (40) is an equivalent form of (35), which represents the standard GZNN model. Since a regularization parameter $\beta$ is involved in the the definition of the mass matrix $M\left(t\right)$ in (40), $M\left(t\right)$ is invertible. From [6] (Theorem 1), restated in Lemma 3, the unknown matrix $Y\left(t\right)$ in (35) converges to the exact solution as $t\to \infty$. According to Theorem 1(b), the matrix equation $E_G\left(t\right)=0$ is always solvable and its solutions are included in ${\mathsf{\Theta }}_Q$.    □

3.3. Newton ZNN Model (Version 1)

Applying the limit representation of the pseudoinverse [35]

$$\underset{\epsilon \to 0}{lim}{\left(A^{\mathrm{T}}A+{\delta }_{{}_{A^{\mathrm{T}}A}}I\right)}^{-1}{A}^{\mathrm{T}}=A^{†},\underset{\epsilon \to 0}{lim}{C}^{\mathrm{T}}{\left(C{C}^{\mathrm{T}}+{\delta }_{{}_{C{C}^{\mathrm{T}}A}}I\right)}^{-1}=C^{†},$$

the error function (13) of the Newton ZNN version 1 (NZNNV1) model will be defined by

$$E_N\left(t\right)=A^{†}\left(t\right)(A\left(t\right)Y\left(t\right)C\left(t\right)-B\left(t\right))C^{†}\left(t\right),\{A,B,C\}\in {\mathcal{D}}_G.$$

(41)

So, the first variant of the Newton ZNN approach considers the smooth arbitrary TV matrices $A\left(t\right)\in {\mathbb{R}}^{m\times n}$, $C\left(t\right)\in {\mathbb{R}}^{k\times h}$ and $B\left(t\right)\in {\mathbb{R}}^{m\times h}$ for solving the following GLMEs:

$$A^{†}\left(t\right)A\left(t\right)Y\left(t\right)C\left(t\right)C^{†}\left(t\right)=A^{†}\left(t\right)B\left(t\right)C^{†}\left(t\right),\{A,B,C\}\in {\mathcal{D}}_G,$$

(42)

where the desired solution is $Y\left(t\right)\in {\mathbb{R}}^{n\times k}$.

The time derivative of $E_N\left(t\right)$ is given as

$${\dot{E}}_N\left(t\right)={\dot{A}}^{†}(AYC-B)C^{†}+A^{†}(\dot{A}YC+A\dot{Y}C+AY\dot{C}-\dot{B})C^{†}+A^{†}(AYC-B){\dot{C}}^{†}.$$

(43)

The time derivative of the pseudoinverse of $P\left(t\right)\in {\mathbb{R}}^{m\times n}$ is equal to

$${\dot{P}}^{†}=-P^{†}\dot{P}{P}^{†}+P^{†}{\left(P^{†}\right)}^{\mathrm{T}}{\dot{P}}^{\mathrm{T}}(I-P{P}^{†})+(I-P^{†}P){\dot{P}}^{\mathrm{T}}{\left(P^{†}\right)}^{\mathrm{T}}{P}^{†}.$$

(44)

Then, considering the linear ZNN design (2), it is obtained

$$\begin{array}{l}{\dot{A}}^{†}(AYC-B)C^{†}+A^{†}(\dot{A}YC+A\dot{Y}C+AY\dot{C}-\dot{B})C^{†}+A^{†}(AYC-B){\dot{C}}^{†}\\ =-\lambda A^{†}(AYC-B)C^{†},\{A,B,C\}\in {\mathcal{D}}_G,\end{array}$$

(45)

or equivalently

$$\begin{array}{ll}{A}^{†}A\dot{Y}C{C}^{†}=& -\lambda A^{†}(AYC-B)C^{†}-{\dot{A}}^{†}(AYC-B)C^{†}-A^{†}(\dot{A}YC+AY\dot{C}-\dot{B})C^{†}\\ & -A^{†}(AYC-B){\dot{C}}^{†},\{A,B,C\}\in {\mathcal{D}}_G,\end{array}$$

(46)

Applying the vectorization and the Kronecker product, the (46) dynamics is transformed into

$$\begin{gathered} \begin{array}{l}\left(\left(C{C}^{†}\right)\otimes \left(A^{†}A\right)\right)\mathrm{vec}\left(\dot{Y}\right)\\ =\mathrm{vec}\left(-\lambda A^{†}(AYC-B)C^{†}-{\dot{A}}^{†}(AYC-B)C^{†}-A^{†}(\dot{A}YC+AY\dot{C}-\dot{B})C^{†}-A^{†}(AYC-B){\dot{C}}^{†}\right), \\ \\ \{A,B,C\}\in {\mathcal{D}}_G,\end{array} \end{gathered}$$

(47)

and setting

$$\begin{array}{cc}\hfill W\left(t\right)=& (\left(C{C}^{†}\right)\otimes \left(A^{†}A\right))\hfill \\ r\left(t\right)=& \mathrm{vec}\left(-\lambda A^{†}(AYC-B)C^{†}-{\dot{A}}^{†}(AYC-B)C^{†}-A^{†}(\dot{A}YC+AY\dot{C}-\dot{B})C^{†}-A^{†}(AYC-B){\dot{C}}^{†}\right)\\ \hfill \dot{\mathbf{y}}\left(t\right)=& \mathrm{vec}\left(\dot{Y}\left(t\right)\right),\mathbf{y}\left(t\right)=\mathrm{vec}\left(Y\left(t\right)\right),\hfill \end{array}$$

(48)

the subsequent ZNN design is generated:

$$W\left(t\right)\dot{\mathbf{y}}\left(t\right)=r\left(t\right).$$

(49)

The mass matrix $W\left(t\right)$ in (49) is singular under the rank conditions $\mathrm{rank}\left(A\right)<min\{m,n\}$ or $\mathrm{rank}\left(C\right)<min\{k,h\}$. The regularization principle leads to the regular mass matrix

$$M\left(t\right)=\left\{\begin{array}{cc}W\left(t\right),\hfill & \mathrm{rank}\left(A\right)=min\{m,n\}\&\mathrm{rank}\left(C\right)=min\{k,h\}\hfill \\ W\left(t\right)+\beta I_{kn},\hfill & \mathrm{rank}\left(A\right)<min\{m,n\}\mathrm{or}\mathrm{rank}\left(C\right)<min\{k,h\}\hfill \end{array}\right.$$

(50)

such that $\beta \ge 0$ defines the ridge regression parameter. Accordingly, the ZNN dynamical system

$$M\left(t\right)\dot{\mathbf{y}}\left(t\right)=r\left(t\right)$$

(51)

is used instead of (49), where $M\left(t\right)$ is the nonsingular mass matrix. The differential system (51) is termed NZNNV1, and it is solvable using ode Matlab solvers. Theorem 3 proves the exponential convergence rate of the NZNNV1 evolution (51) to the accurate TV solution of TV GLMEs (42).

Theorem 3.

Let $A\left(t\right)\in {\mathbb{R}}^{m\times n}$, $C\left(t\right)\in {\mathbb{R}}^{k\times h}$ and $B\left(t\right)\in {\mathbb{R}}^{m\times h}$ be differentiable. If the condition (16) is satisfied, the NZNNV1 model (51), starting form any initial value $\mathbf{y}\left(0\right)$, exponentially converges to one element from the set ${\mathsf{\Theta }}_Q$ defined in (18), which solves the TV GLME (42).

Proof. 

To acquire the solution $\mathbf{y}\left(t\right)$ to (42), the Zhangian matrix is defined as in (41), in conjunction with the initiated ZNN dynamics. The model (45) is obtained by adopting the linear design for zeroing (41), which produces the NZNN design. From [6] (Theorem 1), restated in Lemma 3, each error matrix equation in the error matrix equation group (45) converges to the accurate solution as $t\to \infty$. According to Theorem 1(c), the matrix equation $E_N\left(t\right)=0$ is always solvable and its solutions are included in ${\mathsf{\Theta }}_Q$. Consequently, the solution of (45) converges to the solution of TV GLMEs (42) as $t\to \infty$. The derivation procedure of (51) confirms that it is an equivalent form of (45).    □

3.4. Newton ZNN Model (Version 2)

This subsection presents the second version of the Newton ZNN model for solving the TV GLMEs of (42). The Newton ZNN version 2 (NZNNV2) model includes two error functions. More precisely, the error function (41) is converted as follows:

$$E_1\left(t\right)=X\left(t\right)(A\left(t\right)Y\left(t\right)C\left(t\right)-B\left(t\right))Z\left(t\right),\{A,B,C\}\in {\mathcal{D}}_G.$$

(52)

The matrix $X\left(t\right)\in {\mathbb{R}}^{n\times m}$ in (52) is defined as the zero of the error function for finding the pseudoinverse of an arbitrary TV matrix $A\left(t\right)$ (see [38])

$$E_2\left(t\right)=A^{\mathrm{T}}\left(t\right)(I_m-A\left(t\right)X\left(t\right)),m\ge n\ge \mathrm{rank}\left(A\left(t\right)\right),$$

(53)

while $Z\left(t\right)\in {\mathbb{R}}^{h\times k}$ is the zero of the error function for finding the pseudoinverse of an arbitrary TV matrix $C\left(t\right)$

$$E_3\left(t\right)=(I_h-Z\left(t\right)C\left(t\right))C^{\mathrm{T}}\left(t\right),h\ge k\ge \mathrm{rank}\left(C\left(t\right)\right).$$

(54)

The derivative of (52) is

$$\begin{gathered} {\dot{E}}_1\left(t\right)=\dot{X}(AY-B)Z+X(\dot{A}YC+A\dot{Y}C+AY\dot{C}-\dot{B})Z+X(AYC-B)\dot{Z}, \\ \{A,B,C\}\in {\mathcal{D}}_G, \end{gathered}$$

(55)

while the derivative of (53) is

$${\dot{E}}_2\left(t\right)={\dot{A}}^{\mathrm{T}}(I_m-AX)-A^{\mathrm{T}}\dot{A}X-A^{\mathrm{T}}A\dot{X},m\ge n\ge \mathrm{rank}\left(A\right),$$

(56)

and

$${\dot{E}}_3\left(t\right)=(I_h-ZC){\dot{C}}^{\mathrm{T}}-\dot{Z}C{C}^{\mathrm{T}}-Z\dot{C}{C}^{\mathrm{T}},h\ge k\ge \mathrm{rank}\left(C\right),$$

(57)

Then, considering the linear ZNN design (2), it can be obtained:

$$\left\{\begin{array}{c}\dot{X}(AY-B)Z+X(\dot{A}YC+A\dot{Y}C+AY\dot{C}-\dot{B})Z+X(AYC-B)\dot{Z}=-\lambda X(AYC-B)Z,\hfill \\ {\dot{A}}^{\mathrm{T}}(I_m-AX)-A^{\mathrm{T}}\dot{A}X-A^{\mathrm{T}}A\dot{X}=-\lambda A^{\mathrm{T}}(I_m-AX),\hfill \\ (I_h-ZC){\dot{C}}^{\mathrm{T}}-\dot{Z}C{C}^{\mathrm{T}}-Z\dot{C}{C}^{\mathrm{T}}=-\lambda (I_h-ZC)C^{\mathrm{T}}\hfill \end{array}\right.$$

(58)

or in equivalent form

$$\left\{\begin{array}{c}XA\dot{Y}CZ+\dot{X}(AY-B)Z+X(AYC-B)\dot{Z}=-\lambda X(AYC-B)Z-X(\dot{A}YC+AY\dot{C}-\dot{B})Z,\hfill \\ -A^{\mathrm{T}}A\dot{X}=-\lambda A^{\mathrm{T}}(I_m-AX)-{\dot{A}}^{\mathrm{T}}(I_m-AX)+A^{\mathrm{T}}\dot{A}X,\hfill \\ -\dot{Z}C{C}^{\mathrm{T}}=-\lambda (I_h-ZC)C^{\mathrm{T}}-(I_h-ZC){\dot{C}}^{\mathrm{T}}+Z\dot{C}{C}^{\mathrm{T}}.\hfill \end{array}\right.$$

(59)

The dynamics (59) are modified using vectorization and the Kronecker product as follows:

$$\left\{\begin{array}{c}{\left(CZ\right)}^{\mathrm{T}}\otimes \left(XA\right)\mathrm{vec}\left(\dot{Y}\right)+{\left((AY-B)Z\right)}^{\mathrm{T}}\otimes I_n\mathrm{vec}\left(\dot{X}\right)+I_k\otimes \left(X(AYC-B)\right)\mathrm{vec}\left(\dot{Z}\right)\hfill \\ =\mathrm{vec}\left(-\lambda X(AYC-B)Z-X(\dot{A}YC+AY\dot{C}-\dot{B})Z\right),\hfill \\ (-I_m\otimes A^{\mathrm{T}}A)\mathrm{vec}\left(\dot{X}\right)=\mathrm{vec}\left(-\lambda A^{\mathrm{T}}(I_m-AX)-{\dot{A}}^{\mathrm{T}}(I_m-AX)+A^{\mathrm{T}}\dot{A}X\right),\hfill \\ (-C{C}^{\mathrm{T}}\otimes I_h)\mathrm{vec}\left(\dot{Z}\right)=\mathrm{vec}\left(-\lambda (I_h-ZC)C^{\mathrm{T}}-(I_h-ZC){\dot{C}}^{\mathrm{T}}+Z\dot{C}{C}^{\mathrm{T}}\right).\hfill \end{array}\right.$$

(60)

Setting

$$\begin{array}{l}{w}_1\left(t\right)={\left(CZ\right)}^{\mathrm{T}}\otimes \left(XA\right)\\ w_2\left(t\right)={\left((AY-B)Z\right)}^{\mathrm{T}}\otimes I_n,\\ w_3\left(t\right)=I_k\otimes \left(X(AYC-B)\right),\\ w_4\left(t\right)=-I_m\otimes A^{\mathrm{T}}A,\\ w_5\left(t\right)=-C{C}^{\mathrm{T}}\otimes I_h,\\ r_1\left(t\right)=\mathrm{vec}\left(-\lambda X(AYC-B)Z-X(\dot{A}YC+AY\dot{C}-\dot{B})Z\right)\\ r_2\left(t\right)=\mathrm{vec}\left(-\lambda A^{\mathrm{T}}(I_m-AX)-{\dot{A}}^{\mathrm{T}}(I_m-AX)+A^{\mathrm{T}}\dot{A}X\right)\\ r_3\left(t\right)=\mathrm{vec}\left(-\lambda (I_h-ZC)C^{\mathrm{T}}-(I_h-ZC){\dot{C}}^{\mathrm{T}}+Z\dot{C}{C}^{\mathrm{T}}\right)\\ W\left(t\right)=\left[\begin{array}{ccc}{w}_1& w_2& w_3\\ O_{mn,kn}& w_4& O_{mn,kh}\\ O_{kh,kn}& O_{kh,mn}& w_5\end{array}\right],r\left(t\right)=\left[\begin{array}{c}{r}_1\\ r_2\\ r_3\end{array}\right],\dot{\mathbf{y}}\left(t\right)=\left[\begin{array}{c}\mathrm{vec}\left(\dot{Y}\right)\\ \mathrm{vec}\left(\dot{X}\right)\\ \mathrm{vec}\left(\dot{Z}\right)\end{array}\right],\mathbf{y}=\left[\begin{array}{c}\mathrm{vec}\left(Y\right)\\ \mathrm{vec}\left(X\right)\\ \mathrm{vec}\left(Z\right)\end{array}\right],\end{array}$$

(61)

the following dynamical system is generated

$$W\left(t\right)\dot{\mathbf{y}}\left(t\right)=r\left(t\right),$$

(62)

with the mass matrix $W\left(t\right)$. To solve the problem with singularity of $W\left(t\right)$ in the case $\mathrm{rank}\left(A\right)<min\{m,n\}$ or $\mathrm{rank}\left(C\left(t\right)\right)<min\{k,h\}$, the following nonsingular mass matrix is used:

$$M\left(t\right)=\left\{\begin{array}{cc}W\left(t\right),\hfill & \mathrm{rank}\left(A\right)=min\{m,n\}\&\mathrm{rank}\left(C\right)=min\{k,h\}\hfill \\ W\left(t\right)+\beta I_{mnkn},\hfill & \mathrm{rank}\left(A\right)<min\{m,n\}\mathrm{or}\mathrm{rank}\left(C\right)<min\{k,h\},\hfill \end{array}\right.$$

(63)

wherein $\beta \ge 0$. Then the following ZNN model is used instead of (62):

$$M\left(t\right)\dot{\mathbf{y}}\left(t\right)=r\left(t\right),$$

(64)

where $M\left(t\right)$ is a nonsingular mass matrix. The NZNNV2 design (64) is solvable with an ode solver available in Matlab. The global and exponential convergence rate of the NZNNV2 flow (64) to the exact TV solution of (42) is verified in Theorem 4.

Theorem 4.

Let $A\left(t\right)\in {\mathbb{R}}^{m\times n}$, $C\left(t\right)\in {\mathbb{R}}^{k\times h}$ and $B\left(t\right)\in {\mathbb{R}}^{m\times h}$ be differentiable. If the condition (16) is satisfied, the NZNNV2 model (64) initialized by an arbitrary initial value $\mathbf{y}\left(0\right)$, exponentially converges to the exact solution of the TV GLMEs (42) in the form (18).

Proof. 

Similar to the proof of Theorem 3.    □

3.5. Newton ZNN Model (Version 3)

This subsection presents the third version of the Newton ZNN model for solving the TV GLMEs of (65). The Newton approach considers the smooth arbitrary TV matrices $A\left(t\right)\in {\mathbb{R}}^{m\times n}$, $C\left(t\right)\in {\mathbb{R}}^{k\times h}$ and $B\left(t\right)\in {\mathbb{R}}^{m\times h}$ for solving the following GLMEs:

$$\begin{gathered} {\left(A^{\mathrm{T}}A+\gamma I_n\right)}^{-1}{A}^{\mathrm{T}}AYC{C}^{\mathrm{T}}{\left(C{C}^{\mathrm{T}}+\gamma I_k\right)}^{-1}={\left(A^{\mathrm{T}}A+\gamma I_n\right)}^{-1}{A}^{\mathrm{T}}B{C}^{\mathrm{T}}{\left(C{C}^{\mathrm{T}}+\gamma I_k\right)}^{-1}, \\ \{A,B,C\}\in {\mathcal{D}}_G, \end{gathered}$$

(65)

where $Y\in {\mathbb{R}}^{n\times k}$ is the desired solution. Note that $\gamma \ge 0$ denotes the regularization parameter.

Based on the aforementioned, the error function in the Newton ZNN version 3 (NZNNV3) model will be defined by:

$$E_N\left(t\right)={\left(A^{\mathrm{T}}A+\gamma I_n\right)}^{-1}{A}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{\left(C{C}^{\mathrm{T}}+\gamma I_k\right)}^{-1},\{A,B,C\}\in {\mathcal{D}}_G,$$

(66)

Given that the time derivative of the inverse of $P\left(t\right)$ is

$${\dot{P}}^{-1}\left(t\right)=-P^{-1}\dot{P}{P}^{-1},$$

(67)

and setting

$$S_1\left(t\right)={(A^{\mathrm{T}}A+\gamma I_n)}^{-1},m\ge n\ge \mathrm{rank}\left(A\right),$$

(68)

$$S_2\left(t\right)={(C{C}^{\mathrm{T}}+\gamma I_k)}^{-1},h\ge k\ge \mathrm{rank}\left(C\right),$$

(69)

with

$${\dot{S}}_1\left(t\right)=-{\left(A^{\mathrm{T}}A+\gamma I_n\right)}^{-1}({\dot{A}}^{\mathrm{T}}A+A^{\mathrm{T}}\dot{A}){\left(A^{\mathrm{T}}A+\gamma I_n\right)}^{-1},m\ge n\ge \mathrm{rank}\left(A\right)$$

(70)

and

$${\dot{S}}_2\left(t\right)=-{\left(C{C}^{\mathrm{T}}+\gamma I_k\right)}^{-1}(\dot{C}{C}^{\mathrm{T}}+C{\dot{C}}^{\mathrm{T}}){\left(C{C}^{\mathrm{T}}+\gamma I_k\right)}^{-1},h\ge k\ge \mathrm{rank}\left(C\right),$$

(71)

the time derivative of (41) is given by

$$\begin{array}{ll}{\dot{E}}_N\left(t\right)=& {\dot{S}}_1{A}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{S}_2+S_1{\dot{A}}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{S}_2\\ & +S_1{A}^{\mathrm{T}}(\dot{A}YC+A\dot{Y}C+AY\dot{C}-\dot{B})C^{\mathrm{T}}{S}_2\\ & +S_1{A}^{\mathrm{T}}(AYC-B){\dot{C}}^{\mathrm{T}}{S}_2\\ & +S_1{A}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{\dot{S}}_2,\{A,B,C\}\in {\mathcal{D}}_G.\end{array}$$

(72)

Then, considering the linear ZNN dynamics (2), it can be obtained:

$$\begin{array}{ll}{\dot{S}}_1{A}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{S}_2& +S_1{\dot{A}}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{S}_2\\ & +S_1{A}^{\mathrm{T}}(\dot{A}YC+A\dot{Y}C+AY\dot{C}-\dot{B})C^{\mathrm{T}}{S}_2\\ & +S_1{A}^{\mathrm{T}}(AYC-B){\dot{C}}^{\mathrm{T}}{S}_2+S_1{A}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{\dot{S}}_2\\ & =-\lambda S_1{A}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{S}_2,\end{array}$$

(73)

or equivalently

$$\begin{array}{ll}{S}_1{A}^{\mathrm{T}}A\dot{Y}C{C}^{\mathrm{T}}{S}_2=& -\lambda S_1{A}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{S}_2\\ & -{\dot{S}}_1{A}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{S}_2-S_1{\dot{A}}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{S}_2\\ & -S_1{A}^{\mathrm{T}}(\dot{A}YC+AY\dot{C}-\dot{B})C^{\mathrm{T}}{S}_2-S_1{A}^{\mathrm{T}}(AYC-B){\dot{C}}^{\mathrm{T}}{S}_2\\ & -S_1{A}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{\dot{S}}_2,\end{array}$$

(74)

The vectorization and the Kronecker product transform (46) into the equivalent form

$$\begin{array}{l}({\left(C{C}^{\mathrm{T}}{S}_2\right)}^{\mathrm{T}}\otimes \left(S_1{A}^{\mathrm{T}}A\right))\mathrm{vec}\left(\dot{Y}\right)=\mathrm{vec}(-\lambda S_1{A}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{S}_2\\ -{\dot{S}}_1{A}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{S}_2-S_1{\dot{A}}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{S}_2\\ -S_1{A}^{\mathrm{T}}(\dot{A}YC+AY\dot{C}-\dot{B})C^{\mathrm{T}}{S}_2\\ -S_1{A}^{\mathrm{T}}(AYC-B){\dot{C}}^{\mathrm{T}}{S}_2-S_1{A}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{\dot{S}}_2).\end{array}$$

(75)

Then setting

$$\begin{array}{l}W\left(t\right)=({\left(C{C}^{\mathrm{T}}{S}_2\right)}^{\mathrm{T}}\otimes \left(S_1{A}^{\mathrm{T}}A\right))\\ \begin{array}{ll}r\left(t\right)=\mathrm{vec}(& -\lambda S_1{A}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{S}_2\\ & -{\dot{S}}_1{A}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{S}_2-S_1{\dot{A}}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{S}_2\\ & -S_1{A}^{\mathrm{T}}(\dot{A}YC+AY\dot{C}-\dot{B})C^{\mathrm{T}}{S}_2\\ & -S_1{A}^{\mathrm{T}}(AYC-B){\dot{C}}^{\mathrm{T}}{S}_2-S_1{A}^{\mathrm{T}}(AYC-B)C^{\mathrm{T}}{\dot{S}}_2)\end{array}\\ \dot{\mathbf{y}}\left(t\right)=\mathrm{vec}\left(\dot{Y}\left(t\right)\right),\mathbf{y}\left(t\right)=\mathrm{vec}\left(Y\left(t\right)\right),\end{array}$$

(76)

the following system of differential equations is obtained:

$$W\left(t\right)\dot{\mathbf{y}}\left(t\right)=r\left(t\right).$$

(77)

Since $W\left(t\right)$ is singular in the case $\mathrm{rank}\left(A\right)<min\{m,n\}$ or $\mathrm{rank}\left(C\right)<min\{k,h\}$, the utilization of the Tikhonov principle leads to the invertible mass matrix

$$M\left(t\right)=\left\{\begin{array}{cc}W\left(t\right),\hfill & \mathrm{rank}\left(A\right)=min\{m,n\}\&\mathrm{rank}\left(C\right)=min\{k,h\}\hfill \\ W\left(t\right)+\beta I_{kn},\hfill & \mathrm{rank}\left(A\right)<min\{m,n\}\mathrm{or}\mathrm{rank}\left(C\right)<min\{k,h\}\hfill \end{array}\right.$$

(78)

in which it is $\beta \ge 0$. As a result, the following dynamical evolution is used instead of (77):

$$M\left(t\right)\dot{\mathbf{y}}\left(t\right)=r\left(t\right),$$

(79)

where $M\left(t\right)$ is a nonsingular mass matrix. The ZNN flow (79) will be denoted as NZNNV3, and it is solvable using one of the ode Matlab solvers.

Theorem 5.

Let $A\left(t\right)\in {\mathbb{R}}^{m\times n}$, $C\left(t\right)\in {\mathbb{R}}^{k\times h}$ and $B\left(t\right)\in {\mathbb{R}}^{m\times h}$ be differentiable. If the condition (16) is satisfied, the NZNNV3 model (79), starting from arbitrary initial state $\mathbf{y}\left(0\right)$, exponentially converges to the theoretical TV solution of the TV GLMEs (65) in the form (18).

Proof. 

Similar to the proof of Theorem 3.    □

3.6. Direct ZNN Model

The direct approach considers the smooth arbitrary TV matrices $A\left(t\right)\in {\mathbb{R}}^{m\times n}$, $C\left(t\right)\in {\mathbb{R}}^{k\times h}$ and $B\left(t\right)\in {\mathbb{R}}^{m\times h}$ for solving the GLME

$$Y\left(t\right)=A^{†}\left(t\right)B\left(t\right)C^{†}\left(t\right),\{A,B,C\}\in {\mathcal{D}}_G,$$

(80)

where the desired solution is $Y\left(t\right)\in {\mathbb{R}}^{n\times k}$. The direct approach always calculates the solution produced directly by the pseudoinverse of $A\left(t\right)$ and $C\left(t\right)$. As a result, the direct ZNN (DZNN) model includes three error functions. The first error function in the DZNN model is defined by

$$E_1\left(t\right)=Y\left(t\right)-X\left(t\right)B\left(t\right)Z\left(t\right),\{A,B,C\}\in {\mathcal{D}}_G,$$

(81)

wherein $X\left(t\right)\in {\mathbb{R}}^{n\times m}$ is the desired zero of the second error function for calculating $A^{†}\left(t\right)$ (see [38]):

$$E_2\left(t\right)=A^{\mathrm{T}}\left(t\right)(I_m-A\left(t\right)X\left(t\right)),m\ge n\ge \mathrm{rank}\left(A\left(t\right)\right),$$

(82)

and $Z\left(t\right)\in {\mathbb{R}}^{h\times k}$ is the zero of the third error function for finding $C^{†}\left(t\right)$:

$$E_3\left(t\right)=(I_h-Z\left(t\right)C\left(t\right))C^{\mathrm{T}}\left(t\right),h\ge k\ge \mathrm{rank}\left(C\left(t\right)\right).$$

(83)

The time derivative of (81) is defined by

$${\dot{E}}_1\left(t\right)=\dot{Y}-\dot{X}BZ-X\dot{B}Z-XB\dot{Z},\{A,B,C\}\in {\mathcal{D}}_G,$$

(84)

the time derivative of (82) is

$${\dot{E}}_2\left(t\right)={\dot{A}}^{\mathrm{T}}(I_m-AX)-A^{\mathrm{T}}\dot{A}X-A^{\mathrm{T}}A\dot{X},m\ge n\ge \mathrm{rank}\left(A\right)$$

(85)

and the time derivative of (83) is given by

$${\dot{E}}_3\left(t\right)=(I_h-ZC){\dot{C}}^{\mathrm{T}}-\dot{Z}C{C}^{\mathrm{T}}-Z\dot{C}{C}^{\mathrm{T}},h\ge k\ge \mathrm{rank}\left(C\right).$$

(86)

Then, considering the ZNN model (2), it can be obtained

$$\left\{\begin{array}{c}\dot{Y}-\dot{X}BZ-X\dot{B}Z-XB\dot{Z}=-\lambda (Y-XBZ),\hfill \\ {\dot{A}}^{\mathrm{T}}(I_m-AX)-A^{\mathrm{T}}\dot{A}X-A^{\mathrm{T}}A\dot{X}=-\lambda A^{\mathrm{T}}(I_m-AX),\hfill \\ (I_h-ZC){\dot{C}}^{\mathrm{T}}-\dot{Z}C{C}^{\mathrm{T}}-Z\dot{C}{C}^{\mathrm{T}}=-\lambda (I_h-ZC)C^{\mathrm{T}},\hfill \end{array}\right.$$

(87)

or in equivalent form

$$\left\{\begin{array}{c}\dot{Y}-\dot{X}BZ-XB\dot{Z}=-\lambda (Y-XBZ)+X\dot{B}Z,\hfill \\ -A^{\mathrm{T}}A\dot{X}=-\lambda A^{\mathrm{T}}(I_m-AX)-{\dot{A}}^{\mathrm{T}}(I_m-AX)+A^{\mathrm{T}}\dot{A}X,\hfill \\ -\dot{Z}C{C}^{\mathrm{T}}=-\lambda (I_h-ZC)C^{\mathrm{T}}-(I_h-ZC){\dot{C}}^{\mathrm{T}}+Z\dot{C}{C}^{\mathrm{T}}.\hfill \end{array}\right.$$

(88)

Using the vectorization and the Kronecker product, (88) is transformed into

$$\left\{\begin{array}{c}{I}_{nk}\mathrm{vec}\left(\dot{Y}\right)-({\left(BZ\right)}^{\mathrm{T}}\otimes I_n)\mathrm{vec}\left(\dot{X}\right)-(I_k\otimes \left(XB\right))\mathrm{vec}\left(\dot{Z}\right)=\mathrm{vec}\left(-\lambda (Y-XBZ)+X\dot{B}Z\right)\hfill \\ (-I_m\otimes A^{\mathrm{T}}A)\mathrm{vec}\left(\dot{X}\right)=\mathrm{vec}\left(-\lambda A^{\mathrm{T}}(I_m-AX)-{\dot{A}}^{\mathrm{T}}(I_m-AX)+A^{\mathrm{T}}\dot{A}X\right)\hfill \\ (-C{C}^{\mathrm{T}}\otimes I_h)\mathrm{vec}\left(\dot{Z}\right)=\mathrm{vec}\left(-\lambda (I_h-ZC)C^{\mathrm{T}}-(I_h-ZC){\dot{C}}^{\mathrm{T}}+Z\dot{C}{C}^{\mathrm{T}}\right).\hfill \end{array}\right.$$

(89)

Setting

$$\begin{array}{l}{w}_1\left(t\right)=I_{nk}\\ w_2\left(t\right)=-{\left(BZ\right)}^{\mathrm{T}}\otimes I_n,\\ w_3\left(t\right)=-I_k\otimes \left(XB\right),\\ w_4\left(t\right)=-I_m\otimes A^{\mathrm{T}}A,\\ w_5\left(t\right)=-C{C}^{\mathrm{T}}\otimes I_h,\\ r_1\left(t\right)=\mathrm{vec}\left(-\lambda (Y-XBZ)+X\dot{B}Z\right)\\ r_2\left(t\right)=\mathrm{vec}\left(-\lambda A^{\mathrm{T}}(I_m-AX)-{\dot{A}}^{\mathrm{T}}(I_m-AX)+A^{\mathrm{T}}\dot{A}X\right)\\ r_3\left(t\right)=\mathrm{vec}\left(-\lambda (I_h-ZC)C^{\mathrm{T}}-(I_h-ZC){\dot{C}}^{\mathrm{T}}+Z\dot{C}{C}^{\mathrm{T}}\right)\\ W\left(t\right)=\left[\begin{array}{ccc}{w}_1& w_2& w_3\\ O_{mn,kn}& w_4& O_{mn,kh}\\ O_{kh,kn}& O_{kh,mn}& w_5\end{array}\right],r\left(t\right)=\left[\begin{array}{c}{r}_1\\ r_2\\ r_3\end{array}\right],\dot{\mathbf{y}}=\left[\begin{array}{c}\mathrm{vec}\left(\dot{Y}\right)\\ \mathrm{vec}\left(\dot{X}\right)\\ \mathrm{vec}\left(\dot{Z}\right)\end{array}\right],\mathbf{y}=\left[\begin{array}{c}\mathrm{vec}\left(Y\right)\\ \mathrm{vec}\left(X\right)\\ \mathrm{vec}\left(Z\right)\end{array}\right],\end{array}$$

(90)

the following differential system with the mass matrix $W\left(t\right)$ is obtained

$$W\left(t\right)\dot{\mathbf{y}}\left(t\right)=r\left(t\right).$$

(91)

Since $W\left(t\right)$ is singular when $\mathrm{rank}\left(A\left(t\right)\right)<min\{m,n\}$ or $\mathrm{rank}\left(C\left(t\right)\right)<min\{k,h\}$, the following regular mass matrix is used:

$$M\left(t\right)=\left\{\begin{array}{cc}W\left(t\right),\hfill & \mathrm{rank}\left(A\left(t\right)\right)=min\{m,n\}\&\mathrm{rank}\left(C\left(t\right)\right)=min\{k,h\}\hfill \\ W\left(t\right)+\beta I_{mnkn},\hfill & \mathrm{rank}\left(A\left(t\right)\right)<min\{m,n\}\mathrm{or}\mathrm{rank}\left(C\left(t\right)\right)<min\{k,h\}\hfill \end{array}\right.$$

(92)

such that $\beta \ge 0$. As a result, the following ZNN model is used instead of (91):

$$M\left(t\right)\dot{\mathbf{y}}\left(t\right)=r\left(t\right),$$

(93)

where $M\left(t\right)$ is a nonsingular mass matrix. The system (93) is termed as DZNN model, and it is solvable with an ode Matlab solver.

Theorem 6.

Let $A\left(t\right)\in {\mathbb{R}}^{m\times n}$, $C\left(t\right)\in {\mathbb{R}}^{k\times h}$ and $B\left(t\right)\in {\mathbb{R}}^{m\times h}$ be differentiable. If the condition (16) is satisfied, the DZNN model (93) starting from any initial value $\mathbf{y}\left(0\right)$, converges exponentially to the theoretical TV solution of the TV GLMEs (80) in the form (18).

Proof. 

Analogous to the proof of Theorem 3. □

4. Simulation Examples

This section compares performances of the TZNN model (29), the GZNN design (40), the NZNNV1 model (51), the NZNNV2 model (64), the NZNNV3 model (79), and the DZNN model (93) on six numerical examples (NE), involving square or rectangular, as well as singular or nonsingular input TV matrices. The diagram in Figure 1 is presented to understand better the solutions produced by these six models.

This diagram shows how the GZNN, NZNNV1, NZNNV2, and NZNNV3 models generate various solutions for different initial conditions. In contrast, the TZNN and DZNN models generate just the pseudoinverse solution of the TV GLME (1) for any initial conditions.

As preliminaries, a few parameters and symbols must be defined, as well as some additional information for the subsequent NEs. The considered time interval is $[0,10]$ in all NE, while the ZNN scaling parameter is assigned to $\lambda =10$, and the regularization parameter is $\beta =1e-8$. The initial condition (IC1) in all NEs is

$$Y\left(0\right)=A^{\mathrm{T}}\left(0\right)I_{m,h}{C}^{\mathrm{T}}\left(0\right)$$

for all models, with $X\left(0\right)=A^{\mathrm{T}}\left(0\right)$ and $Z\left(0\right)=C^{\mathrm{T}}\left(0\right)$ for the NZNNV2 model (64), the NZNNV3 model (79), and the DZNN model (93). The second initial condition (IC2) for the last three NEs has been set as

$$Y\left(0\right)=(1:k)\odot {\mathbf{1}}_{n\times k},$$

with $X\left(0\right)=A^{\mathrm{T}}\left(0\right)$ and $Z\left(0\right)=C^{\mathrm{T}}\left(0\right)$ for the NZNNV2 model (64), the NZNNV3 model (79) and the DZNN model (93), and $\gamma =1e-1$ for the NZNNV3 model (79). In the figures legends, the notation $\mathrm{TZNN}$, $\mathrm{GZNN}$, $\mathrm{NZNNV}1$, $\mathrm{NZNNV}2$, $\mathrm{NZNNV}3$ and $\mathrm{DZNN}$ denote the solutions or errors produced by the corresponding models solutions, while the notations $Y_1^{*}\left(t\right),Y_2^{*}\left(t\right),Y^{*}\left(t\right)$ refer to $Y_1^{*}\left(t\right)=A^{†}B{C}^{†}$ and $Y_2^{*}\left(t\right)={\tilde{Y}}_{Y\left(0\right)}\left(t\right)$, and $Y^{*}\left(t\right)=Y_1^{*}=Y_2^{*}$.

4.1. Example 1

This NE is about a TV GLME that considers the following square matrices of dimensions $4\times 4$:

$$\begin{array}{l}A\left(t\right)=\left[\begin{array}{cccc}3+cos\left(t\right)& 1+1/2sin\left(t\right)& \cdots & 1+1/nsin\left(t\right)\\ 1+1/2sin\left(t\right)& 3+cos\left(t\right)& \cdots & 1+1/(n-1)sin\left(t\right)\\ \cdots & \cdots & \cdots & \cdots \\ 1+1/msin\left(t\right)& 1+1/(m-1)sin\left(t\right)& \cdots & 3+cos\left(t\right)\end{array}\right],\\ C\left(t\right)=\left[\begin{array}{cc}5+cos\left(t\right)& -3+sin\left(t\right)\\ 3+cos\left(t\right)& 2+sin\left(t\right)\end{array}\right],\\ B\left(t\right)=\left[\begin{array}{cc}7+cos\left(t\right)& 6+sin\left(2t\right)\\ 5+sin\left(t\right)& -4-sin\left(t\right)\\ 1+cos\left(t\right)& -6+cos\left(t\right)\\ 3-cos\left(t\right)& -2+cos\left(t\right)\end{array}\right],\end{array}$$

such that $\mathrm{rank}\left(A\left(t\right)\right)=4$. The input matrices $A\left(t\right),C\left(t\right)$ and $B\left(t\right)$ are nonsingular matrices.

4.2. Example 2

This NE is about a TV GLME that considers rectangular matrices

$$\begin{array}{l}A\left(t\right)={\left[\begin{array}{ccc}6-cos\left(\pi t\right)& -3-cos\left(\pi t\right)& 1-cos\left(\pi t\right)\\ 5-cos\left(\pi t\right)& -8-cos\left(\pi t\right)& 2-cos\left(\pi t\right)\end{array}\right]}^{\mathrm{T}},\\ C\left(t\right)=\left[\begin{array}{cccc}5+cos\left(t\right)& -3+sin\left(t\right)& -2+sin\left(t\right)& -1+sin\left(t\right)\\ 3+cos\left(t\right)& 2+sin\left(t\right)& 3+sin\left(t\right)& 4+sin\left(t\right)\\ 5+cos\left(t\right)& 7+sin\left(t\right)& 3+sin\left(t\right)& 2+sin\left(t\right)\end{array}\right],\\ B\left(t\right)=\left[\begin{array}{cccc}2A(:,1)& 3A(:,2)& -3A(:,2)& -2A(:,1)\end{array}\right],\end{array}$$

such that $\mathrm{rank}\left(A\left(t\right)\right)=2$. That is, the input $A\left(t\right)$ is a full-column rank $3\times 2$ matrix, and the input $C\left(t\right)$ is a full-row rank $3\times 4$ matrix.

4.3. Example 3

This NE is about a TV GLME that takes into account the rectangular matrices below:

$$\begin{array}{l}A\left(t\right)=\left[\begin{array}{cccc}3+cos\left(t\right)& 1+1/2sin\left(t\right)& \cdots & 1+1/nsin\left(t\right)\\ 1+1/2sin\left(t\right)& 3+cos\left(t\right)& \cdots & 1+1/(n-1)sin\left(t\right)\\ \cdots & \cdots & \cdots & \cdots \\ 1+1/msin\left(t\right)& 1+1/(m-1)sin\left(t\right)& \cdots & 3+cos\left(t\right)\end{array}\right],\\ C\left(t\right)=\left[\begin{array}{cccc}5+cos\left(t\right)& -3+sin\left(t\right)& -2+sin\left(t\right)& -1+sin\left(t\right)\\ 3+cos\left(t\right)& 2+sin\left(t\right)& 3+sin\left(t\right)& 4+sin\left(t\right)\end{array}\right],\\ B\left(t\right)=\left[\begin{array}{c}\mathrm{diag}(1:k)\\ {\mathbf{1}}_k\\ 2\times {\mathbf{1}}_{2,k}\end{array}\right]+sin\left(t\right),\end{array}$$

where $\mathrm{rank}\left(A\left(t\right)\right)=min\{m,n\}$ and $m=10$, $n=6$ and $k=4$. That is, the input matrix $A\left(t\right)$ is a full-column rank $10\times 6$ matrix, and the input $C\left(t\right)$ is a full-row rank $2\times 4$ matrix.

4.4. Example 4

This example concerns a TV GLME that takes into account the input matrices

$$\begin{array}{l}A\left(t\right)=\left[\begin{array}{c}1+sin\left(t\right)\\ 1+1/2sin\left(t\right)\\ 1+1/3sin\left(t\right)\\ 1+1/4sin\left(t\right)\end{array}\right]\odot {\mathbf{1}}_4,\\ C\left(t\right)=\left[\begin{array}{cc}5+cos\left(t\right)& -3+sin\left(t\right)\\ 5+cos\left(t\right)& -3+sin\left(t\right)\end{array}\right],\\ B\left(t\right)=\left[\begin{array}{cc}3A(:,1)& 2A(:,4)\end{array}\right]\end{array}$$

such that $A\left(t\right)$ and $C\left(t\right)$ are singular and of dimensions $4\times 4$ and $2\times 2$, respectively, with $\mathrm{rank}\left(A\left(t\right)\right)=\mathrm{rank}\left(C\left(t\right)\right)=1$.

4.5. Example 5

This NE is about a TV GLME that takes the rectangular matrices below into account:

$$\begin{array}{l}A\left(t\right)={\left[\begin{array}{ccccc}6-cos\left(\pi t\right)& -5-cos\left(\pi t\right)& 1+sin\left(\pi t\right)& 2+sin\left(t\right)& -2+sin\left(t\right)\end{array}\right]}^{\mathrm{T}}\odot \left[\begin{array}{cc}1& 2\end{array}\right]\odot {\mathbf{1}}_{5,2},\\ C\left(t\right)=\left[\begin{array}{cccccc}5& -3& -2& -1& 2& -3\\ 8& -4& -2& 7& 2& -3\\ 5& 7& 3& 2& -3& 2\end{array}\right]+sin\left(t\right),\\ B\left(t\right)=\left[\begin{array}{ccc}5A(:,1)& 2A(:,2)& {\mathbf{1}}_{5\times 4}\end{array}\right]\end{array}$$

where $\mathrm{rank}\left(A\left(t\right)\right)=1$ and $\mathrm{rank}\left(C\left(t\right)\right)=1$. The input matrix $A\left(t\right)$ is a $5\times 2$ rank-deficient matrix, and $C\left(t\right)$ is a full-row rank $3\times 6$ matrix.

4.6. Example 6

This NE is about a TV GLME that takes the rectangular matrices below into consideration:

$$\begin{array}{l}A\left(t\right)=\left[\begin{array}{cccc}1+sin\left(t\right)& 1+1/2sin\left(t\right)& \cdots & 1+1/msin\left(t\right)\end{array}\right]\odot {\mathbf{1}}_{m\times n},\\ C\left(t\right)=\left[\begin{array}{cccc}5+cos\left(t\right)& -3+sin\left(t\right)& -2+sin\left(t\right)& -1+sin\left(t\right)\end{array}\right]{\mathbf{1}}_{k\times h},\\ B\left(t\right)=\left[\begin{array}{cccc}3A(:,1)& 2A(:,1)& 5A(:,1)& {\mathbf{1}}_m\end{array}\right],\end{array}$$

where $\mathrm{rank}\left(A\left(t\right)\right)=1$ with $m=8$ and $n=5$, and $\mathrm{rank}\left(C\left(t\right)\right)=1$ with $k=3$ and $h=4$. The input matrix $A\left(t\right)$ is rank-deficient and of size $8\times 5$, and $C\left(t\right)$ is a rank deficient $3\times 4$ matrix.

4.7. General Discussion

Since $A\left(t\right)$ and $C\left(t\right)$ in Example 1 are nonsingular, the solvability condition (16) is satisfied. As a consequence, the TV GLME (1) is solvable with the unique solution $Y_1^{*}\left(t\right)=A^{†}B{C}^{†}$. On the other hand, the solvability condition (16) for the matrix equation $E\left(t\right)=0$ is not satisfied in all another Examples 2–6. So, the equation $E\left(t\right)=0$ is not solvable in Examples 2–6. On the other hand, according to Theorem 1, the matrix equations $E_G\left(t\right)=0$ and $E_N\left(t\right)=0$ are solvable in all examples.

In this subsection, the performance of six proposed ZNN models for solving TV GLME (1) is investigated through six NEs. All the models generate the solution $Y_1^{*}\left(t\right)$, i.e., the pseudoinverse solution of the TV GLME (1), under the IC1 in all NEs, while the GZNN, NZNNV1, NZNNV2, and NZNNV3 models generate the solution $Y_2^{*}\left(t\right)$ under the IC2 in NEs Section 4.4, Section 4.5 and Section 4.6. More precisely, NEs Section 4.1, Section 4.2 and Section 4.3, respectively, deal with square nonsingular $A\left(t\right)$ and $C\left(t\right)$, a rectangular full column rank $A\left(t\right)$, and a rectangular full row rank $C\left(t\right)$. As a result, all the models generate the pseudoinverse solution of the TV GLME (1), i.e., $Y^{*}\left(t\right)=Y_1^{*}\left(t\right)=Y_2^{*}\left(t\right)$, for any initial condition. It is worth noting that the TZNN is only applicable in NE Section 4.1, which has square nonsingular input matrices $A\left(t\right)$ and $C\left(t\right)$. To substantiate the claim mentioned above, the TZNN has been used in NE Section 4.2, where Figure 2b demonstrates that the TZNN error matrix does not converge to zero. Furthermore, NE Section 4.4 deals with square singular $A\left(t\right)$ and $C\left(t\right)$, and NEs Section 4.5 and Section 4.6 deal with rectangular rank deficient $A\left(t\right)$ and $C\left(t\right)$. As a result, the GZNN, NZNNV1, NZNNV2, and NZNNV3 generate the solution $Y_1^{*}\left(t\right)$ under the IC1 and the solution $Y_2^{*}\left(t\right)$ under the IC2. It is worth noting that $Y_1^{*}\left(t\right)\ne Y_2^{*}\left(t\right)$ in NEs Section 4.4, Section 4.5 and Section 4.6 under the IC2.

Figure 2, Figure 3 and Figure 4 follow the next schedule: first rows in the figures display the convergence of the error functions involved in the tested models, i.e., (25) of TZNN, (33) of GZNN, (41) of NZNNV1, (52) and (53) of NZNNV2 (NZNNV2-E1 and NZNNV2-E2), (66) of NZNNV3, (81) and (82) of DZNN (DZNN-E1 and DZNN-E2) Frobenius norms; the second rows in figures exhibit the convergence behavior of the models on Error 1: ${∥Y\left(t\right)-Y_2^{*}\left(t\right)∥}_F$, where $Y\left(t\right)$ refers to solutions generated by considered models; the third rows in the figures display trajectories of generated solutions $Y\left(t\right)$; the fourth rows show the convergence of the models on Error 2: ${∥A\left(t\right)Y\left(t\right)C\left(t\right)-B\left(t\right)∥}_F$ in NE Section 4.1, and Error 2: ${∥A^{\mathrm{T}}\left(t\right)(A\left(t\right)Y\left(t\right)C\left(t\right)-B\left(t\right))C^{\mathrm{T}}\left(t\right)∥}_F$ in NEs Section 4.2, Section 4.3, Section 4.4, Section 4.5 and Section 4.6. Note also that Figure 2 contains results about NEs Section 4.1, Section 4.2 and Section 4.3, Figure 3 contains results about NEs Section 4.4, Section 4.5 and Section 4.6 under the IC1, and Figure 4 contains results about NEs Section 4.4, Section 4.5 and Section 4.6 under the IC2.

The following can be deduced from the NEs. The exponential convergence of the models that is proven in Theorems 2–6 can be observed in the first row figures. Particularly, Figure 2a–c present the exponential convergence of the models in NEs Section 4.1, Section 4.2 and Section 4.3, respectively, whereas Figure 3a–c and Figure 4a–c present the exponential convergence of the models in NEs Section 4.4, Section 4.5 and Section 4.6 under IC1 and IC2, respectively. More particularly, while all models begin with an initial value other than the optimal one, all NEs have received differentiable $A\left(t\right),B\left(t\right)$, and $C\left(t\right)$ matrices that meet the condition (16). In Figure 2a–c, Figure 3a–c and Figure 4a–c, the error functions convergence begins at $t=0$ in the range $[{10}^1,{10}^6]$, but it ends before $t=2$ with lowest values in the range $[{10}^{-10},{10}^{-1}]$. It is also crucial to note that when the value of the design parameter $\lambda$ is higher, the models will converge faster and the overall error will be lowered even further. Furthermore, all other figures behave in the same way due to the convergence tendency of the error functions. In other words, the graphs associated with the models in the remaining figures begin at $t=0$ in a very different value from the objective and reach it before $t=2$. As a result, it is evident that Theorems 2–6 are proven true.

In general, according to the first row figures, all the ZNN models converge to zero, where the NZNNV1 and NZNNV3 models have the fastest convergence rate, and the TZNN and DZNN have the second and third fastest, respectively. In contrast, the GZNN and NZNNV2 models have the slowest and almost similar convergence rates. According to the second row figures under IC1 and fourth row figures, the NZNNV1 and NZNNV3 models have the fastest convergence rate, the GZNN has the second fastest, and the NZNNV2 model has the slowest. The NZNNV2 and DZNN models have the lowest overall errors when $A\left(t\right)$ and $C\left(t\right)$ are rectangular, specifically in NEs Section 4.2 and Section 4.5. The second row figures under IC2 show that the NZNNV1, NZNNV2, NZNNV3, and GZNN models perform similarly to the second row figures under IC1. However, the DZNN model does not converge to zero, as expected. Finally, the third row figures demonstrate that under IC1, all models solutions match $Y^{*}\left(t\right)$, however under IC2, the DZNN model solutions match $Y_1^{*}\left(t\right)$, while the NZNNV1, NZNNV2, NZNNV3, and GZNN models solutions match $Y_2^{*}\left(t\right)$.

Graphs included in Figure 3a–c and Figure 4a–c, on the other hand, show that the IC1 and IC2 have no effect on the models’ convergence, proving the Theorems 2–6 claim that the models’ performance is unaffected by the initial condition.

To summarize, the TZNN is only applicable in the case where $m=n=\mathrm{rank}\left(A\left(t\right)\right)$ and $k=h=\mathrm{rank}\left(C\left(t\right)\right)$. It is important to mention that $A\left(t\right)$ becomes $C\left(t\right)$ and $C\left(t\right)$ becomes $A\left(t\right)$ when $n<m$ and $k<h$. As a result, the cases where $n<m$ and $k<h$ is disregarded in the NEs. The GZNN has a lower overall error than NZNNV1 and NZNNV3, although they have the same convergence speed. Even though the NZNNV2 has the slowest convergence speed, its overall error is between NZNNV1 and GZNN. Compared to GZNN, NZNNV1, and NZNNV3, the DZNN model’s convergence speed is slightly slower, but its overall error is the lowest. Remember that the DZNN model’s primary distinction from all other models is that, regardless of the initial conditions, it generates only a single solution based on the pseudoinversion. That is, all the models work excellently in solving the TV GLMEs (1).

5. Conclusions

The problem of solving TV GLMEs with random TV real input matrices is resolved in this paper by applying the ZNN design. Six ZNN models for calculating the online solution to general TV GLMEs are defined, analyzed, and compared, including TZNN, which is based on the traditional ZNN approach, GZNN which is based on the gradient ZNN approach, NZNNV1, NZNNV2, and NZNNV3 which are based on the Newton’s optimization method, and DZNN which is based on the ZNN approach for finding the direct solution produced with the use of the pseudoinverse. Six numerical examples involving square or rectangular, and singular or nonsingular matrices show that all models successfully solve TV GLMEs. However, their effectiveness varies and depends on the input matrix, while the NZNNV1 and NZNNV3 models converge to exact solutions faster than the other models.

There are a few prospective study areas that can be explored.

• The streams of NZNNV1, NZNNV2, NZNNV3, and DZNN models accelerated by appropriate nonlinear activations, as well as nonlinear NZNNV1, NZNNV2, NZNNV3, and DZNN model flows with a finite convergence, can all be investigated.
• Another research stream is to use carefully selected parameters defined in fuzzy environments. Such research will be a continuation of research presented in [25,26,38,39].
• The presented TZNN, GZNN, NZNNV1, NZNNV2, NZNNV3, and DZNN models are not noise-tolerant, so all noise types have a significant influence on the accuracy of the suggested ZNNs. Consequently, related future investigation can be oriented on adjusting the proposed dynamical systems to appropriate more general integration-enhanced and noise-tolerant ZNN classes.
• Numerous results concerning the solution of important equations of a special type have appeared in recent years. For example, there is a number of recent papers concerning the Sylvester equation [40,41] or related to Lyapunov equation [42,43,44]. Studying such equations can be one of the goals of our research.