Quaternionic Multilayer Perceptron with Local Analyticity

Abstract

A multi-layered perceptron type neural network is presented and analyzed in this paper. All neuronal parameters such as input, output, action potential and connection weight are encoded by quaternions, which are a class of hypercomplex number system. Local analytic condition is imposed on the activation function in updating neurons’ states in order to construct learning algorithm for this network. An error back-propagation algorithm is introduced for modifying the connection weights of the network.

1. Introduction

Processing multi-dimensional data is an important problem for artificial neural networks. A single neuron can take only one real value as its input, thus a network should be configured so that several neurons are used for accepting multi-dimensional data. This type of configuration is sometimes unnatural in applications of artificial neural networks to engineering problem, such as processing of acoustic signals or coordinates in the plane. Thus, complex number systems have been utilized to represent two-dimensional data elements as a single entity. Application of complex numbers to neural networks have been extensively investigated, as summarized in the references [1,2,3].

Though complex values can treat two-dimensional data elements as a single entity, what we should treat data with more than two-dimension in artificial neural networks? Although this problem can of course be solved by applying several real-valued or complex-valued neurons, it would be useful to introduce a number system with higher dimensions, the so-called hypercomplex number systems.

Quaternion is a four-dimensional hypercomplex number system introduced by Hamilton [4,5]. This number system has been extensively employed in several fields, such as modern mathematics, physics, control of satellites, computer graphics, etc. [6,7,8]. One of the benefits provided by quaternions is that affine transformations of geometric figures in three-dimensional spaces, especially spatial rotations, can be represented compactly and efficiently. Applying quaternions to the field of neural networks has been recently explored in an effort to naturally represent high-dimensional information, such as color and three-dimensional body coordinates, by a quaternionic neuron, rather than complex-valued or real-valued neurons.

Thus, there has been a growing number of studies concerning the use of quaternions in neural networks. Multilayer perceptron (MLP) models have been developed in [9,10,11,12,13,14]. The use of quaternion in MLP models has been applied to several engineering problems such as control problems [10], color image compression [12], color night vision [15,16], and predictions for the output of chaos circuits and winds in three-dimensional space [13,14]. Other types of network models has also been explored, such as the computational ability of a single quaternionic neuron [17] and the existence condition of an energy function in continuous-time and continuous-state recurrent networks [18]. There are several types of quaternionic Hopfield-type networks with discrete-time driven, such as bipolar state [19,20], continuous state [21,22], multistate by phase representation [23,24]. Learning schemes for these networks have also been proposed in [25].

One of the difficulties in constructing neural networks in the quaternionic domain is about the introduction of suitable functions for the activation function in updating the neurons’ states. A typical type of activation function is the so-called “split” type function, in which a real-valued function is appliedto update each component of a quaternionic value [10]. Real-valued sigmoidal function and hyperbolic function, which are analytic (differentiable) functions, are often used for this purpose. However, as an activation function, the split-type quaternionic function is not appropriate due to lack of analyticity. Thus, it is necessary to define other types of differential functions in order to construct learning schemes such as the error back-propagation algorithm. The Cauthy-Riemann-Fueter (CRF) equation defines the analytic condition for the quaternionic functions, which corresponds to the Cauthy-Riemann equation for the complex-valued functions. The functions satisfying the CRF equation turn out to be only linear functionsor constants; therefore it is impossible to introduce non-linearity into the updates of the neurons’ states.

Recently, another class of analyticity for the quaternionic functions has been developed [26,27,28]. Called “local analyticity”, this analytic condition is derived at a quaternionic point with its local coordinate, rather than in a quaternionic space with a global coordinate. The derivation of local analytic condition shows that a quaternion in the local coordinate system is isomorphic to the complex number system and thus it can be treated as a complex value. A neural network with an activation function with local analyticity has been first proposed and analyzed in [13,14] in terms of MLP-type network. It is shown that the performance for this network is superior to the network with a split-type activation function through several applications. Another application of analytic activation function to quaternionic neural networks has been investigated in terms of Hopfield-type network [22]. The local analytic condition is constructed based on [28], and the stability conditions are derived in the case of quaternionic tanh function being used for an activation function is deduced in this network, in which the complex-valued tanh function used in [29] can be used as a quaternionic function.

This paper presents an MLP-type quaternionic neural network with locally analytic activation function. All variables in this network, such as input, output, action potential and connection weights, are encoded by quaternions. A learning scheme, a quaternionic equivalent of error back-propagation algorithm, is presented and theoretically explored. The derivation of the learning scheme in this papera dopts the Wirtinger calculus [30], which has been invented in the field of complex analysis, where a quaternionic value and its conjugate are treated independent of each other. This calculus enables the derivations to be more straightforward of its description, than by using the conventional description, i.e., Cartesian representation.

2. Quaternionic Algebra

2.1. Definition of Quaternion

Quaternions forma class of hypercomplex numbers consistingofareal number and three imaginary numbers-i, j, and k. Formally, a quaternion number is defined as a vector x in a four-dimensional vector space,

(1)

where x^(e), x⁽ⁱ⁾, x^(j), and x^(k) are real numbers. The division ring of quaternions, H, constitutes the four-dimensional vector space over the real numbers with bases 1, i, j, and k. Equation (1) can also be written using 4-tuple or 2-tuple notation as

(2)

where = { x⁽ⁱ⁾, x^(j), x^(k)}. In this representation, x^(e) is the scalar part of x, and forms the vector part. The quaternion conjugate is defined as

(3)

Quaternion bases satisfy the following identities,

(4)

(5)

known as the Hamilton rule. From these rules, it follows immediately that multiplication of quaternions is not commutative.

Next, we define the operations between quaternions p = (p^(e), ) = (p^(e), p⁽ⁱ⁾, p^(j), p^(k)) and q = (q^(e), ) = (q^(e), q⁽ⁱ⁾, q^(j), q^(k)). The addition and subtraction of quaternions are defined in a similar manner as for complex-valued numbers or vectors, i.e.,

(6)

(7)

The product of p and q is determined by Equation (5) as

(8)

where · and × denote the dot and cross products, respectively, between three-dimensional vectors and . The conjugate of the product is given as

(9)

The quaternion norm of x, denoted by |x|, is defined as

(10)

2.2. Quaternionic Analyticity

It is important to introduce an analytic function (or differentiable function) to serve as the activation function in the neural network. This section describes the required analyticity of the function in the quaternionic domain, in order to construct activation functions for quaternionic neural networks.

The condition for differentiability of the quaternionic function f is given by

(11)

The analytic condition for the quaternionic function, called the Cauchy-Riemann-Fueter (CRF) equation, yields:

(12)

This is an extension of the Cauchy-Riemann (CR) equations defined for the complex domain. However, only linear functions and constants satisfy the CRF equation [14,26,27].

An alternative approach to assure analyticity in the quaternionic domain has been explored in [26,27,28]. This approach is called local analyticity and is distinguished from the standard analyticity, i.e., global analyticity. In the following, we introduce local derivatives with Wirtinger representation and analytic conditions for quaternionic functions, with reference to [28].

A quaternion x can be alternatively represented as:

(13)

(14)

(15)

From the definition in Equation (15), we deduce that u² _x = −1. If u_x holds a commutative property against a difference of x, then the system with u_x can be regarded as locally isomorphic to the complex number system.

A quaternionic difference of x, denoted by dx =(dx^(e), dx⁽ⁱ⁾, dx^(j), dx^(k)), can be decomposed by using:

(16)

where,

Then, the following relations hold:

when we set dx_⊥ = 0, i.e., dx + u_xdxu_x = 0,which results in u_xdx = dxu_x. This leads to u_x × d =0, because u_x is a quaternion without a real part. Thus, u_x and d are parallel to each other. Then, d = u_xδ can be obtained, where δ is a real-valued constant. From Equation (14), it follows that

Then,

Considering = u_xr and d = u_xδ, we obtain

Hence, δ = dr is derived and d = u_xdr is obtained. dx is represented as .

Figure 1 shows a schematic coordinate system for defining a local complex plane. The component k is omitted (x^(k) = 0) in this figure due to difficulties in representing a four-dimensional vector space. In this example, for a given quaternion x, its unit vector u_x is defined in the i-j plane. Then, a complex plane is defined by spanning the components x^(e)(real axis) and x^(r) in the quaternionic space, and the analytic condition is constrained in this plane.

Figure 1. A schematic illustration of local complex plane in a quaternionic space, where the component k is omitted for simplicity.

Figure 1. A schematic illustration of local complex plane in a quaternionic space, where the component k is omitted for simplicity.

The local derivative operators are introduced, corresponding to the form of dx_‖, as follows:

where

with the properties

Note that the variables x and x^∗ turn out to be independent of each other. These derivative operators are quaternionic equivalents to the well-known Wirtinger derivative in the complex domain [30].

F (x + dx) can be expanded using the above-mentioned representations as

(17)

where,

When dx_⊥ = 0, the local derivative of F (x) is written as

and the local analytic condition for the function F (x) is given by

in the corresponding local complex plane. This result corresponds to the one presented in [27], where dx_⊥ =0 always holds.

Moreover, if F is a function with the two arguments, x and x^∗, it becomes:

(18)

with x and x^∗ being independent of each other. As a result, we can treat quaternionic functions in the same manner as complex-valued functions under the condition of local analyticity.

3. Quaternionic Multilayer Perceptron

3.1. Network Model

The structure of the network assumed in this paper is shown in Figure 2. This network is a so-called multilayer perceptron network with one hidden layer, and the parameters in the network are encoded by quaternionic values.

The numbers of neurons in the input, hidden and output layers are set to M, N and K, respectively. A set of quaternionic signals denoted by z is input to the neurons in the input layer of the network. The outputs of the neurons in the input layer are the same as input z’s. In the hidden layer, each neuron takes the weighted sum of the output signals from the input layer. The (connection) weight from the n-th neuron in the input layer to the m-th neuron in the hidden layer is denoted by v_nm. The output of the neuron in the hidden layer, denoted by x_n, is determined by

(19)

where g is a quaternionic activation function introducing non-linearity between the action potential and output in the neuron. This function satisfies the following condition:

(20)

Figure 2. The structure of the multilayer perceptron in this paper.

Figure 2. The structure of the multilayer perceptron in this paper.

Processing the neurons’ outputs in the output layer can be defined in the same manner as in the hidden layer. The output of the neuron in the output layer, y_k, is defined as

(21)

where the function h is a quaternionic activation function from the action potential to the output, and w_kn is the connection weight between the n-th neuron in the hidden layer and the k-th neuron in the output layer. The function h also satisfies

(22)

The connection weights should be modified by the so-called learning algorithms, in order to obtain the desired output signals with respect to the input signals. One of the learning algorithms for MLP-type networks is the error back-propagation (EBP) algorithm. The following section describes our derivation of this algorithm for the presented network.

3.2. Learning Algorithm

An EBP algorithm works so that the output error, calculated by the neurons’ outputs at the output layer and the desired output signals, is minimized. In the case of networks with three layers as shown in Figure 2, the connection weights between the hidden and output layers are first modified, and then the weights between the input and hidden layers are modified. In general (networks with n layers), EBP algorithms first modify the connection weights between the n-th layer (the output layer) and (n − 1)-th layer, and then between the (n − 1)-th layer and the (n − 2)-th layer, etc. This section only describes the three-layers case.

First, let d_k be a quaternionic desired signal for the k-th output neuron when z’s are input to the network. The connection weights affect the output signals with respect to a set of input signals, thus the error E is regarded as a function with arguments w_kn’s and w_kn ^∗’s. The output error E at the time t is then defined as

(23)

The output error should be real-valued so that it can be minimized.

Suppose that the connection weights are updated at the time (t + 1) by

(24)

where Δw_kn is a quantity in updating. Then, the output error at the time (t + 1) can be written as

(25)

Note that the local analytic condition in quaternionic domain should be satisfied in calculating the derivatives. Thus, if we set Δw_kn as

(26)

where µ is a quaternionic constant, the temporal difference of the output error, ∆E, becomes

(27)

If the real part of µ is positive, ∆E ≤ 0 holds. This indicates that the output error would decrease upon updating weights according to Equations (24) and (26). For calculating the updated quantity in Equation (26), the component ∂E/∂w_kn^∗_‖ are expanded by using chain rule of derivative and ∂y/∂w^∗_‖= 0 from the local analytic condition is applied:

(28)

where h′ is the (local) derivative of the activation function h, and δ_k is defined as .

Similarly, the updates for the connection weights v’s can be deduced. The output error function E isa function with arguments v_nm’s and v^∗_nm’s, thus the output error at the time (t + 1) can be represented by

(29)

Hence, v_nm is updated with the quantity ∆v_nm,

(30)

(31)

This leads to ∆E ≤ 0 under the condition of the real part of µ being positive. The component ∂E/∂^*v_nm‖ can be expanded by chain rules and local analytic conditions, ∂x/∂v^*_‖ = 0, ∂y/∂x^*_‖ = 0, and ∂x/∂v^*_‖ = 0 are applied:

Using the derivatives , and , we finally obtain

(32)

Once a set of network output {y_k} is obtained for a set of net work input, the output error with respect to a target set {d_k} can be calculated by Equation (23). Then, the connection weights between hidden and output layers are modified by Equations (24-28). The connection weights between input and hidden layers are finally modified by Equations (30-32).

3.3. Universal Approximation Capability

As an example of activation functions for neurons (g and h), the quaternionic tanh function [22] can be used. Other types of activation functions are also available, because complex-valued functions can be used for the presented network and the properties of several functions have been explored for the activation functions in [29]. It is important to consider the capability of the proposed quaternionic network with these activation functions, i.e., whether the proposed network can approximate given functions.

This concern is known as the universal approximation theorem [31,32]. In the real-valued MLPs with single hidden layer,the universality has been proven with a so-called sigmoidal function being used as an activation function of neurons. Other than sigmoidal functions, the bounded and differentiable functions are also available for activation functions.

This theorem is also discussed in the case of complex-valued networks [29]. The condition for boundedness in the real-valued MLPs is not required in some complex-valued MLPs. There are three types of activation functions discussed in [29], which are categorized by the properties of complex-valued functions, and for each of them, it is shown that universal approximation can be achieved.

The first type of complex-valued functions concerns the functions without anysingular points. These functions can be used as activation functions and the networks with this type of activation functions are shown as good approximators. Although some of the functions are not bounded, they can be used by introducing bounding operation for their regions. The second type concerns the functions having the bounded singular points, e.g., the discontinuous functions. These singularities can be removed and thus they can also be used for activation functions and can achieve their universality. The last type is for the functions with the so-called essential singularities, i.e., their singularities cannot be removed. These functions can also be used as activation functions, with the consideration of restricting the regions for them so that their regions never cover their singularities.

In the proposed quaternionic MLPs, for example, a quaternionic tanh function can be used as an activation function. This function is unbounded and may contain several kinds of singularities as in the case of complex-valued functions described above. Thus this quaternionic MLPs would face the same problem, i.e., the existence of singularities, but this can also be handled similarly as in the case of complex-valued MLPs for the removal or avoidance of such singularities. It could be possible to show the universality with handling singularities for the proposed MLPs according to the ways adopted in the complex-valued MLPs [29],but it remains as our future work.

4. Conclusions and Discussion

This paper has proposed a multilayer type neural network and an error back-propagation algorithm for its learning scheme in the quaternionic domain. The neurons in this network adopt locally analytic activation functions. The quaternionic functions with local analytic conditions are isomorphic to the complex functions, thus several activation functions, such as complex-valued tanh function, can be used extendedly in the quaternionic domain. The Wirtinger calculus, where a quaternion and its conjugate are treated as independent of each other, makes the derivation of the learning scheme clear and compact.

Analytic conditions for quaternionic functions are derived by defining a complex plane at a quaternionic point, which is a kind of reduction from quaternionic domain to complex domain. There exists another type of reduction in quaternionic domains, such as the commutative quaternion, which is a four-dimensional hypercomplex number system with commutativity in its multiplication [33,34,35,36]. A principal property is that a commutative quaternion can be decomposed and represented by two complex numbers with two linearly independent bases (called the decomposed form by idempotent bases). Commutative quaternions have been applied to neural networks only in terms of Hopfield-type network [24], but defining multilayer perceptron type networks can alsobe straightforward. Thus, it will be interesting to explore the relationship between commutative quaternion-based networks and networks with the local analyticity.

Showing the universality of the proposed network is also an important issue. Quaternionic functions, such as tanh function, may contain several kinds of singularities where the values of functions or their differentials are not defined in particular regions. In complex-valued networks with fully complex-valued function [29], the universality of the networks can be shown by dealing with these singularities, so that such singularities are removed or avoided by restricting the regions. It is expected to show the universality of the proposed network, in a similar way to the case of complex-valued networks.

Also, it is necessary to investigate the performances of the proposed network, though in this paper the experimental exploration could not be accomplished. The proposed network is similar to the networks proposed in [13,14], due to the introduction of local analytic function in quaternionic domain, thus the performances for both types of networks may have similar tendencies. Performance comparisons can also be conducted between the proposed network and the ones in [29], because both networks adopt the same representation in their constructions, i.e., Wirtinger calculus. A wide variety of activation functions have been investigated including the split-type function, phase-preserving function [37], and circular-type function [38]. Similar experiments should be conducted for the quaternionic networks.

Application of the presented network to engineering problems is also challenging. The processing of three or four dimensional vector data, such as color/multi-spectral image processing, predictions for three-dimensional protein structures, and controls of motion in three-dimensional space, will be the candidates from now on.