Predication-Error-Based Intrinsically Motivated Saccade Learning ^†

Abstract

The quick, simultaneous movements of both eyes in the same direction is called a saccade, and the process of developing an internal model for the eyes’ movement-control based on visual stimuli is called saccade learning. All humans use this type of eye motion to bring salient objects to the foveal locations of the retina, even if the objects are located randomly in the surrounding environment. To begin with, infants are not able to perform this type of eye motion, but sensory information motivates them to start learning saccadic behavior. In this paper, a sensory prediction-error-based intrinsically motivated model is proposed for learning saccadic eye movements, and this approach is more consistent with biological systems for saccade learning. Predicted Coding/Biased Competition using Divisive Input Modulation (PC/BC-DIM) network is used for saccade learning using sensory prediction errors. The quantification of sensory prediction errors provides an intrinsic reward. A simulated humanoid agent, iCub, is used to assess and quantify the performance of the proposed model. The performance metrics used for this purpose are percentage mean post-saccadic distance and standard deviation. The mean post-saccadic distance for the proposed model was less than 1°, which is biologically plausible.

1. Introduction

The human visual system’s method of learning behavior effectively demonstrates the importance of saccade learning. Inspired by human saccadic eye movements, an algorithm has been developed that shows 92% efficiency during different experimental tests [1]. All humans use saccades to get the most salient information at foveal locations. The best naturally occurring example of saccade learning derives from newly born infants. To begin with, infants are not able to perform this type of motion, but sensory information motivates them towards learning. During this time, they spend a certain period in the learning phase predicting sensory error information without performing eye movements; then they become able to perform such kinds of motion. It is a common biological phenomenon that sensory prediction errors are used as an intrinsic motivation factor for saccade learning. Therefore, sensory prediction errors play a key role in intrinsic motivation, which is the best choice for training the artificial network in an intrinsically motivated manner. This idea was first introduced by Marco Mirolli et al. in [1]. In essence, in sensory prediction-error-based intrinsic motivation, error magnitude is used as an intrinsic motivation factor [2]. In this research, such intrinsic motivation is used to train the artificial visual neural network. The majority of this work has been related to action-based intrinsic motivation, not sensory-based motivation [3]. The PC/BC-DIM eye control model, for instance, was developed for both saccadic and vergence tasks, and the intrinsic motivation factor completely dependent on the success of both saccadic and vergence task behaviors. This idea was also not proposed according to biological saccade learning. Similarly, there were many other problems related to coordination that needed biological attention when the robot tried to reach the objects through visual feedback. These problems have been solved by using Locally Weighted Projection Regression (LWPR) algorithm in the past, but this algorithm does not work under the sensory-based prediction-error intrinsic motivation principle [4]. In short, most of the work that has been done in the past used action-based information as an intrinsic motivation factor. However, in order to make the humanoid robot more biological plausible, in this research only sensory prediction information is used as an intrinsic motivation to perform the saccade. The PC/BC-DIM neural network is used in this research for saccade learning. The proposed architecture was tested using a humanoid robot iCub simulator.

2. Method

2.1. The PC/BC-DIM Neural Network

The PC/BC-DIM neural network is used in this research. It is, in essence, a version of Predictive Coding (PC) [5], implemented through Divisive Input Modulation (DIM) [6]. The network architecture mainly consists of two types of major functional layers. The learning of these layers can change the behavior of the network. In this network, three types of weights are used: learned feedforward weights, feedback weights, and pooling weights for any unique head-centric location. The PC/BC-DIM neural network consists of three types of neuron populations. The behavior of these neurons is calculated by using the following three equations:

r = Vy

(1)

$$e=x\oslash \left({ϵ}_2+r\right)$$

(2)

$$y\leftarrow \left({ϵ}_1+y\right)\otimes We$$

(3)

2.2. Inputs Encoding Method

An input image and retinal window are generated. The input image then transforms into two-dimensional matrix form for the purpose of MATLAB encoding and simulation. This encoded image is then placed in the form of color at the generated retina window. The input encoding section consists of three sections: eye-centered encoding, eye position encoding, and input patterns for PC/BC network training. In the eye-centered encoding process, the two-dimensional neurons that are generated with the help of Gaussian populations are uniformly presented at evenly spaced Cartesian space. These Gaussian populations are produced with the help of the following given function:

$$G_i (x,y)=G_{max} exp(\frac{{\left(x-a_i\right)}^2+{\left(y-b_i\right)}^2}{2{\sigma }^2}) \mathrm{for} i=1, 2, 3$$

(4)

After distributing the Gaussian populations uniformly at a specific location of the retina, they are then multiplied with a binary image. The activity of each Gaussian is calculated, then each Gaussian activity is summed up inside the receptive retina filters at each location, and the obtained values are normalized by their maximum value after each summation. The output information is then encoded in terms of retinal response. The eye position encoding process consists of pan and tilt neuron information. In this proposed model the pan and tilt neuron information of both eyes is encoded separately by using one-dimensional Gaussian populations.

2.3. Network Training Procedure

The PC/BC network is trained locally by using eye-centered information and randomly selected eye-position signal information for one eye (left or right) at a time. The basis function and pooling neuron weights are learned locally. This pair of learned weights is used for the activation of both local network layers. These local layers of the network are trained separately, as a result of which the computational cost increases. The sensory prediction error information of the neurons is used as an intrinsic motivation reward in this research. The network training process will start if the sensory prediction error value is above 1°.

2.4. Network Testing Procedure

If the sensory prediction error value is less than 1°, then saccadic motor commands are computed to perform the saccade. The computational cost that is required to learn all objects is too high in this experiment; 100,000 iterations are required to learn the maximum number of objects.

3. Results

The experimental results are obtained by using PC/BC-DIM network with the humanoid robot iCub simulator. MATLAB is also used here for simulation purposes. The humanoid robot iCub simulator is used here in a form whose body and head are kept fixed for learning random object head-centric locations, with only the eyes being able to move around a specific range (where the pan range was from −20° to +20° and the tilt range from −12° to +12° along with step size 1). The pan and tilt signal information of both eyes are encoded through one-dimensional Gaussian populations where each Gaussian peak difference is 4 and sigma 2. The iCub eyes’ position control signals are issued by using low-level IPosition control. The stimulus is produced at the retina and is used to calculate the retinal response. The shield is also placed in front of the iCub eyes at a specific distance to neglect the surrounding environment and to get the high-intensity image of the visual object. The retinal plane is populated with a Cartesian population of 121 receptive fields with σ = 5 pixels. The peak difference between two receptive fields along both directions is set at 11 pixels. The iCub simulator was trained through a visual object that looks like a box (that has the same values for all dimension parameters equal to 0.038) which is randomly placed at almost all head-centric locations as shown in Figure 1. The object locations are set in Cartesian space from 0.14 to −0.07 along the x-axis and 1.041 to 0.831 along the y-axis with the difference of 0.015 between object centers, while the distance along the z-axis was kept constant at 0.5. The sensory prediction-error value of the neurons is used as an intrinsic motivation reward. The intrinsic motivation for learning will be low when the sensory prediction error value is below 1° and it will be high when this value is above 1°. This is experimentally proved in this research. When learning motivation is low, then eye motor commands are calculated to perform the saccadic task.

Network Accuracy

The chances of performing a saccade are greatest at low sensory prediction-error values. After performing the saccade test, the distance of the object from the foveal location in the retinal plane is measured as post-saccadic distance. The percentage mean value and standard deviation (SD) of post-saccadic distance are used for different foveal receptive fields. At the same time, the mean value and SD of the sensory prediction-error value are computed at those patterns where the saccade is performed successfully. The trend of percentage mean post-saccadic distance with a change in sensory prediction error is shown in Figure 2. Shuhei Takano et al. in [7] presented a relationship between standard deviation of saccadic error and strength of suppression. To obtain the S_No-blank values, the model is fitted to the displacement detection sensitivity data. The parameters of the model were fixed to the values obtained by fitting them with average data from all observers, excluding S_No-blank. Each dot represents the observer’s data, while the line represents the linear regression line.

4. Conclusions

In this paper, we proposed a prediction-error-based saccade learning using PC/BC-DIM neural network as a core substrate. The proposed saccade learning approach outperformed the state-of-the-art saccade learning approach and yielded biologically plausible saccade accuracy of less than 1°.