Remote Eye Gaze Tracking Research: A Comparative Evaluation on Past and Recent Progress

Abstract

Several decades of eye related research has shown how valuable eye gaze data are for applications that are essential to human daily life. Eye gaze data in a broad sense has been used in research and systems for eye movements, eye tracking, and eye gaze tracking. Since early 2000, eye gaze tracking systems have emerged as interactive gaze-based systems that could be remotely deployed and operated, known as remote eye gaze tracking (REGT) systems. The drop point of visual attention known as point of gaze (PoG), and the direction of visual attention known as line of sight (LoS), are important tasks of REGT systems. In this paper, we present a comparative evaluation of REGT systems intended for the PoG and LoS estimation tasks regarding past to recent progress. Our literature evaluation presents promising insights on key concepts and changes recorded over time in hardware setup, software process, application, and deployment of REGT systems. In addition, we present current issues in REGT research for future attempts.

1. Introduction

The speed of eye movements, regularity of blinks, lengths of fixations, and patterns of visual search behavior are all significant to how a person is responding to any kind of visual stimulus [1]. This is because our eyes automatically follow what interests or threatens us. The eyes are a vital part of human physiology that have continued to hold the attention of industry and academic researchers over several decades. Eye research and systems have progressed over four distinct periods distinguishable by the type of data they provided and the nature of their intrusiveness. The characteristics of these four periods are shown in Figure 1.

The first period was characterized by the basic study of the eye’s structure (see Figure 1a) and the theories of eye movement (see Figure 1b) [2]. The relevance of parts of the eye, including the pupil, cornea, iris, and sclera; and eye movements, such as saccade, fixation, smooth pursuit, and blink, has been studied extensively for REGT systems in [3]. In the second period, eye occulography and tracking emerged, using intrusive instruments that are placed on the human body [4,5]; examples are shown in Figure 1c–f. The progress achieved in hardware processors and image processing techniques paved the way for non-intrusive gaze tracking applications in the third period [6]. These capabilities were explored even further in the fourth period for the remote deployments shown in Figure 1g–k, which were described as remote eye gaze tracking (REGT) systems in [7,8].

The four periods were characterized by specific concepts with similar terminologies, which are: eye occulography, eye tracking, and eye gaze tracking. Eye occulography identifies eye movements. The electro-oculogram (EOG), for example, was described in [9,10] for recording eye movement using electrodes around the eye to measure skin potentials. The electro-retinogram (ERG), which is another neurophysiological means for recording eye movement with a head-mounted eye monitor camera, was described in [11]. Eye tracking locates the subject’s eye(s) in an image or video, as demonstrated using methods such as electro-encephalography (EEG) in [12], the scleral search coil in [13], psychophysical experimentation with foveal cone-target in [14], and the video-oculogram (VOC) in [15]. Eye gaze tracking estimates the drop point of visual attention known as the point of gaze (PoG) or point of regard (PoR); see early [8] and recent [16,17] research attempts. Eye gaze tracking is also used for estimating the direction of visual attention known as line of sight (LoS); see early [18] and recent [19,20,21] research attempts. In the literature eye gaze tracking and eye gaze estimation terminologies refer to the same thing. However, eye gaze estimation has been used as a broader term in [22], to refer not only to processing continuous data with time dependencies, but also to refer to static images in order to determine where the subject is gazing at on a device interface or in a real-world scene.

The operations of REGT systems can be classified into two modules, hardware and software. In Figure 2, the hardware module comprising the display and camera is setup in apposition (i.e., remotely) to the subject in order to feed the required data to the software module for either PoG or LoS estimation.

To achieve PoG (2D) or LoS (3D) estimation, two components of the human physiology are used: the eyeballs and the head pose [23]. Several research attempts at estimating gaze using features of the eye have been reported in [24,25,26,27,28], and for the pose of the human head in [29,30,31,32]. To improve the accuracy of gaze estimation further, researchers measured and combined data from eye features and head pose in [33,34,35,36], and more recently in [37,38]. This method is popularly used to compensate for the errors caused by the head movements, as further discussed in Section 2.3.

We presented the highlight-based evolution of REGT systems (before/after 2015) in [39]. This paper proposes an extension by:

• Collecting more literature on REGT systems’ evolution (past periods before 2015, and recent periods after 2015).
• Comprehensively comparing the key concepts and changes recorded in the evolutionary periods of REGT’s hardware setups, software processes, and applications.
• Presenting current issues in REGT systems’ research for future attempts.

To the best of our knowledge, this level of detail and coverage is not found in similar published literature on this topic, as earlier review papers only focused on past periods and recent review papers focused on recent periods.

The remainder of this paper is arranged and discussed in modules. Section 2 discusses the various components of REGT systems’ hardware modules. Next, popular techniques and algorithms for eye feature detection, extraction, gaze mapping, and datasets for training and validating gaze estimation methods are reported in Section 3. REGT systems as solutions across deployment platforms are discussed in Section 4. In Section 5, our conclusion recapitulates on research changes in REGT systems across the different modules, where we also present current issues for future attempts.

2. Hardware Setup

The physical setup of an REGT system is the hardware module. To achieve the gaze estimation task, this module must provide data for the software module to process. The physical setup is usually made up of various parts consisting of the device interface, which provides the interaction point between the subject and the tracking software; an illuminator which creates patterns of near-infrared on the subject’s eyes for the active light methods discussed in Section 2.2; the camera, which captures images of the subject’s eyes, patterns of near-infrared for processing by the tracking software; and lastly, the subject whose eye gaze is being tracked.

It is imperative to note that there is no universal standard for setting up the hardware module with regard to the number of cameras, number of illuminators, and subject to camera positions. These factors are usually dependent on the purpose, method, and deployment requirements of the REGT system. More so, hardware requirements for an REGT setup have changed remarkably over time. In Figure 3, the traditional REGTs utilized in the past required more hardware components and sessions to achieve the gaze estimation task as compared to the more modern ones. Modern REGT systems have drastically reduced sessions required for the same task, owing to advancements in hardware technologies and software techniques. These advancements are highlighted and discussed in the subsequent sections describing the four hardware components necessary for an REGT setup.

2.1. Interface

In the literature, several interfaces (e.g., phone, tablet, laptop, desktop, and TV) have been utilized as points of interaction between the subject and the tracking software [40]. These device interfaces may operate as standalone (unmodified) gaze trackers or operate with other components (modified) that make up the gaze tracker, as shown in various setups in Figure 1g–k. A modified desktop device with an external light and camera is presented in Figure 1g [41]. An unmodified laptop device embedded with web camera gaze tracker software is presented in Figure 1h [42]. In Figure 1i, an unmodified tablet device is embedded with web camera gaze tracker software [43]. In Figure 1j, a modified TV device has web camera gaze tracker software [44], and Figure 1k presents an unmodified mobile phone device with embedded web camera gaze tracker software [45].

The ideal setup for an REGT system’s interface, also known as display or screen, has been of great concern, particularly when there are no standards. In a desk-based device interface, setup orientation in traditional REGT systems was largely determined by the subject’s seating or standing position, screen size and positioning, and tracking distance, as illustrated in Figure 4a. In addition, the interface being parallel with the subject’s face was essential; great importance was attached to subject’s face being within a defined distance from the REGTs camera—e.g., 30 cm [46], 45–60 cm [47], 50–60 cm [48], 60 cm [49], 80 cm [50], and 81 cm [51]. Modern REGTs setup orientations, however, are more concerned about the subject being anywhere within the camera’s field of view (FOV), from any arbitrary position away from the REGTs camera [52,53]. FOV is the amount of scene that a particular camera can capture. Wide FOV cameras or pan-tilt mechanisms are famously used in the modern REGT setup to extend the area of the tracking regardless of the subject’s position or distance from the REGTs camera, as shown in Figure 4b.

The pan–tilt mechanism is more versatile than the traditional fix in place setup in Figure 4a. The mechanism is commonly based on the camera [54,55]; it gives the tracker the ability to focus on the subject from a range of angles (x, y, and z), without having to take the camera down for focus adjustments.

2.2. Illumination

The common sources of illumination for REGT systems in the literature are visible and infrared lights [50]. The visible light source referred to as passive light is natural illumination found indoors or outdoors. On the other hand, active light is light from an infrared source. Researchers in [56] demonstrated the use of single near infrared lights (NIR) for a traditional REGT setup. In order to improve facial and eye feature detection for more accurate gaze estimation, researchers further demonstrated the use of two or more NIR in [57,58,59] and [7,60], respectively. In recent times, the illumination requirements for REGTs have changed because active light methods do not work accurately outdoors (in the real world) due to the uneven lightening properties. They achieve better results under controlled or constant laboratory lightening.

Passive light methods have been explored by researchers [61]. Those authors demonstrated how they tracked the human face in visible light using a real-time video sequence to extract the eye regions for gaze estimation. Both the active and passive light methods have illumination properties that may affect their accuracies for gaze estimation [62]. We describe those properties in Figure 5 together with mitigation strategies.

The illumination properties that affect active light methods affect passive light methods as well. Effects common to both are: the light intensity, which is the amount of light produced by a specific IR source, or outdoor visibility intensity created by the sun; the light ray becomes a problem when it does not spread in a direction that is favorable to the gaze estimation methods; an illumination source placement is more pertinent to the active light methods, which becomes a problem when the source of illumination is not placed relative to the subject’s position.

2.3. Camera

A camera that is capable of providing high quality images and a reasonable trackable area is very crucial for setting up gaze estimation experiments that achieve good results for traditional REGT systems. A wide trackable area provided by wide-angle cameras tolerates the subject’s free movements, whereas narrow-angle (zoom) cameras restrict such freedom, but on the other hand, provide more focus on the subject being tracked. Past attempts by researchers, such as [63], have demonstrated the use of two or more-camera systems to address the issue of continuous change in the head position. The use of narrow-angle cameras was demonstrated to allow huge head movement; for instance, Kim et al. [64] used one narrow-angle camera with mirrors. The mirrors rotated to follow head movements in order to keep the eyes within the view of the camera. Similarly, researchers in [51] used narrow-angle cameras supported by pan–tilt to extend the tracking area, and in [65,66,67] demonstrated the use of a wide-angle camera to estimate a rough location of the eye region and directed another active pan–tilt-zoom camera to focus on the eye region for eye feature extraction.

Selection of a suitable REGT camera will depend on the software process adopted. A typical setup for feature-based methods would include an RGB camera that extracts essential 2D local eye features, such as pupil location, pupil contours, cornea reflection (glints), and eye corner location information [68,69,70]. If 2D-regression is used to estimate the gaze, this information is enough to facilitate the calibration procedure and gaze mapping function [71,72]. To compensate for the errors caused by the head movements, the model-based methods that involve developing a 3D geometric model of the eye have to be employed. The 3D model-based gaze estimation method uses stereo [73,74,75,76,77,78] and depth [79,80] cameras that provide additional information, such as the subject’s head or face rotation (h_x, h_y, h_z) information and distance from the camera, as described in Figure 6.

Common to both feature-based and model-based methods is the use of NIR cameras. There have been other efforts to estimate PoG and 3D LoS using visible light RGB cameras; we could refer to these attempts as passive light feature or model-based methods. Some authors in [81] introduced a passive light method that estimates the 3D visual axis using both an eye model and eye features that can work both indoors and outdoors.

Modern REGT based on conventional appearance-based methods requires RGB-D cameras equipped with a set of depth-sensing hardware (such as LiDAR, NIR projectors and detectors) for 3D face model reconstructions [82,83,84,85]. This 3D face model provides head pose and facial landmark information that can be used to estimate the 3D LoS. Recent appearance-based methods, in contrast, use images captured by visible light RGB cameras, and then employ machine or deep learning algorithms that directly regress on these images for gaze estimation.

2.4. Subject

The subject whose eye gaze is being tracked is central to the gaze estimation task. The subject’s eye orientation is vital data, and in some cases the head pose is required. The trackable area of the REGT system is determined by the correlation between the camera’s FOV and the subject’s position vertically and horizontally.

In a traditional REGTs setup, the scope is established with a given distance. Figure 7 illustrates that a larger portion of the screen can be tracked if the eye tracker is placed farther away from the subject, and a smaller portion of the screen can be tracked if the eye tracker is placed closer to the subject. That was exemplified in [86] with Tobii’s X series eye trackers: the subject was located within 65 cm from the eye tracker; they could do eye tracking at an angle up to 35–36° out from the center of the built-in camera.

3. Software Process

The software module of an REGT system includes algorithms that process data acquired through the hardware module for gaze estimation. In these reviews [68,71,72,87,88], the software process for gaze estimation is commonly classified by the source of light, kind of data acquired from the subject’s face, and mapping techniques employed. In the existing literature, this classification is made with variety of naming schemes such as appearance-based versus model-based [52,89,90], appearance-based versus feature-based [88], shape-based versus model-based [91], and appearance-based versus geometry-based [16,87]. It is, however, apparent that some of these schemes represent alternative names for the same schemes: appearance-based was referred to as learning-based in [40,92], and as view-based in [71]; model-based was referred to as geometry-based in [16]. Classification into three broad categories was suggested by researchers in [71,93] to clear up the confusion surrounding these ambiguous naming schemes. We expanded on this in

Table 1. Summary of REGT systems’ methods by mapping techniques, data, and light, along with their merits and demerits. Ex. = examples of gaze mapping techniques from the literature.

Methods	Ex. Gaze Mapping Techniques	Data	Light	Merit	Demerit
Feature-based	2D Regression [94,95,96] Neural Network [36,97] Cross Ratio [24,98,99]	Vectors, anchor points	Active, passive	Creates mapping to gaze with relative high accuracy [71]. Discriminative in nature, because it focuses on extracting specific rich features [71,89]. Can be used for active light for PCCR, ICCR, and Purkinje image [28,100] and passive light using PC-EC, IC-EC [61,101].	Don’t work well on low quality images [87]. Generally unsuitable for miniaturized, or calibration free gaze applications [50]
Model-based	2D/3D Geometry Fitting [43,70,102,103,104,105,106,107]	Eye model	Active, passive	Generative in nature, fits a geometry eye model to the eye image [52]. Commonly use the iris contours, eye ball centre to infer gaze based on model parameters [71]. Geometric models can be used for active light [98] and passive light [26].	Modelling features generatively to achieve fairly good accuracy may require more resources
Appearance-based	K-Nearest Neighbors [93,108], Artificial Neural Networks [109,110], Convolutional Neural Networks [111,112,113,114], Support Vector Machines [115,116], Random Forest [16,117,118], Local Linear Interpolation [119,120,121,122,123], Gaussian Process [124,125].	Pixel intensity, texture deference	Passive	Regress from eye images to gaze direction using machine learning [16,115] or deep learning [112,114] algorithms. Works well on low quality images captured from web cameras with visible lightning [89]. Widely used for miniaturized, or calibration free gaze applications [50].	Commonly use low quality images to deduce gaze instead of rich features thus suffer low accuracy [71].

to further describe the REGT software methods by mapping techniques, data, and light, along with their merits and demerits.

REGTs based on active light methods rely on NIR to create reflection data known as glint (dark or bright), and techniques that find the relationship between these reflections and the screen coordinates [68]. Common examples include Pupil Centre Corneal Reflection (PCCR) [7,24,25,100,126], Iris Centre Corneal Reflection (ICCR) [28], Purkinje image reflection [27,127], and the eye model [98,99]. On the other hand, REGTs based on passive light methods rely on web cameras for visible data and techniques to map data to screen coordinates [68]. Pupil Centre Eye Corner (PC-EC) [128], Iris Centre Eye Corner (IC-EC) [61,101,118], the eye model [26,41,70], Appearance Image Pixel [129,130,131], and 3D Image Reconstruction [93] are common examples.

Real-time gaze estimation is achieved using any of these methods, when the eyes move and the image of the target object in a world scene or point in a device interface fixates (i.e., settle) on the fovea of the retina. At this point, a high acuity area of vision is achieved with approximately one degree of visual angle [132]. This fixation lasts for 100–1000 ms or 200–500 ms in a few and most situations, respectively, depending on the current cognitive load and quality of information being processed [132,133]. Based on this, REGT’s accuracy is commonly determined against one degree (1°) of visual angle spanning approximately 1 cm at 57 cm from the subject’s eye.

3.1. Feature-Based Versus Model-Based Methods

Active light feature-based and model-based methods are the foremost NIR methods, and have been demonstrated to be more effective than passive light methods for accurate gaze estimation [8]. In Figure 8, we describe the major components and procedures for feature-based and model-based methods.

3.1.1. Image Acquisition and Pre-Processing

NIR images captured from infrared cameras are required for active light methods, whereas visible RGB images are required for passive light methods. These images can be acquired from static or continuous data. In real-time deployment of REGTs, the use of continuous data is common for obtaining facial images of the subject via a video camera. Raw data from the acquisition process may require some pre-processing before they can be used for gaze estimation. When the images are input, most gaze estimation methods start by applying binarization and normalization to convert them into grayscale and scale them to having a reasonable threshold, resolution, and size. Noise reduction filters such as a Gaussian filter are then applied to compensate for any noise present due to any anomalies in the camera used.

3.1.2. Feature Detection

Feature detection is a popular Computer Vision (C.V) technique for identifying interest points, which closely defines a feature in an image. Interest points can be a corner, ridge, or edge. Traditional CV techniques capable of detecting rich features of interest are commonly used by the feature-based and model-based methods [134,135], such as circular Hough transform (CHT) [136,137,138], longest line detector (LLD) [139], subpixel detectors [140], and Haar detectors [141,142]. A pupil feature detector, the Haar detector [142], was applied to roughly estimate the pupil region and reduce the overall search space of the eye region algorithms. This is a widely used approach of isolating the pupil blob to assume that it is the darkest element in the image and then applying intensity thresholding to isolate it from the background. Pupil thresholding is applied in [143]. Then k-means clustering algorithms can be applied to the histogram image to obtain pupil and background pixels, respectively, as proposed in [144]. To remove unwanted edges and reflections, and mitigate problems of eye occlusion features, such as eyelashes, a series of open morphological operations are applied [144,145,146,147,148].

3.1.3. Feature Extraction

Feature extraction for active light feature-based methods uses the feature points or positions. The authors of [7,24,126] extracted intensity points of the pupil and glint area. This happens when an IR light is shone into the subject’s eye and a reflection occurs on the surface of the pupil/cornea. The reflection makes a bright spot (known as glint) on the pupil/cornea (see Figure 9a); the position of the glint varies according to the gaze direction. To estimate the point of gaze, Yoo and Chung [7] applied a projective invariant. Assuming the property of the projective space as a relation between the main display screen and the reflected screen, the projective invariant is constant under any projective transform. The projective invariant of the main display screen (Invariant (DScreen_x, DScreen_y)) is equal to that of the reflected screens (InvariantRScreen₁, InvariantRScreen₂, InvariantRScreen₃). As an alternative, given the detected glint and pupil points in [24,126], calibration procedures such as linear polynomial, second-order polynomials, homography matrix, and interpolation are commonly used to find the relationship between extracted feature parameters (e.g., glint and pupil points) and screen coordinates. These methods are discussed a little further on.

The active light model-based method in [98,99] uses edge detection methods (such as a canny edge detector) to get pupil contours (see Figure 9b), but these approaches can be computationally inefficient [145,149]. These pupil contours are extracted and evaluated using the ellipse fitting method, looking for the best candidate for the pupil contour [145,150]. Ellipse evaluation and fitting is the final stage in the pupil detection algorithms. The best method is to point to the exact pupil location by using the ellipse fitting approach. The commonly used method is the ellipse least-squares fitting method [150], but errors made in the pupil feature detection phase can highly influence the results. Instead, the random sample consensus (RANSAC) method is sometimes used, as it is effective in the presence of a large percentage of outliers of the pupil’s ellipse feature points [145,151]. When this is done, a calibration procedure to find the relation of the ellipse feature to screen coordinates is performed using homography matrix.

Some examples of passive light featured-based applications in the existing literature can be found in [61,101,128]. The supervised descent method was used in [101] to localize the inner eye corners and perform the convolution of the integer-differential for the eye localization method. In two studies [61,101], the authors extracted the feature points of the iris center and eye corner (see Figure 9d). The pupil center and eye corner (see Figure 9c) were extracted in [128]. In [26,41,70], the authors extracted feature contours using passive light. In [101], the second-order polynomial was applied for the calibration procedure.

3.1.4. Gaze Calibration and Mapping

To map the extracted eye parameters to the screen coordinates, a relationship between locations in the scene image and the eye parameters must be estimated through a calibration procedure. The user-specific parameters are obtained by running calibration routines, such as screen marker calibration and natural features calibration. Using Figure 6 as a reference, some commonly used calibration procedures for both traditional feature-based and model-based REGTs are discussed as follows:

• Five-point linear polynomial: The linear polynomial calibration points are the simplest. The method presents a five-point marker on a screen for the subject to look at. By looking at these points and clicking on them, the mapping between screen coordinates and the extracted feature parameters is performed using following equation derived in [152]:

$$\begin{array}{l}{s}_x=a_0+a_1\ast f_x\\ s_y=b_0+b_1\ast f_y\end{array}$$

(1)

where ($s_x$, $s_y$) are screen coordinates and ($f_x$, $f_y$) are the extracted feature parameters, e.g., pupil-glints vectors. Using the direct least squares method proposed in [150], the unknown coefficients $a_0$, $a_1$ and $b_0$, $b_1$ can be found during calibration. The problem with this simple linear method is that calibration mapping becomes inaccurate as the subject’s head moves away from its original position [8].

• Nine or 25-point second-order polynomial: By fitting higher order polynomials, the second-order polynomial has been shown to increase the accuracy of this system compared to linear ones [8]. A second-order polynomial calibration function was used with a set of nine calibration points in [25,101] and 25 calibration points in [152]. The polynomial is defined as:

$$\begin{array}{l}{s}_x=a_0+a_1\ast f_x+a_2\ast f_y+a_3\ast f_x\ast f_y+a_4\ast f_x{}^2+a_5\ast f_y{}^2\\ s_y=b_0+b_1\ast f_x+b_2\ast f_y+b_3\ast f_x\ast f_y+b_4\ast f_x{}^2+b_5\ast f_y{}^2\end{array}$$

(2)

where ($s_x$, $s_y$) are screen coordinates and ($f_x$, $f_y$) are the extracted feature parameters, e.g., pupil-glints vectors. During calibration, the coefficients $a_0$ − $a_5$ and $b_0$ − $b_5$ can be found using the least squares method proposed in [150].

• Homography matrix: Under homography, the calibration routines capture homogeneous coordinates as screen points $s$ = ($s_x$, $s_y$, 1), and their corresponding feature points $e$ = ($f_x$, $f_y$, 1) are captured (homogeneous coordinates). The transformation of point in 3D from screen points $s$ to the feature points $e$ is given by;

$$s=He\iff \left[\begin{array}{c}{s}_x\\ s_y\\ 1\end{array}\right]=\left[\begin{array}{ccc}a& b& c\\ d& e& f\\ g& h& i\end{array}\right]\left[\begin{array}{c}{f}_x\\ f_y\\ 1\end{array}\right]$$

(3)

where H is a (3 × 3) homography matrix, and He is a direct mapping of points in the screen. Once the matrix H is determined, the gaze in the screen can be estimated.

• Interpolation: The authors of [140] had the subject look at several points on a screen to record the corresponding eye feature points and positions. These points served as the calibration points. Then they computed the gaze coordinates by interpolation (a 2D linear mapping from the eye feature to the gaze on screen). The details of this mapping function are as follows:

$$s=s_1+\frac{fx-f{x}_1}{f{x}_2-f{x}_1}(s_2-s_1),f=f_1+\frac{fy-f{y}_1}{f{y}_2-f{y}_1}(f_2-f_1)$$

(4)

where ($fx$, $fy$) are the eye feature vectors; the calibration points $p_1$ and $p_2$ for screen coordinates and eye features are (($s_1$, $f_1$), ($f{x}_1$, $f{y}_1$)) and (($s_2$, $f_2$), ($f{x}_2$, $f{y}_2$)), respectively.

3.1.5. Calibration Error Calculation

The averaged mapping error for the calibration methods we have described can be calculated by the following equations derived in [152]. Firstly, the mapping error of individual calibration point ($i_{error}$) is computed as:

$$i_{error}=\sqrt{{(s_x-f_x)}^2-{(s_y-f_y)}^2}$$

(5)

where ($s_x$, $s_y$) are the actual screen coordinates, and ($f_x$, $f_y$) are the feature vectors. Further, the average calibration mapping error can be computed as:

$$cali{b}_{error}=\frac{{\displaystyle \sum _{i=1}^n{i}_{error}}}{n}$$

(6)

where $cali{b}_{error}$ is the calibration technique mapping error, and $n$ is the number of calibration points.

3.2. Appearance-Based Methods

Old reflection-based methods face challenges when applied in real-world settings because they require dedicated NIR devices. These methods are majorly subject to eye occlusion, and other reflection irregularities, such as pupil dilation [19,153,154]. To solve these bottlenecks, recent attempts by researchers have been based on techniques which focus on image appearance to extract the characteristics of the entire eye region instead of specific features that require a dedicated device [155]. As described in Figure 10, the appearance-based methods are inspired by machine and deep learning algorithms and depend on the features learned.

3.2.1. Image Acquisition and Pre-Processing

Regardless of the gaze estimation method, similar procedures for image acquisition and pre-processing are commonly used. The feature detection and extraction procedure by appearance-based methods is most often treated as a part of pre-processing. This procedure commonly uses modern C.V techniques based on machine or deep learning for extracting features from visible RGB or depth images. These images are obtained of a subject gazing at a known location on the screen to generate gaze coordinates as training data, which are used to train models that make predictions on the new data.

3.2.2. Model Training

The learning procedure for appearance-based methods depends on the characteristics (i.e., annotation) of the training data, such as the gaze-vector angles and the 3D location of the eye in the camera coordinate system, and the 2D coordinates of the gaze point on the screen in the screen coordinate system. The supervised appearance-based methods described in [53,155,156] rely on appropriately labelled or ground truth gaze data. This approach, however, is expensive and time consuming. Unsupervised appearance-based methods in [157,158,159] have demonstrated the effectiveness of learning on unlabeled data. Both procedures learn the mapping from a large set of data, and generalize this mapping to other subjects via training. Specifically, the task of feature extraction and feature vector to gaze point mapping have been demonstrated using common machine learning algorithms, such as the genetic algorithms (GA) [160], the Bayesian classifier [161], the Kalman and adaptive thresholding algorithm [162], AdaBoost [163,164], k-nearest neighbor (KNN) [93,108,117,130], adaptive linear regression (ALR) [37], random forest (RF) regression [16,117,118], gaussian process (GP) regression [124,125], linear ridge regression (LRR) [165], support vector machines (SVM) [36,115,116,166], artificial neural networks (ANNs) [109,110,167,168], and generalized regression neural networks (GRNNs) [36]. Instances of deep learning algorithms include region-based convolutional neural networks (RCNNs) [164,169,170] and You Only Look Once (YOLO) [171,172], convolutional neural networks (CNNs) [17,38,53,173,174,175], recurrent neural networks (RNNs) [20,176,177], and generative adversarial networks (GANs) [178,179,180,181,182]. Typically, using these learning-based algorithms for mapping requires a training period or implicit calibration [183]. The subject is asked to look at a number of predefined points on the screen while their gaze and eye locations are estimated and recorded for each of the points. The model is trained on the recorded points, depending on the size of training sample; the model is expected to implicitly learn the mapping between the camera coordinate system and screen coordinate system.

The training complexity of machine and deep learning algorithms used for appearance-based gaze estimation in the literature is between O(n³) and O(log(n)). ALR, SVM, and GP regression deal with sparse collections of training samples. These algorithms are not fit for operation on large datasets because their complexities can reach O(n³) for datasets with sample size n. RF and KNN are friendly to large datasets and could computationally train rapidly on such datasets at more moderate complexities, O(n²) and O(n). An ANN outperformed other baseline machine learning methods for appearance-based gaze estimation, such as RF and KNN, for a large scale complex problem (O(n), O(log(n))). It is able to perform in complex computations involving much data, making it possible for researchers to explore deeper attributes of data—now popularized as deep learning (DL). Recent DL models utilized for gaze estimation include CNNs, RNNs, and GANs.

• Convolutional Neural Networks

CNNs, also known as ConvNets, use perceptrons to analyze data [184]. Typical components of a CNN include the input and output layers, and various hidden layers. These hidden layers include convolutional layers, which detect patterns from images, a pooling layer, which reduces number of parameters and amount of computation in the network to control overfitting, and a fully connected (FC) layer, which gives the output. CNNs’ successful use in image processing tasks, particularly 2D image classification, has inspired its use for gaze estimation in the recent literature [155,185]; see Figure 11.

Features are directly extracted from input data. These data include images of the eye, face, and others (described in Table 2) that need to be fed into the CNN. Using some mathematical functions, results are passed between successive layers [184,186]. A single-region CNN processes its inputs through a single network, as shown in Figure 11a, whereas a multi-region CNN processes its inputs in multiple separate networks for increased efficiency of the general network, as shown in Figure 11b. The outputs of a CNN for gaze estimation are classification results, i.e., discrete values suitable for gaze zone classification; or regression results, i.e., continuous values suitable for estimating more specific gaze angles and points of gaze.

One of the early attempts that demonstrated the use of CNNs for gaze estimation was reported in [53]. Their attempt was based on a LeNet inspired by MnistNet architecture [187] consisting of two convolutional layers, two connecting maxpooling layers, and a fully connected layer. To predict gaze angle vectors, linear regression training is performed on top of the fully connected layer. Motivated by this progress, the authors improved the idea and proposed the GazeNet framework, the first deep appearance-based gaze estimation method based on a 13 convolutional layer VGG Network, in [38]. They combined the data from face pose and eye region and injected head angle into the first fully connected layer, as described in Figure 11a, and then trained a regression model on the output layer that predicted with angular accuracy of 10.8° for cross-dataset evaluation. Comparatively, Itracker [17] was based on AlexNet architecture. It uses a multi-region network with various inputs to provide more valuable information than using eye images alone and achieved a prediction error for a pixel distance of between 1.71 and 2.53 cm without calibration. Since then, several other works have applied CNNs and other deep learning models for gaze estimation using different architectures and structures, as summarized in Table 2.

Table 2. Recent (2015–2021) work on CNNs and other DL models for gaze estimation, presented by network type.

Deep Network Classification	Literature	Year	Input	Network Description	Output
Single-region CNN	[114]	2017	Full face	Spatial weighted CNN	Point of gaze (2D)
Multi-region CNN	[17]	2016	Right & Left eye, face, and face grid	Four-region CNN model AlexNet backbone Dark knowledge method
	[188]	2017	Head pose, and eye	Two-region CNN model AlexNet backbone Gaze transform method
	[189]	2020	Right & Left eye, full face, and face depth	Two-region CNN model ResNet-18 backbone Facial landmarks global optimization
Single-region CNN	[53]	2015	Double eye, and head pose	LeNet backbone	Gaze angle (3D)
	[38]	2019	Double eye, and head pose	VGG backbone
	[190]	2020	Full face	ResNet-50 backbone
Multi-region CNN	[191]	2016	Right & Left eye	Two-region CNN model Modified Viola-Jones algorithm
	[111]	2018	Right & Left eye, and head pose	Four-region CNN model Asymmetric Regression (AR-Net), and Evaluation Network (E-Net).
	[112]	2018	Right & Left eye, and face	Three-region CNN model Semantic image inpainting Net, landmark detection deep Net, and head pose estimation Net
	[173]	2019	Right & Left eye	Two-region CNN model Based on RT-GENE [51], and blink detection Net
CNN with RNN fusion	[177]	2018	Full-face, eye-region, facial landmarks	Two-region CNN for static feature extraction. VGG-16 backbone, and ADAM optimizer Many-to-one recurrent Net for temporal feature extraction and final prediction.	Gaze angle (3D)
	[176]	2019	Left & Right eye, and face	Two-region CNN for static feature extraction. AlexNet backbone Many-to-one bi-LSTM for temporal feature extraction and final prediction.
	[20]	2019	Full face	Multi-frame bidirectional LSTM for temporal features and final prediction. ResNet-18 backbone
	[192]	2020	Right & Left eye	ResNet-18 architecture backbone GRU Cell for temporal model
GAN	[193]	2020	Full face	Self-Transforming GAN (ST-ED) for gaze and head redirection	Gaze angle (3D)
GAN	[194]	2021	Eye region, head pose	Multi-task conditional GAN (cGAN) for gaze redirection
CNN with GAN fusion	[195]	2017	Eye	SimGAN to improve realism of synthetic images
	[178,179]	2020	Eye	ResNet backbone EnlightenGAN to improve to lightening condition of dark input images.

Table 2. Recent (2015–2021) work on CNNs and other DL models for gaze estimation, presented by network type.

Deep Network Classification	Literature	Year	Input	Network Description	Output
Single-region CNN	[114]	2017	Full face	Spatial weighted CNN	Point of gaze (2D)
Multi-region CNN	[17]	2016	Right & Left eye, face, and face grid	Four-region CNN model AlexNet backbone Dark knowledge method
	[188]	2017	Head pose, and eye	Two-region CNN model AlexNet backbone Gaze transform method
	[189]	2020	Right & Left eye, full face, and face depth	Two-region CNN model ResNet-18 backbone Facial landmarks global optimization
Single-region CNN	[53]	2015	Double eye, and head pose	LeNet backbone	Gaze angle (3D)
	[38]	2019	Double eye, and head pose	VGG backbone
	[190]	2020	Full face	ResNet-50 backbone
Multi-region CNN	[191]	2016	Right & Left eye	Two-region CNN model Modified Viola-Jones algorithm
	[111]	2018	Right & Left eye, and head pose	Four-region CNN model Asymmetric Regression (AR-Net), and Evaluation Network (E-Net).
	[112]	2018	Right & Left eye, and face	Three-region CNN model Semantic image inpainting Net, landmark detection deep Net, and head pose estimation Net
	[173]	2019	Right & Left eye	Two-region CNN model Based on RT-GENE [51], and blink detection Net
CNN with RNN fusion	[177]	2018	Full-face, eye-region, facial landmarks	Two-region CNN for static feature extraction. VGG-16 backbone, and ADAM optimizer Many-to-one recurrent Net for temporal feature extraction and final prediction.	Gaze angle (3D)
	[176]	2019	Left & Right eye, and face	Two-region CNN for static feature extraction. AlexNet backbone Many-to-one bi-LSTM for temporal feature extraction and final prediction.
	[20]	2019	Full face	Multi-frame bidirectional LSTM for temporal features and final prediction. ResNet-18 backbone
	[192]	2020	Right & Left eye	ResNet-18 architecture backbone GRU Cell for temporal model
GAN	[193]	2020	Full face	Self-Transforming GAN (ST-ED) for gaze and head redirection	Gaze angle (3D)
GAN	[194]	2021	Eye region, head pose	Multi-task conditional GAN (cGAN) for gaze redirection
CNN with GAN fusion	[195]	2017	Eye	SimGAN to improve realism of synthetic images
	[178,179]	2020	Eye	ResNet backbone EnlightenGAN to improve to lightening condition of dark input images.

The accuracy of gaze estimation methods largely depends on the effectiveness of feature detection and extraction [196,197]. Appearance-based gaze estimation using deep learning has recently demonstrated robustness against image input noise, blur, and localization errors [191,198]. Several works have primarily focused on appearance-based methods using CNN within the past five years. A few are described in Table 2, and others are reported in [52,68,89,155]. In addition, the recently published results shown in Table 3 suggest the appearance-based methods using CNNs work better with unconstrained setups than feature-based and model-based methods [38,52]. However, a feature-based or model-based method with constrained illumination (i.e., NIR) still achieves better accuracy for the gaze estimation than an appearance-based method under visible light [199]. For this reason, researchers have intensified efforts to improve the accuracy of unconstrained appearance-based gaze estimation [38,114], hence the exploration of other deep learning frameworks, such as RNNs and GANs.

2.

Recurrent Neural Network

RNNs use long short-term memory (LSTM) to process temporal information in video data. The use of temporal (i.e., sequential) features was demonstrated in [20,176,177,192] to be effective at improving gaze estimation accuracy. This is because the video data contain more valuable information than image data. A framework in which fusion information from static features is obtained from the images and sequential features are obtained from video, is described in Figure 12.

The authors of [177] used a multi-region CNN to extract and process static features from the face, eye region, and face landmarks to estimate gaze. They then fed these learned features of all the frames in a sequence to an RNN that predicted the 3D gaze vector of the last frame. They achieved 14.6% superiority over the state-of-the-art method [53] on EyeDiap dataset using static feature only, adding that the temporal feature further improved the performance by 4%. Similarly, the authors of [176] enhanced the itracker network proposed in [17]. They removed face-grid, thereby reducing one network branch from the original, and then used a static feature from a concatenate of the two-eye region images to predict gaze. They further employed the bidirectional LSTM (bi-LSTM) to fit the temporal feature between frames to estimate the gaze vector for a video sequence. They achieved improved performance with the enhance itracker network—11.6% over the state-of-the-art methods, such as those in [17,114,199], on MPIIGaze dataset for static features, and a further 3% improvement with the bi-LSTM for temporal features on the EyeDiap dataset. Both works achieved better estimation accuracy by combining a static network and temporal network instead of only using a static network for gaze estimation.

3.

Generative Adversarial Networks

GANs are effective for data generation. The basic theory of the GAN and its generative models has been discussed in [200]. The use of GANs for gaze estimation was demonstrated recently in [178,179,193,194,195]. In [193], the authors used adversarial learning to manipulate the gaze of a given face image with respect to a desired direction. Similarly, the authors of [194] adopted the use of flow learning and adversarial learning; the network is described in Figure 13a.

Eye region x and the head pose h are taken in as encoder inputs. The decoder outputs fine-grained image f_x; the generator g outputs the residual image r, which is added to f_x. The refined results rf_x and the ground truth g_r are fed to the discriminator d. The discriminator network with gaze regression learning ensures that the refined results and the ground truth have the same distribution and the same gaze angles. Gaze redirection error achieved was about 5.15°.

The authors of [178] used a fusion of a GAN and a CNN for recovering the missing information of images captured under low-light conditions using the framework described in Figure 13b. They first used the GAN to recover (i.e., separating the noise) near to original eye images from the low-light images, and then fed the recovered images into the CNN network proposed in [38] to estimate the gaze. Their experimental results on the GAN enhanced image dataset demonstrated an improved performance compared to GazeNet by 6.6% under various low light conditions.

3.3. Evaluation and Performance Metrics for REGTs

The performance of REGT is usually described by using three terms, which are: accuracy, robustness, and stability [22]. All three terms are important considerations for well performing REGT systems, but the accuracies and precisions of REGT systems hold greater implications for real applications [201].

3.3.1. Precision Evaluation of REGT Systems

Precision is the ability of the REGT system to reliably predict or reproduce relative gaze positions. It measures the variations in the recorded gaze positions using a confusion matrix. Precision $m_{precision}$ can be computed for classification problems as:

$$m_{precision}=\frac{(T{P}_{b1}+T{P}_{b2}+\dots +T{P}_{bn})}{((T{P}_{b1}+T{P}_{b2}+\dots +T{P}_{bn})+(F{P}_{b1}+F{P}_{b2}+\dots +F{P}_{bn}))}$$

(7)

where TP is the true positive predictions; FP is the false positive predictions of successive samples $b_1$–$b_n$ as described in Figure 14b. The regression problem as the root mean square error $a_{error}$ of successive samples $b_1$–$b_n$, derived in [62,202], is:

$$a_{error}=\sqrt{\frac{1}{n}{\displaystyle \sum _{i=1}^n{\alpha }_i^2}}=\sqrt{\frac{{\alpha }_1^2+{\alpha }_2^2+\dots +{\alpha }_n^2}{n}}$$

(8)

where $\alpha$ is the visual angle in degrees and $n$ is the number of recorded samples in the dataset. ${\alpha }_1$–${\alpha }_n$ is the successive visual angle between samples $b_1$–$b_n$. Precision is calculated for each eye individually and as a mean of both eyes.

3.3.2. Accuracy Evaluation of REGT Systems

Described in Figure 14a, accuracy has been reported through various metrics in the literature as angular resolution, recognition rate, and pixel distance. The angular resolution describes the mean error of estimating the gaze angle $\alpha$, as a deviation between real gaze position (ground truth) $a$ and the estimated gaze position $b$, measured in degrees. The lesser the deviation in gaze angle between real gaze position and the estimated position, the better the accuracy of REGTs. The recognition rate is measured in percent, and it describes the average recognition rate of ground truth gaze positions; the pixel distance, also known as the Euclidean distance, describes the mean error in distance between the real gaze position and estimated gaze position in millimeters or centimeters.

Measuring the classification accuracy of REGTs is simply done as the ratio of correct predictions (true positives TP and true negative TN) to total predictions $n_{predictions}$ of samples. Using a confusion matrix, the accuracy $m_{accuracy}$ can be computed for the recognition rate as:

$$m_{accuracy}=\frac{(TP+TN)}{n_{predictions}}$$

(9)

On the other hand, the measuring regression accuracy of REGT systems can be computed for angular resolution and pixel distances (as derived in [62]) as follows:

$$pixe{l}_{error}=\sqrt{{(ac{t}_x-es{t}_x)}^2+{(ac{t}_y-es{t}_y)}^2}$$

(10)

$pixe{l}_{error}$ is derived for both eyes, $ac{t}_x$ and $ac{t}_y$ are the actual gaze positions, $es{t}_x$ and $es{t}_y$ are the estimated gaze positions.

$$angula{r}_{error}=\frac{(pixe{l}_{size}\times pixe{l}_{error}\times \cos {(gaz{e}_{angle})}^2)}{gaz{e}_{point}}$$

(11)

$angula{r}_{error}$ is derived for both eyes, $pixe{l}_{error}$ is the mean value of $gaz{e}_{point}$, $gaz{e}_{angle}$ is the mean value of the gaze angle, and $gaz{e}_{point}$ is the mean distance of the eyes from the tracker. Computations for $gaz{e}_{point}$ and $gaz{e}_{angle}$ are derived in [62].

In Table 3, cited accuracies are presented for various gaze estimation methods reported as angular resolution, gaze recognition rate and the pixel distance. The error for angular resolution and pixel distances are the commonly used metric to measure the accuracy of REGTs methods and systems in the literature. Deviating from this practice, the authors of [68] presented an argument against these metrics and their implications for inter-comparison of gaze estimation methods. They also stressed the need for a holistic consideration of sources of error that arise from the different components that went into the gaze estimation, such as the subject, device, and environment. Thus, they suggested that various parameters be evaluated within each component and across deployment platforms (for example, subject component: head pose variation, eye occlusion, human eye condition; device: camera quality, properties of the deployment device; environment: user distance, illumination changes, and motion caused by deployment device) to make fair inter-comparisons of gaze estimation methods possible.

Table 3. An accuracy comparison of different gaze estimation methods. Abbreviations: near infrared light (NIR), point interpolation (PI), linear interpolation (LI), eye corner (EC), ellipse shape (ES), iris center (IC).

Methods	Literature	Accuracy	Hardware Setup	Software Process
Active light feature-based	[203]	<1°	Desktop, stereo infrared camera, 3 NIR	Purkinje Image, 1 point
	[24]	0.9°	Desktop, 1 infrared camera, 2 NIR	PCCR, Multiple points
	[204]	96.71%	Desktop, 2 infrared camera, 4 NIR	PCCR, Multiple points
	[205]	10.3 mm	1 infrared camera, 4 NIR	PCCR, Multiple point
Active light model-based	[206]	<1°	Desktop, 1 infrared camera, 4 NIR	Eye model, 1 point
	[207]	1°	Desktop, stereo camera, Pan-tilt infrared camera, 1 NIR	Eye model, 2 points
	[98]	<1°	Desktop, 1 infrared camera, 2 NIR	Eye model, Multiple
Passive light feature-based	[208,209]	1.6°	Desktop, 1 web camera	PC-EC, GP, Grid
	[61,101,128]	1.2°–2.5°	Desktop, 1 web camera	PC-EC, PI, Grid
	[210]	>3°	Desktop, 1 web camera	EC, LI
Passive light model-based	[26]	2.42°	Desktop, 1 web camera	ES-IC, Grid
	[70]	<1°	Desktop, 1 web camera	ES, Grid
	[41]	~500 Hz	Desktop, 2 web camera	Eye model, Grid
Passive light appearance-based with machine learning	[118]	1.53°	Desktop, 1 web camera	RF, 25 points
	[124]	2°	Desktop/Handheld, 1 web camera	GP, Grid
	[120,121,122]	2.2°–2.5°	Desktop, 1 web camera	LLI, Grid
	[110]	<3.68°	Desktop, 1 web camera	ANN, 50 points
	[119,123]	3.5°–4.3°	Desktop, 1 web camera	LLI, Saliency
	[108]	4.8°–7.5°	Desktop, 1 web camera	KNN, Calibration free
Passive light appearance-based with deep learning	[211]	7.74°	Handheld, 1 web camera	CNN, Calibration free
	[191]	81.37%	Desktop, 1 web camera	CNN, Calibration free
	[17]	1.71 cm and 2.53 cm	Handheld, 1web camera	CNN, Calibration free

Table 3. An accuracy comparison of different gaze estimation methods. Abbreviations: near infrared light (NIR), point interpolation (PI), linear interpolation (LI), eye corner (EC), ellipse shape (ES), iris center (IC).

Methods	Literature	Accuracy	Hardware Setup	Software Process
Active light feature-based	[203]	<1°	Desktop, stereo infrared camera, 3 NIR	Purkinje Image, 1 point
	[24]	0.9°	Desktop, 1 infrared camera, 2 NIR	PCCR, Multiple points
	[204]	96.71%	Desktop, 2 infrared camera, 4 NIR	PCCR, Multiple points
	[205]	10.3 mm	1 infrared camera, 4 NIR	PCCR, Multiple point
Active light model-based	[206]	<1°	Desktop, 1 infrared camera, 4 NIR	Eye model, 1 point
	[207]	1°	Desktop, stereo camera, Pan-tilt infrared camera, 1 NIR	Eye model, 2 points
	[98]	<1°	Desktop, 1 infrared camera, 2 NIR	Eye model, Multiple
Passive light feature-based	[208,209]	1.6°	Desktop, 1 web camera	PC-EC, GP, Grid
	[61,101,128]	1.2°–2.5°	Desktop, 1 web camera	PC-EC, PI, Grid
	[210]	>3°	Desktop, 1 web camera	EC, LI
Passive light model-based	[26]	2.42°	Desktop, 1 web camera	ES-IC, Grid
	[70]	<1°	Desktop, 1 web camera	ES, Grid
	[41]	~500 Hz	Desktop, 2 web camera	Eye model, Grid
Passive light appearance-based with machine learning	[118]	1.53°	Desktop, 1 web camera	RF, 25 points
	[124]	2°	Desktop/Handheld, 1 web camera	GP, Grid
	[120,121,122]	2.2°–2.5°	Desktop, 1 web camera	LLI, Grid
	[110]	<3.68°	Desktop, 1 web camera	ANN, 50 points
	[119,123]	3.5°–4.3°	Desktop, 1 web camera	LLI, Saliency
	[108]	4.8°–7.5°	Desktop, 1 web camera	KNN, Calibration free
Passive light appearance-based with deep learning	[211]	7.74°	Handheld, 1 web camera	CNN, Calibration free
	[191]	81.37%	Desktop, 1 web camera	CNN, Calibration free
	[17]	1.71 cm and 2.53 cm	Handheld, 1web camera	CNN, Calibration free

To unite and standardize the diversity of evaluation protocols used in the literature, they proposed a framework in [212] to report system performance in formats that are quantitative and uniform for angular accuracy metrics, statistical metrics, sensitivity metrics, and a new metric based on receiver operating characteristic (ROC). This evaluation framework takes into consideration the characteristics of gaze estimation methods to fairly describe the quality of REGT systems. The ROC curve is plotted using Equation (12).

$$TPR=\frac{TP}{TP+FN},FPR=\frac{FP}{FP+TN}$$

(12)

In order to approximate the values of the true positive rate (TPR) and false positive rate (FPR), data from the gaze tracker are obtained to estimate the numbers of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Then, validating the ROC curve, the TPR and FPR values are computed with random error thresholds for TP, TN, FP, and FN. This metric, however, is yet to gain much recognition for evaluating REGT systems.

3.4. Dataset

Datasets are important data acquisition means for REGTs, built to store image or video data. Data stored in a dataset are usually labeled or annotated for head pose, gaze direction, gaze point, resolution, and illumination conditions. In early literature on appearance-based methods, datasets presented collected data under controlled laboratory conditions from synthesized images [117], and natural faces were referred to as realistic or naturalistic images [124,213,214,215,216,217,218]. However, because it was difficult to obtain well-labeled data in these datasets, the authors of [16,219,220] presented well-labeled datasets for naturalistic images, and dataset of synthesized images are presented in [93,221]. Recent methods for gaze estimation are focused on the real world; they use datasets containing data from real-world settings for validation. In

Table 4. Recent datasets (2015–2020) for real-world gaze estimation. The listed datasets contain varied data for gaze targets, head pose, and illumination conditions. Ex. = example of accuracies reported for training sets. # = number of samples.

Sample	Dataset	Year	Mode (Annotation)	# Samples	Resolution (Pixels)	Ex. Accuracy for Training
Image	MPIIGaze [53]	2015	RGB eye, 2D & 3D gaze	213,659	1280 × 720	7.74° [211], 4.3° [112], 4.8° [114]
	GazeCapture [17]	2016	RGB full face, 2D gaze	2,445,504	640 × 480	3.18° [185]
	RT-GENE [112]	2018	RGB-D full face, 3D gaze	277,286	1920 × 1080	7.7° [112], 24.2° [20], 8.4° [222]
	RT-BENE [173]	2019	RGB-D full face, 3D gaze	210,000	1920 × 1080	0.71° [173]
	Gaze360 [20]	2019	RGB full face, 3D gaze	172,000	4096 × 3382	2.9° [20]
	XGaze [190]	2020	RGB full face, 2D & 3D gaze	1,083,492	6000 × 4000	4.5° [190]
Video	EyeDiap [223]	2014	RGB-D full face, 2D & 3D gaze	94	1920 × 1080	5.71° [222], 5.84° [176], 5.3° [224]
	TabletGaze [16]	2017	RGB full face, 2D gaze	816	1280 × 720	3.63° [53], 3.17° [16], 2.58° [17]
	EVE [192]	2020	RGB full face, 2D & 3D gaze	161	1920 × 1080	2.49° [192]

, we present some datasets that have been used for real-world gaze estimation in the recent literature.

Each of these datasets was composed under varied illumination conditions suitable for passive light gaze estimation. The XGaze [190] dataset was collected in the laboratory using 18 custom high definition SLR cameras with 16 adjustable illumination conditions. Participants included 110 subjects (63 male, 47 female) who were required to stare at an on-screen stimulus. EVE [192] was collected using a desktop setup. The study included 54 participants and four camera views, and natural eye movements as opposed to following specific instructions or smoothly moving targets. Gaze360 [20] captured videos of 238 subjects, and five indoor (53 subjects) and two outdoor (185 subjects) locations, over nine recording sessions using a Ladybug5 camera. RT-BENE [173] was released based on the RT-GENE [112] dataset collection. RT-GENE was captured using mobile eye tracking glasses, and contains recordings of 15 participants, 9 male, 6 female; two participants were recorded twice. TabletGaze [16] was captured on a mobile tablet setup using 51 subjects of varying race, gender, and glasses prescription. GazeCapture [17] contains data from over 1450 subjects, recorded using crowdsourcing: 1249 subjects used iPhones, whereas 225 used iPads. MPIIGaze [53] was collected with a laptop setup involving 15 participants during natural everyday laptop use. EyeDiap [223], was collected from 16 subjects (12 males and 4 females) using a Kinect sensor and an HD camera.

The characteristics of these datasets largely determine the kinds of applications they are suitable for. In Figure 15, we describe some important characteristics of the dataset and their application suitability.

Image samples, for example, would make a dataset with them more suitable for image-based gaze estimation and not video-based. If a dataset includes on screen gaze annotation, then it is most applicable for 2D gaze estimation. The resolution and quality of input images impacts the model’s resources. Lower resolution would perform much better with lighter deployment resources, a shallower model architecture, and a lower model inference speed. Having a larger number of data samples is suitable for a deeper model, but could negatively impact resource-constrained deployment, for instance, on mobile devices. However, for training any model, error decreases considerably as the number of samples is increased, showing the significance of collecting much data.

Important progress has been made for large-scale dataset collection in the recent literature. First, collecting large-scale images required by supervised gaze estimation methods through synthesis became popular because it is time consuming and tedious to label naturalistic images [225]. Even though learning-by-synthesis has the potential to save time and resource for data collection, the desired performance for highly accurate gaze estimation is still not achieved by this method due to the different illumination distribution. While early attempts in [226,227,228] trained models to improve the realism of the synthetic images, a different approach was presented to purify naturalistic images in [225,229,230]. The authors adopted the style transfer technique proposed in [227] to convert an outdoor naturalistic image distribution into indoor synthetic images. Their proposed method utilizes three separate networks: a coarse segmentation network, a feature extraction network, and a loss network. The coarse network takes in two images (naturalistic image and a synthetic reference style image) with their mask as the input for segmentation into the pupil (white) and the iris (red), followed by a feature extraction network that extracts gaze direction and pupil center position as the image content of the naturalistic image. Then, through a standard perception loss network, they retain the information of the naturalistic image and the distribution of the synthetic image (i.e., image color structure and semantic features) to the fullest extent in the output image. Using the proposed method, they purified the MPIIGaze [53] dataset. Experimental results have shown how purifying naturalistic images for training an appearance-based gaze estimator leads to improved performance compared to some state-of-the-art techniques.

Second, a broader range of settings for datasets was addressed. The authors of [20,190] proposed a broader range of settings for datasets in order to increase robustness to a larger variety of environmental conditions, as previous datasets are limited to relatively narrow ranges of head poses and gaze directions, such as the frontal face setting. In [190], the authors collected and made public the ETH-XGaze dataset of varied viewpoints, lighting, extreme gaze angles, resolutions, and occluders, such as glasses. Their dataset provided maximum head poses and gaze in horizontal (around yaw axis) and vertical (around pitch axis) directions in the camera coordinate system: ±80°, ±80°, and ±120°, ±70° respectively. The image resolution is 6000 × 4000. Similarly, the dataset [20] in provides ±90°, unknown, and ±140°, −50° for maximum head poses and gaze in the same axis with image resolution of up to 4096 × 3382. These attempts have paved the way towards robust gaze estimation for unconstrained environmental conditions, particularly with respect to lighting and coverage of extreme head pose and gaze, which are critical for emerging gaze-based applications.

Benchmarks for Evaluating REGT Performance

At present, five evaluation protocols are available in the literature, two of which are popular:

• Cross-dataset Evaluation

A given gaze estimation method is trained on one dataset (e.g., dataset A) and tested on another dataset (e.g., dataset B). These data samples may have the same or different characteristics, which correlates with the purpose of this method, which is to show the generalization capabilities of a given gaze estimation method. Examples of gaze estimation research that have utilized this evaluation method can be found in [20,38,53,112,231].

2.

Within-dataset Evaluation

Examples of this evaluation can be found in [117]. The gaze estimation method is trained and tested with the same dataset, which is randomly separated into training and test samples. These samples may also have the same or different data characteristics. However, the generalization capability is verified less with this method compared to the cross-data evaluation method.

Recent studies have proposed subject-specific, cross-device, and robustness evaluations. These new evaluation methods have focused on specific data and device characteristics for evaluating emerging gaze estimation methods as follows:

3.

Subject-specific Evaluation

This evaluation technique became popular due to the recent attention paid to subject-specific gaze estimation [185]. The evaluation is based on sampling a little subject-specific training data to improve the adaptation of a generic gaze estimator to a specific person from a few samples due to insufficient data samples. Other works [16,19,232] have also used this method.

4.

Cross-device Evaluation

Emergence of cross-device training necessitated cross-device evaluation. This method evaluates a given gaze estimation method across multiple devices to measure cross-device performance. Researchers in [40] demonstrated the significance of this evaluation method on five common devices (mobile phone, tablet, laptop, desktop computer, smart TV).

5.

Robustness Evaluation

Researchers in [190] proposed assessing the robustness of gaze estimation methods across head poses and gaze directions. The authors have stressed its importance to previous gaze estimation methods that only report the mean gaze estimation errors for cross-dataset, within-dataset, and person-specific evaluations on insufficient data samples that do not cover a wide range of head pose and gaze direction. They also argued that knowing the performance of a gaze estimation method with respect to robustness across head poses and gaze directions is important, since a method with a higher overall error might have a lower error within a specific range of interest. They further demonstrated and compared recent methods to assess their robustness and reported gaze estimation error distribution across head poses and gaze directions in horizontal and vertical directions to facilitate research into robust gaze estimation methods.

4. Applications

The use of gaze applications as a solution has expanded over time. We present how gaze application usage has evolved over time across fields of human endeavors and deployment platforms in Figure 16. The authors of [233] presented an early review on gaze applications. They broadly classified gaze application usage into two fields, diagnostic (see examples in [234,235,236]) and interactive (see examples in [237,238]). Later reviews in [63,91,239] presented wider usage for gaze applications, which included that by REGT systems, augmented reality systems [240,241,242,243,244], human behavior analysis [245,246,247,248,249,250], and bio-metric authentication [251,252,253,254,255]. Recently, the usage of REGTs has gotten even wider with emerging usage in privacy-aware interactions.

We have identified emerging eye solutions and classified them under the existing categories, such as device interactions (IoT smart home control [256], semi-autonomous driving [257,258], artistic drawing in robots [259,260,261,262]), human behavior analysis (confusion prediction [263,264,265], intention extraction [266,267], driver’s attention [30,32,268,269,270], detecting personality traits [271]), medical support (medical image interpretation [272,273,274,275], patients support [276]), augmented reality (social games [277,278], virtual space control [279]), and privacy issues (privacy-aware eye tracking [280,281,282,283], gaze-touch authentication [284,285,286,287,288]). The arrow pointers indicate a solution application in a field and across deployment platforms.

The emerging solutions are responsible for the recent shift in requirements for REGT system deployment on desktop platforms as a decades old practice [68] to those for dynamic platforms such as handheld devices and wearables [155]. In addition to applications, the large number of REGT systems produced today is research based, most of which are fabricated to suit research objectives. Several commercial REGT systems are also available on the market. Recently, ref. [289] ranked the top 12 REGT companies based on the number of patents and equipment testing. Tobii [290], SensoMotoric Instruments [291], now acquired by Apple Inc, and EyeLink [292], were ranked as leading companies. Several open-source REGT systems were released for free online to give new researchers a starting point, ensuring the sustainability of this research field. We have classified and described a few from academia and industry in

Table 5. Available open-source REGT software.

Method	Provider	Language	Description
Passive-light	Itracker [17]	Python, Matlab	A CNN based eye tracker, which runs in real time (10–15 fps) on a modern mobile device
	RecurrentGaze [177]	Python	Based on a fusion of CNN-RNN
	NNET [293]	-	ANN based eye tracker implementation for iPad devices
	EyeTab [294]	Pthon, C++	Webcam model-based approach for binocular gaze estimation
	Opengazer [295]	C++, C	Based on the Viola-Jones face detector, that locates the largest face in the video stream capture from PC webcam
	TurkerGaze [296]	JavaScript, HTML	A webcam-based eye tracking game for collecting large-scale eye tracking data via crowdsourcing
	Camgaze [297]	Python	Binocular gaze estimation for webcam
	ITU gaze tracker [298]	-	Based on remote webcam setup
	CVC ET [299]	C++	Enhanced Opengazer with head repositioning feature which allows users to correct their head pose during eye tracker usage in order to improve accuracy.
	xLabs [300]	-	Webcam-based eye tracker, built as a browser extension for Google Chrome.
	Gazepointer [301]	C#, HTML	Windows-based web camera gaze estimation
	MyEye [302]	-	Gaze-based input designed for use by people with amyotrophic lateral sclerosis (ALS), a neuromuscular disease.
	NetGazer [303]	C++	Port of Opengazer for the Windows platform
Active-light	OpenEyes [304]	Matlab	Based on infrared illumination.
	Ogama [305]	C#.NET	Uses infrared ready webcams
	GazeParser [306]	Python	Based on infrared illumination. Python.
	Pygaze [307]	Python	Wrapper for EyeLink, SMI, and Tobii systems.

.

It is likely that most of the passive light open-source applications will be supported and remain active much longer than the active light ones, because the active light open-source applications have been made available for a long time with no further support. Recently, ref. [308] has been providing implementation files (i.e., code, models, and datasets) for open-access academic papers.

5. Summary

We have discussed REGT systems that process both continuous and static data for eye gaze estimation. We focused on REGT’s remote applications for PCs, laptops, and mobile devices, and have presented a thorough evaluation of the past and recent research. We have classified and discussed our evaluation in three parts: hardware, software, and applications. In our evaluation, we have compared key research trends in REGT to show how its research has changed over time. These key research trends have been summarized in

Table 6. Summary of research trends in REGT. The cited works present examples of the points stated.

Component	Past Research Trend (<2015)	Recent Research Trend (>2015)
Hardware setup	Gaze tracking on modified devices was a common practice [41,48,267] Relied on NIR illumination, quality cameras, high computational power, and a defined subject position [26,28,41,98,100,127] Multiple cameras used in device setup to address the change in head position [63,64]	Unmodified devices utilized for gaze tracking [16,43] Gaze interaction from arbitrary positions and orientations [52], multiple displays [40] Utilization of low-resolution cameras for visible light gaze tracking in both indoor and outdoor [17,61,92,112]
Image acquisition	Gaze data was acquired via video camera [61,110], and from dataset e.g., video [213] or image [215,218]	The acquisition of large quantity images required for training deep learning gaze estimators are generated by synthesis [93,117,221]
Feature extraction	Largely depended on high quality images to extract rich features e.g., CHT [136,137,138], LLD [139], Subpixel [140] Extraction of features from low quality images commonly used hand-engineered classifiers e.g., haar-cascade [141,142]	Scalable classifiers was used to extract visible and deep features based on Machine learning, e.g., Genetic algorithms [160], SVM [36], and Deep leaning e.g., R-CNN [169,170], and YOLO [171,172]
Gaze mapping	Naming schemes for gaze estimation methods were broadly categorized into two with so much ambiguity [87]. Supervised learning was used for gaze mapping by early appearance-based methods [53,155,156] Explicit calibration was largely required for gaze mapping [24,41] Gaze mapping techniques were driven by face or eye model [61]	Naming schemes for gaze estimation methods regrouped into 3 broad categories [71,93] Directly regress from eye image to gaze direction using machine, or deep learning algorithms [89,108,112,211] Unsupervised learning for appearance-based methods demonstrated [157,158,159] Calibration requirement became implicit (i.e., less required), researchers achieved gaze mapping through automatic calibration procedures [119,183] Gaze mapping techniques were driven by large and well annotated data [17,93] Proposal to have a standardized evaluation metric for gaze estimation methods that ensure accurate validation and comparison [68,190]
Dataset	Data collection was largely done under controlled laboratory conditions [124,213,214,215,216,217,218] Datasets were limited to the frontal face setting that provides only narrow range of head poses and gaze directions [215,218,223]	Presentation of datasets with data from the real-world [53,223] Attempts to purify naturalistic images through style transfer [225,229,230] Cross-dataset evaluation for utilizing several datasets and domain adaptation for a more generalized, robust gaze model validation [20,38,231,232] Attempts to create a broader range of settings for datasets [20,190] to ensure robustness to a larger variety of conditions such as varied viewpoints, extreme gaze angles, lighting variation, input image resolutions
Application	Early gaze applications were used as diagnostic [233,234,235], and interactive [236,237] tools Desktop-based platforms for deployment was a commonly used [68]	REGTs have been used recently in Social games [271,272], and Privacy-aware interactions [273,274,275,276] Gaze application deployed on dynamic platforms e.g., handheld devices, and wearables [155]

. We mean by past progress, earlier information provided in the literature before 2015, and by recent progress, new information provided in the literature after 2015. Although our literature search expanded over seven publishers (i.e., IEEE, ACM, Springer, MDPI, Elsevier, BOP, and PLOS) of academic journals and conference papers, we only cited some, not all literature relevant to the components discussed in this paper. The search for relevant literature was done using the keywords: remote eye gaze tracking; hardware setup for eye gaze tracking; software process (techniques and datasets) for eye gaze tracking; application for eye gaze tracking. In some cases, eye gaze tracking was replaced with gaze estimation.

In the past five (5) years, efforts by researchers have been made to eliminate the inherent challenges that have made highly accurate gaze estimation results difficult. Various attempts at appearance-based gaze estimation were focused mainly on addressing:

• Feature detection and extraction: The accuracy of gaze estimation depends largely on effective feature detection and extraction. Recent learning-based feature detection and extraction have brought remarkable progress for appearance-based gaze estimation, as researchers have continued to demonstrate its effectiveness [170,171,172]. However, the issue of deviations in location and degree of eye openness remains a challenge for this method.
• Variability issues in data collection: Collection of relevant data of varied characteristics has been a challenge for appearance-based gaze estimation. Recent attempts by researchers to create and make available robust datasets have been presented in [20,112,190]. However, the reported accuracies on these datasets are still not satisfactory, and thus require more attention.
• Subject calibration: Gaze estimation methods require a subject calibration process for different individuals. Appearance-based methods have demonstrated less stringent processes of calibration, where only a few samples of calibration data are required before they work for other individuals [185,232]. Even with this development, an appearance-based gaze estimation method that tries to either make the calibration process less stringent or absolutely remove it does so at the expense of estimation accuracy. Thus, the need to develop absolute appearance-based calibration-free methods with very good estimation accuracy remains a challenge.
• Head fixation: Although several appearance-based gaze estimation methods have been able to perform well with considerable accuracies, without requiring fixed head poses [93,108,111,113,211], most of them still can only handle small head movements to achieve high accuracy [110,116]. As such, more robust methods that freely allow for head movement are still sought.

Notwithstanding, more recent attempts in the past 2–3 years have focused on appearance-based deep learning methods. The use of CNNs, RNNs, and GANs has shown greater promise in addressing quite effectively some of the challenges mentioned for appearance-based methods. However, there are still factors negatively influencing overall performance of appearance-based deep learning methods. These factors include eye blink, occlusion, and dark scenes. Eye blink and occlusion are caused by obstructions which affect eye feature detection. Eye blinking is a temporary closure of both eyes, involving movements of the upper and lower eyelids. Occlusions may occur from eye illness, aging, and objects such as lenses and glasses. Dark scenes produce dark images, which reduce the sensitivity of local features, thereby affecting effective eye feature detection. Other concerns generated by appearance-based using deep learning that can lower accuracy are:

• Model input: Researchers are trying to determine if the choice of model input has any effect on the performance of the model. For instance, does a model that uses both eyes perform better than one that uses one eye in terms of accuracy? We have seen several attempts with models that use one (single) eye [188,191], both (double) eyes [17,53,111,112], or full face images [114,190], augmented with other inputs, such as head pose and face grid information, to estimate gaze. At the moment, there is not a clear understanding of how to determine the best or standard model input for deep learning gaze estimation methods. It is done at the discretion of researchers, and still remains a choice of convenience for the models proposed.
• Resolution of input images: Does training and testing a model with different input image sizes improve performance? In [38,53], the model trained with images of one size achieved much worse results than the one trained on images of multiple sizes. On the other hand, Zhang et al. [190] demonstrated an improved performance when training and testing with a single image size at high resolutions. Based on this, researchers are suggesting methods to handle training of cross-resolution input images.
• Model augmentation: Are there working step-up techniques employed for improving gaze estimation errors on public datasets from recent years? Early attempts used a single-region CNN model for gaze estimation, as demonstrated in [53,114], but recent step-up attempts are opting for multi-region CNNs [17,112,173,188], where each input is processed through a separate network to provide each network with a higher resolution of the input image and improve the processing capabilities of the networks for a better performance of the overall model. Another step-up technique is to improve early CNNs’ poor generalization performances due to appearance and head pose variations using adversarial learning approaches. As recently attempted by researchers [179,180,181,182], the basic idea is to improve the generalization performance of a traditional CNN-based gaze estimator by incorporating adversarial nets with ConvNet. The adversarial nets are commonly used to improve the input image fed into the ConvNet to estimate gaze.
• Data annotation: Training a deep learning gaze estimation model by supervision requires data annotation for the model to learn the task. Grid-based annotation has been widely adopted for this method. How can its effectiveness be judged? It divides the screen into horizontal $w$, vertical $h$, and grids $g$, which produces $(w\ast h)/g$ grids. This usually causes more calculations and affects the accuracy of the model’s output. Considering these drawbacks, bin-based annotation was proposed in [174] to control the number of labels on the gaze image. It divides the screen into horizontal and vertical bins and set the bin size as $b$, making $(w+h)/b$ bins. The grid-based annotation yields more annotation data, whereas the bin-based annotation yields less annotation data but requires more processing steps. Due to lack of sufficient reports on alternative annotation techniques, it is difficult to judge the effectiveness of the grid-base annotation.

6. Conclusions

This paper has presented a thorough comparative evaluation on REGT research. Our evaluation has discussed key components and research changes over time. In addition, issues inherent in the recent state of REGT research have been outlined. The prevalence of the issues outlined negatively impacts the performance and accuracy of the recently proposed gaze estimation methods which utilize deep learning. This has raised new concerns about how to further enhance the recent appearance-based deep learning methods through continued research efforts for developing new algorithms and augmentation techniques that will ensure REGT systems become more useful throughout fields of human endeavor in the very near future.