Bi-Directional Gaze-Based Communication: A Review

Abstract

Bi-directional gaze-based communication offers an intuitive and natural way for users to interact with systems. This approach utilizes the user’s gaze not only to communicate intent but also to obtain feedback, which promotes mutual understanding and trust between the user and the system. In this review, we explore the state of the art in gaze-based communication, focusing on both directions: From user to system and from system to user. First, we examine how eye-tracking data is processed and utilized for communication from the user to the system. This includes a range of techniques for gaze-based interaction and the critical role of intent prediction, which enhances the system’s ability to anticipate the user’s needs. Next, we analyze the reverse pathway—how systems provide feedback to users via various channels, highlighting their advantages and limitations. Finally, we discuss the potential integration of these two communication streams, paving the way for more intuitive and efficient gaze-based interaction models, especially in the context of Artificial Intelligence. Our overview emphasizes the future prospects for combining these approaches to create seamless, trust-building communication between users and systems. Ensuring that these systems are designed with a focus on usability and accessibility will be critical to making them effective communication tools for a wide range of users.

1. Introduction

Artificial Intelligence (AI) has become a pervasive and influential field, with supporting systems playing an increasingly critical role in various domains [1]. These systems are particularly evident in the health and medical sectors, where significant advancements are being made [2], e.g., as an aid in diagnosis [3] or during surgery [4]. AI has notably enhanced accessibility for individuals with disabilities through innovations in rehabilitation [5,6,7], the development of exoskeletons [8], the integration in wearables [9,10], and advancements in wheelchair navigation technologies [11,12]. Beyond healthcare, AI is increasingly shaping other fields, including gaming [13], domestic automation [14], and education [15,16]. As we integrate AI more into our daily life and certain tasks become more reliant on it, how we interact with and how we expect AI to respond becomes important to our feelings of trust in these systems.

Trust is an important factor in the adoption and effectiveness of AI. This trust is influenced by several key elements, including human characteristics and abilities, the performance and attributes of the AI itself, and the specific context in which tasks are performed [17]. To ensure that AI systems can excel across these dimensions, it is essential to provide clear and precise communication of tasks. However, communicating with AI differs significantly from traditional human-to-human interactions, requiring a deeper understanding of these distinctions and the unique challenges they present [18]. These distinctions include the functional role of AI in communication, the relational dynamics it fosters between humans and technology, and the metaphysical implications of the blurring boundaries between humans, machines, and the nature of communication. Exploring these differences is vital for enhancing the effectiveness and reliability of AI systems [19].

Effective communication with a system often requires that the input data be formatted and structured according to the specific requirements of the system. This ensures that the machine can interpret and process the data correctly. However, achieving this level of precision can be time-consuming and complex, making it challenging for the average user who may lack the expertise or resources to prepare data in the required format consistently. This difficulty highlights the need for more user-friendly communication methods that do not rely heavily on complex data structuring. While Natural Language Processing (NLP) models enable communication with machines through text input [20,21,22], which can promote accessibility, they can also have drawbacks like computational demand [23,24]. More importantly, language-based input modalities exclude a proportion of the population that may not be able to effectively communicate with speech or keyboard input.

System communication can often be a barrier for individuals with motor impairments, where devices like keyboards may be cumbersome, slow, or even physically inaccessible for some individuals. Speech impairments can lead to poor understanding of commands, which can affect usage [25,26]. Persons who experience deteriorating motor abilities from neurodegenerative disorders, like ALS or stroke, can also struggle with various communication modalities. An overview by Story et al. [27] found patients with motor impairments were not taken enough into account regarding the usability of devices (e.g., human factors, ergonomics). One modality for more implicit and natural communication that has been refined over the past 40 years is gaze-based interaction with systems. Gaze-based interaction can offer a more intuitive and resource-efficient way to engage with machines, potentially addressing some challenges associated with text-based input. This paper explores the state of the art of using gaze as an alternative communication method.

2. Structure of the Review

In this review, we consider communication with a system as a two-way process: from the user to the system and back. To take this into account, we have organized the overview around key research questions that are central to understanding and improving this interaction. We start in Section 3 with the question: How is gaze data used in interactive systems, especially in immersive environments? We provide an overview of existing methods for capturing, processing, and analyzing gaze data, with a focus on applications in immersive scenes. In the next Section 4, we look at the question: How can gaze be used effectively as an interaction tool, and what challenges does this pose? Here, we examine general applications of gaze-based interaction, followed by a detailed look at gaze-based object selection. We then assess the research on adapting these methods to immersive environments and consider the particular challenges that arise in such contexts. In Section 5, we explore the question: How can gaze be used to predict user intentions? Since effective communication requires an understanding of user intention, we start with general approaches to gaze-based intention prediction and then focus on studies in which gaze is used for specific applications of intention recognition. In the following Section 6, considerations are made: What types of feedback channels from the system to the user are most effective? We examine different feedback mechanisms and analyze studies on their strengths, limitations, and impact on the user experience. Finally, in Section 7, we address the overarching question: How can these gaze-based techniques be integrated to create a system that encourages intuitive, responsive communication? We discuss how combining these approaches can create a system that gives users a stronger sense of being understood and engaged.

3. Gaze Estimation and Usage in Immersive Scenes

For effective gaze-based communication, precise gaze estimation from a seemingly ambiguous signal is non-trivial. This task is challenging due to the ambiguity of gaze signals, which respond to subtle head movements, lighting variations, and individual differences in eye physiology—factors that can affect the robustness and accuracy of eye-tracking systems [28]. Here, we provide a brief introduction to gaze estimation techniques and the pipeline for their integration into applications. We illustrate our general overview of the pipeline for gaze estimation and common applications in Figure 1.

Gaze estimation is commonly achieved through eye tracking technology, which involves using sensors, such as cameras, to capture and analyze eye movements to determine gaze direction. In typical screen-based setups, remote eye trackers are often employed. These devices are usually mounted on a screen or desk and include systems like RemoteEye [29]. In dynamic environments, such as outdoor settings or Virtual Reality (VR), head-mounted eye trackers are preferred. These devices generally use cameras to track both eyes, as seen in systems like PupilLabs [30]. Some advanced VR headsets, such as the HTC Vive Pro Eye, incorporate integrated eye tracking sensors within the headset itself [31]. Regardless of the system used, proper calibration is crucial to ensure accurate gaze estimation based on eye orientation [32,33,34,35]. During the calibration phase, the system establishes a relationship between the raw, uncalibrated data from the device and the real gaze points. To achieve this, users are usually asked to follow an easily recognizable target, such as an ArUco marker, so that the system can match the gaze data with the exact target positions. Calibration is essential to account for individual differences in eye anatomy and positioning and to ensure accurate gaze estimation.

The output of gaze estimation is typically represented in one of two forms: gaze coordinates, which are x and y coordinates relative to a plane (e.g., Monitor or scene camera), or gaze vectors, which provide the direction in three-dimensional space (x, y, z) or as horizontal and vertical angles. This data allows for the determination of where participants are looking and enables the analysis of eye movements and their events. The most well-known eye movement events are fixations and saccades. Fixations occur when the eye remains stable and focused on a specific point, indicating where attention is directed. In contrast, saccades are rapid movements of the eye as it shifts focus from one location to another, playing a crucial role in scanning the visual environment [36]. Another important eye movement event is smooth pursuit, characterized by slow, continuous tracking as the eyes follow a moving object. There are other eye movement events as well as ways of processing the pupil signal, though they are out of the scope of this current review.

Eye movement events can be detected using threshold-based algorithms like Identification by Dispersion Threshold (I-DT) and Identification by Velocity Threshold (I-VT). The I-DT algorithm effectively identifies fixations by measuring gaze dispersion, classifying a movement as a fixation when dispersion remains below a set threshold, thereby sensitively detecting points of stable focus. On the other hand, the I-VT algorithm detects saccades by analyzing eye movement velocity, classifying a movement as a saccade when velocity exceeds a predefined threshold, thereby effectively capturing rapid shifts in gaze [37]. In addition to these methods, probabilistic models like Hidden Markov Models (HMMs) provide a more advanced approach for identifying fixations and saccades, treating them as hidden states and using velocity as an observed feature to account for the probabilistic nature of eye movement sequences [37]. These algorithms are also applicable in VR environments, where they help to accurately track eye movements within immersive settings [38,39,40]. Smooth pursuits are distinct from saccades and fixations and can be detected using algorithms that analyze shape features, such as those developed by Vidal et al. [41]. Recent advances in eye movement event detection have expanded beyond traditional methods. Santini et al. [42] introduced a Bayesian model that detects fixations, saccades, and smooth pursuits, even in online settings. Fuhl et al. have further advanced the field with deep learning models, including tiny transformers, for highly accurate event classification [43]. A comparative study by Andersson et al. [44] highlights the strengths and weaknesses of various algorithms, guiding the selection of the most suitable method for specific applications.

From the fixation and saccade events, metrics can be derived, such as the mean fixation duration or saccade amplitudes, which provide deeper insights into visual attention and behavior. These features can be used for statistical analysis to infer cognitive states [45], distinguish between individuals [46], identify the task being performed [47], or serve as indicators of usability [48]. The analysis of fixations and saccades can reconstruct the user’s scanpath—the temporal patterns they generate while exploring a scene. Scanpaths vary depending on the task and the interplay of top-down context and bottom-up saliency [49,50] (Figure 2). Additionally, metrics can be derived from eye movements, such as the frequency of revisiting specific Areas of Interest (AOI)—distinct regions within a scene that are pre-defined or dynamically identified based on their relevance—which indicate which parts of the scene are examined in detail [51,52,53]. The calculation of eye-tracking metrics is essential to gain quantitative insights into visual attention, cognitive processes, and user behavior. This allows researchers to interpret how people absorb visual information and react to different tasks or stimuli.

With advancements in gaze estimation technology, particularly in head-mounted devices, the use of gaze tracking in Virtual Environment (VE) is becoming increasingly prevalent. VR immersive nature offers a unique platform for examining gaze interactions [55], enabling applications in areas such as gaming [56,57], training simulations [58,59], and medical diagnostics [60]. This environment also presents new challenges, such as dealing with motion sickness, navigating complex 3D spaces, and ensuring accurate calibration, which are driving ongoing research and development [61]. Compared to traditional 2D screen-based gaze tracking, VR offers richer insights into user behavior by capturing interactions in a fully immersive setting. In VR, the precise control of an AOI within the scene simplifies the automatic detection of when a user is gazing at an object, a task that is more challenging in other mobile eye tracking systems [35,62,63]. Future directions may include improving accuracy, integrating AI for predictive gaze analytics, and enhancing user experiences by enabling more natural and intuitive gaze-based interactions [64,65]. We further overview the current trends in gaze-based interaction methods and how they can support users in immersive environments.

4. Gaze-Based Interaction

Gaze-based interaction has a rich and extensive history, thoroughly reviewed in several survey papers [66,67]. One of the earliest and most futuristic depictions of a gaze-controlled, immersive environment is from [68]. He envisioned a room of multiple monitors where “…Eye—tracking technology, integrated with speech and manual inputs, controls the display’s visual dynamics, and orchestrates its sound accompaniments”. Since then, research has furthered the idea that eye movements can act as output to control tasks in real time. The gaze data collected from eye trackers is utilized as an input method for user interfaces, offering an intuitive and hands-free means of interaction [69]. Additionally, this form of interaction allows for the evaluation and training of specialist knowledge through task-specific tools that use gaze data [50,70].

4.1. Gaze-Based Interaction with Screen Applications

Gaze-based interaction offers a powerful and intuitive way to engage with screen-based applications. It allows users to control and interact with digital interfaces simply by looking at specific areas of the screen. This method not only enhances accessibility but also provides a hands-free alternative for navigating and operating software, making it particularly valuable in scenarios where traditional input devices like keyboards and mice may be impractical or inefficient. In this subsection, we will explore the various techniques and applications of gaze-based interaction within screen environments, highlighting both the potential benefits and challenges associated with this technology.

A significant application of gaze-based interaction is using gaze as an input method for sequences of letters and numbers, such as writing text [71,72,73,74,75]. Gaze-based text entry systems, like those used for typing or selecting characters on a virtual keyboard, have shown considerable promise. For instance, gaze can be effectively utilized to write by selecting letters and numbers on an on-screen interface, though it often requires users to undergo a training period to achieve proficiency. Research demonstrates that with practice, users can significantly improve their typing speed using gaze-based systems. Tuisku et al. conducted a study where participants practiced gaze-based text entry over ten sessions, and the results were compelling: participants increased their typing speed from 2.5 words per minute to 17.3 words per minute using the Dasher [73] text entry system [76]. This increase highlights the potential of gaze-based writing as a viable alternative to traditional input methods, especially for users who might benefit from hands-free interaction. Similarly, gaze-based systems have been explored for secure tasks like entering PINs, where the user’s gaze can be used to select digits without physical input, offering both convenience and a degree of security [77,78]. These applications demonstrate the versatility of gaze as an input modality and its potential to improve user interaction in various digital contexts.

Another valuable application of gaze-based interaction is navigating hierarchical systems, where users can traverse complex menu structures using only their gaze. An exemplary system designed for this purpose is pEye [79,80], which facilitates hierarchical navigation through a rotary interface specifically optimized for eye tracking. This interface relies on two key parameters: pie segmentation, which divides the circular menu into selectable slices, and menu depth, which allows for multiple layers of hierarchical options. The design of pEye enables users to select items from these segmented slices, moving through the layers of the hierarchy with ease. The system is particularly effective for tasks that involve complex decision trees or nested options, where traditional input methods might be cumbersome. In a study conducted by Urbina et al. [81], the effectiveness of this gaze-based rotary interface was tested. The study demonstrated that users could accurately and quickly navigate through hierarchical menus with up to six slices across multiple depth layers. This finding underscores the potential of gaze-based navigation to streamline interactions in hierarchical systems, making it a powerful tool for both everyday users and individuals with specific accessibility needs.

Gaze-based communication in screen applications primarily involves using eye movements to select or interact with objects on a screen, where the gaze itself serves as a pointer to target specific elements. One common method to confirm a selection is the use of dwell time, a technique introduced by Jacob [82]. Dwell time refers to the duration for which a user must fixate on an object before it is registered as a selection. This method can also serve as a metric for measuring user interest, with longer fixations indicating greater interest or intention [83]. However, the reliance on dwell time brings about a significant challenge in gaze-based communication, known as the Midas Touch problem. This issue arises because, in gaze interaction, not every glance is intended to trigger an action; people naturally scan their environment, and these incidental gazes can lead to unintended selections. The Midas Touch problem thus refers to the difficulty of distinguishing between intentional and unintentional eye movements, which can result in a frustrating user experience. To address this challenge, various strategies have been developed (Figure 3). One common solution is to carefully adjust the dwell time threshold to balance responsiveness and accuracy. If the dwell time is too short, it may lead to accidental selections, exacerbating the Midas Touch problem. On the other hand, if it is too long, it may cause delays and make the interaction feel sluggish. Optimizing dwell time is crucial for ensuring that gaze-based systems are both efficient and user-friendly, allowing for accurate and intentional communication without the drawbacks of unintended actions.

An alternative to relying on dwell time for gaze-based selection is to incorporate head gestures as a means of confirming the user’s intention. This approach allows users to make selections with their gaze while using specific head movements to validate or execute those choices. Špakov and Majaranta [84] conducted a study exploring the effectiveness of various head gestures in conjunction with eye gaze for different types of tasks. In their research, participants tested a range of head gestures, such as nodding, turning, and tilting, each mapped to specific actions. The study found that different head gestures were better suited for different tasks. Participants reported that nodding was the most intuitive and effective gesture for confirming selections, providing a natural and efficient way to indicate a choice. Turning the head was preferred for navigation tasks, such as scrolling or moving through a menu, as it felt more aligned with the directional nature of such tasks. Finally, tilting the head was found to be useful for mode switching, allowing users to easily shift between different interaction modes or settings. This combination of gaze and head gestures offers a more nuanced and flexible interaction method, reducing the reliance on dwell time and addressing some of its inherent limitations, such as the Midas Touch problem. By decoupling the selection process from gaze alone, this method provides users with greater control and reduces the likelihood of accidental selections, enhancing the overall user experience.

Another approach to object selection in gaze-based interaction involves using the Pearson correlation between a user’s gaze and the movement of objects within a scene. This method was extensively explored by Vidal et al. [85], who developed a technique to calculate the correlation between the user’s gaze direction and the trajectories of multiple moving objects. In their approach, the system continuously monitors the gaze and compares it with the motion paths of all objects in the scene. If the correlation value for any object exceeds a predefined threshold, the object with the highest correlation is automatically selected. This method allows for dynamic and precise selection, particularly in environments where multiple objects are in motion. Building on this concept, Esteves et al. [86] introduced a novel mechanism for selecting items on a smartwatch using gaze in combination with object orbits. In their system, when a user focuses on an object, an orbit—essentially a circular or elliptical path—appears around it. The user can then confirm their selection by maintaining a gaze that correlates with the movement of the orbit. The system calculates the Pearson correlation between the gaze path and the trajectory of the orbit, and if the correlation is higher than a threshold value, the selection is confirmed. This method not only provides a robust way to ensure that selections are intentional but also enhances interaction with small devices like smartwatches, where traditional input methods may be cumbersome. By coupling gaze dynamics and object motion, these techniques offer an intuitive and efficient way to interact with complex visual environments.

4.2. Gaze-Based Interaction in Immersive Scenes

While many gaze-based interaction systems have traditionally focused on scenarios displayed on computer screens, the emergence of Extended Reality (XR)—including Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR)—has shifted research toward interactions within fully immersive virtual environments. These immersive scenes present unique challenges and opportunities for gaze-based interaction, as users are no longer confined to 2D screen interfaces but are instead navigating complex 3D virtual spaces. Moreover, the virtual realm allows for highly controlled, yet dynamic environments, where potentially dangerous scenarios can be played out (e.g., driving [87], surgical training [88,89], security training [90], etc.). A systematic review by Monteiro et al. [91] highlights that in immersive environments, interaction is most commonly driven by voice commands, followed by head and eye-gaze control. However, there has been a notable increase in research focusing on the potential of eye gaze as an interaction modality, especially in recent years. Still, one of the main challenges in these immersive environments is the Midas Touch. To address this, researchers often combine gaze with other input modalities. In many cases, gaze is used to select a target, while a secondary modality—such as voice—is employed to confirm the action. For instance, Klamka et al. [92] explored the use of foot pedals as a confirmation mechanism, showing that alternative methods can also be effective in avoiding accidental selections. Though this research successfully shows multi-modality for refined interaction, it does not fully address groups of users with specific mobility constraints.

In numerous scenarios, other input methods such as voice or physical controls are also not desired, thus gaze-based interaction can revert to using dwell time as the primary mechanism for target selection. In these cases, both head movements and eye gaze can be utilized for selecting a target, each offering distinct advantages. While eye gaze can precisely indicate a point of interest, head movements provide additional confirmation, adding robustness to the selection process. A comparative study by Qian and Teather [93] evaluated head-based versus gaze-based selection for target detection tasks. Their findings revealed that head-based selection generally outperformed gaze-based selection in terms of both selection time and error rates. Specifically, participants were able to make selections more quickly and accurately when using head movements as the primary input method, likely due to the larger and more controlled motions involved in head-based navigation, which reduce the risk of accidental selections. In contrast, a study by Blatterste et al. [94] produced different results, showing that eye-gaze selection was superior across multiple metrics, including faster task completion times, fewer head movements, and lower error rates. The researchers suggested that the quality of the eye-tracking signal might account for the contrasting outcomes between the two studies. As eye-tracking technology continues to improve, with greater accuracy and reliability, gaze-based input may become more effective for rapid and precise target selection, potentially surpassing head-based methods in specific contexts. This divergence in findings highlights the importance of considering both the context of the task and the quality of the tracking systems when choosing between head and gaze inputs for immersive interactions. In scenarios where eye-tracking is highly accurate, gaze may offer more natural and efficient interaction, whereas head-based inputs may remain preferable when precision is critical, or eye-tracking is less reliable.

Another approach to enhancing gaze-based interaction is to combine both head movements and eye gaze for more accurate and efficient target selection. This integration leverages the strengths of each modality, offering greater control and reducing errors. Sidenmark and Gellersen [95] explored this combination by developing three distinct interaction methods that utilized both head-supported versus eyes-only gaze (Figure 4a). In their study, participants were tasked with answering questions by selecting from four multiple-choice options. Their results show that eye and head techniques enable a new type of gaze behavior that gives users more control and flexibility in rapid gaze alignment and selection. Wei et al. [96] took this integration further by introducing a probabilistic model that combines head and eye gaze endpoints to improve target selection accuracy. Rather than simply using a binary selection based on dwell time or direct gaze, their approach calculated the probability of selecting a particular target based on the distribution of both head and gaze positions. By analyzing where both modalities converge, the system could estimate which target the user was most likely focusing on. This probabilistic approach reduced the likelihood of incorrect selections due to slight variations in gaze or head movement, making the interaction more reliable in complex environments with multiple selectable objects. These methods demonstrate that combining head movements and eye gaze can significantly improve interaction accuracy, speed, and user satisfaction, particularly in immersive scenes where precise control is essential. The integration of these modalities not only addresses the limitations of using gaze or head movement alone but also opens new possibilities for more fluid and intuitive interaction in virtual and augmented reality environments.

One recent interaction paradigm that is purely gaze-based is from Sidenmark et al. [97]. They proposed a method for selecting an object based on vergence (Figure 4b). This approach works on the principle that both eyes are directed in opposite directions to obtain a single, focused image. The angle between these two gaze vectors gives the vergence, which is an indicator of the distance of an object. By exploiting the vergence angle to accurately determine which object is being focused on, even in scenarios where the gaze direction is nearly identical for multiple objects located at different distances. Their approach enhances precision in identifying the selected target object, distinguishing it from other nearby objects that may be in close proximity but at varying depths.

Gaze-based interaction has also attracted great interest in the gaming industry, where it offers unique opportunities to enhance the gaming experience. Research comparing gaze-based and hand-based interactions in games has produced insightful findings. Hülsmann et al. [98] conducted a comparative study on these two modalities and found that while both gaze-based and hand-based interactions were effective, hand-based input generally provided greater accuracy and ease of use. Their study indicated that hand-based controls were more intuitive and precise, which is crucial for games requiring fine motor skills and rapid responses. With the increased accessibility and affordability of eye-tracking technology, further research has emerged exploring the potential of gaze-based interaction in gaming contexts. Luro and Sundstedt [99] compared eye-tracking with traditional hand-tracking methods and observed that, although aiming with the eyes tended to be less accurate compared to hand-based controls, it resulted in lower cognitive load and shorter task completion times. Their findings suggest that while gaze aiming may require some adjustment in terms of precision, it offers advantages in reducing mental effort and speeding up interactions, which could be beneficial in certain gaming scenarios. In a different approach, Lee et al. [100] explored the effectiveness of gaze-based targeting compared to using a physical controller in a baseball-throwing game. Participants were asked to aim at targets either by using a controller or by directing their gaze. The results revealed that while the accuracy of hitting the targets was comparable between the two methods, gaze-based interaction excelled in terms of task completion time and overall usability. This study highlighted the potential of gaze-based controls to streamline gameplay and improve user experience by making interactions more direct and less cumbersome. These studies collectively indicate that while gaze-based interaction in gaming may not yet match the accuracy of traditional hand-based methods, it offers compelling benefits in terms of reduced cognitive load and improved usability. As eye-tracking technology continues to advance, its integration into gaming is likely to offer more engaging player experiences, leveraging its strengths in reducing physical input and enhancing accessibility.

To summarize, using gaze to interact with the environment is a difficult task. To avoid the Midas Touch, different approaches are tested (see

Table 1. Overview of Studies on Gaze-Based Interaction Techniques in VE. Highlighting the use of eye, head, and hand movements across different tasks.

Publication	Eyes	Head	Hands	Task
[93]	X	X		head vs. gaze selection
[94]	X	X		head vs. gaze selection
[95]	X	X		head-supported gaze selection
[96]	X	X		probabilistic model based on head and gaze
[97]	X			vergence based selection
[98]	X		X	gaze vs. hand based interaction in games
[99]	X		X	gaze vs. hand based aiming in games
[100]	X		X	gaze vs. controller in baseball-throwing game
[101]	X			using gaze for common computer activities

). These approaches usually use gaze as a pointing mechanism and other modalities for confirmation. In this way, unwanted selection could be minimized, making gaze-based interaction more practical in dynamic scenarios such as games.

5. Gaze-Based Intention Prediction

In good communication, it is helpful to recognize what the other person wants. While clear signals allow the user to tell the system what they want to interact with, it is still necessary to decide what exactly should be done. The feedback can be used to communicate the intention of the system to the user or to provide the user with information that is helpful for their intention. In order to achieve this, however, the intention must first be recognized. In gaze-based communication, eye movements are the main means of doing this. As Yarbus [49] has already shown, the scanpath differs depending on the underlying intention. Nowadays, Gaze-based intention prediction has different fields of usage like driving scenarios [102,103], interaction with robots [104,105] or to interact with a VE [106]. A good summary of gaze-based intention prediction is given by Belardinelli [107]. This section gives a general overview of gaze-based intention prediction and how it can be used to communicate with an AI.

5.1. How to Use Gaze to Predict Intention

In intention prediction, a person’s intention is considered a latent, unobservable state that must be inferred through their observable actions. In this context, gaze-based intention prediction involves estimating a person’s intention by analyzing their eye movements. Gaze features such as fixation duration or dwell time are commonly extracted, and machine learning models are employed to classify these features. Common classifiers include Support Vector Machine (SVM) [108,109], random forests [47], logistic regression [106], and neural networks [102]. However, given that intention is a hidden state, machine learning models designed for estimating hidden states, such as Dynamic Bayesian Networks and Hidden Markov Models, have also been explored [110,111,112,113,114,115]. These models estimate the probability of an intention based on observable features, often by applying principles from Bayes’ theorem [116]:

$$\begin{array}{c}\hfill P\left(I\right|F)={\displaystyle \frac{P\left(F\right|I)·P(I)}{P\left(F\right)}}\end{array}$$

In this formula, I represents the intention and F represents the observed features. $P\left(I\right)$ is the prior probability of the intention, $P\left(F\right)$ is the probability of observing the features, $P\left(I\right|F)$ is the probability of the intention given the observed features, and $P\left(F\right|I)$ is the likelihood of observing the features given the intention. The goal is to find the intention that maximizes $P\left(I\right|F)$, making the most likely prediction based on the observed eye movements. Since the intention I is hidden, directly estimating $P\left(I\right|F)$ is difficult. However, it is not necessary to calculate $P\left(I\right|F)$ directly. Instead, Bayes’ theorem allows finding the intention that maximizes this probability:

$$\begin{array}{c}\hfill \underset{I}{arg\; max}P\left(I\right|F)=\underset{I}{arg\; max}{\displaystyle \frac{P\left(F\right|I)·P(I)}{P\left(F\right)}}\end{array}$$

Because $P\left(F\right)$ is constant across all intentions, it does not affect which intention maximizes the probability. Therefore, it is sufficient to maximize $P\left(F\right|I)·P(I)$, which considers both the likelihood of the features and the prior probability of the intention. Since the features F are observable, estimating $P\left(F\right|I)$ is feasible, allowing for accurate intention predictions using observed data and general prior probabilities.

5.2. Areas of Application

Gaze-based intention prediction has diverse applications in communication across different domains (Figure 5). One notable application is in human–agent interaction, where understanding a person’s intention is crucial, especially when dealing with ambiguous behaviors. Singh et al. [117] introduced a model for online human intention recognition that integrates gaze data with model-based AI planning. Their findings suggest that this approach could enhance human–agent interaction, particularly in scenarios where the person’s behavior might be uncertain, such as when they are being honest, deceitful, or semi-rational. This is even possible without directly classifying these behaviors, as gaze data can provide insights into the underlying cognitive processes and user behavior. This model could be instrumental in designing more adaptive and responsive systems that can handle these complex human behaviors.

Another significant area of application is in real-time interactions, particularly in immersive environments. Chen and Hou [118] conducted a VR experiment demonstrating that gaze data combined with hand-eye coordination data can be used to predict a user’s interaction intentions. This has promising implications for the development of predictive interfaces that can anticipate user actions, making interactions more seamless and intuitive. Similarly, Newn et al. [119] demonstrated that gaze can serve as a natural and intentional input in socially interactive systems. In their demo, they showed how gaze can be used by interactive agents to collaborate with humans by understanding and responding to their intentions, making the interaction more fluid and engaging.

In the field of assistive technology, gaze-based intention prediction holds significant potential for improving communication, particularly for individuals with impairments. Koochaki and Najafizade [120,121] explored how eye gaze patterns can be leveraged to predict a user’s intention to perform daily tasks. This approach can empower people with disabilities by enabling them to communicate their needs and intentions to assistive systems, thus improving their independence and quality of life.

Gaze-based intention prediction is also increasingly being utilized in human–robot interaction (HRI). For instance, Weber et al. [104,105] demonstrated how gaze-based intention prediction can be used to communicate a target to a robot, enabling more effective and natural interactions between humans and robots. This concept is further extended by Shi et al. [122], who proposed GazeEMD, a method for using gaze cues to communicate the intention to pick up objects in real-world scenarios. Such gaze-driven interactions are particularly beneficial in collaborative environments, where clear and efficient communication between humans and robots is essential. Dermy et al. [123] highlighted this in their work by developing a method that combines gaze with other modalities to control a robot in assisting a human during collaborative tasks. Their approach underscores the value of multimodal communication, where gaze plays a pivotal role in enhancing the coordination between human and robotic partners.

6. Feedback in Gaze-Based Communication

Communication is inherently a two-way process, and it is widely recognized that feedback from the system to the user plays a crucial role in ensuring effective interaction [124]. Visual feedback, for instance, can provide valuable information about the status and resolution of problems. However, it often lacks the ability to effectively capture the user’s attention in urgent situations. Conversely, in teleoperation settings, modalities such as vibration and sound are more effective at alerting users, though they may not convey detailed information as clearly as visual feedback [125]. Given these dynamics, it becomes evident that even in gaze-based communication, exploring different channels for feedback is essential. By incorporating multiple feedback mechanisms, systems can improve responsiveness and enhance the overall interaction experience, ensuring that users receive timely and accurate feedback across various contexts and tasks. As we look further into gaze-based interactions, it is important to explore how these feedback mechanisms can be optimized for a range of applications, especially in immersive and complex environments. In more immersive environments, such as VR or AR, feedback mechanisms must be carefully tuned to maintain the user’s sense of presence while enhancing interaction efficiency. For example, in virtual environments, visual feedback alone may become overwhelming or insufficient due to the complexity of the virtual scene. In such cases, combining feedback channels can significantly improve performance and user experience.

6.1. Haptic Feedback

Haptic feedback such as vibrations are often used to inform the user when an event or action happens. The timing of this feedback is important; if delayed, users may misinterpret the source or cause of the feedback. Kangas et al. [126] demonstrated this in a study where participants were required to locate a target using their gaze, with haptic feedback providing the moment when the correct target was fixated on. Their findings revealed that when the delay between the initial gaze and the feedback was too long, the number of errors increased significantly, indicating that prompt feedback is essential for accurate task performance. These findings are further supported by Rantala et al. [127] in their review of gaze interaction with vibrotactile feedback, which emphasizes the importance of delivering haptic cues in a timely manner. Delays in feedback disrupt the natural flow of interaction and can lead to confusion or mistakes, underscoring the necessity of swift, well-coordinated feedback mechanisms in gaze-based systems.

The need for timely and effective feedback becomes even more apparent in gaze-based typing. In these scenarios, visual feedback generally informs the user of the selected letter and helps monitor the dwell time, which is commonly used to confirm selections. However, once a letter is successfully typed, auditory or haptic feedback is often more effective in signaling completion, as it is not visually distracting. Among these, haptic feedback offers a more discreet and private form of communication, ensuring the user remains focused on the task without distracting external noise [128].

Concerning immersive environments, a study by Sakamak et al. illustrates the integrating gaze and haptic feedback for better interaction during robot navigation [129]. In their research, a robot was guided towards a target using the user’s gaze, with haptic feedback provided to indicate proximity to the target. This additional haptic cue allowed users to adjust their gaze more precisely, helping them reach the target more quickly and effectively, demonstrating the value of multisensory feedback in dynamic gaze-based interactions.

6.2. Audio Feedback

While haptic feedback effectively signals that an event has occurred, audio feedback offers a more versatile means of delivering complex information to the user. Audio cues can range from simple notifications, like clicks or error sounds, to longer tones that indicate specific actions, such as an incoming call. Additionally, audio feedback can convey highly detailed instructions through speech generation, offering real-time guidance or conveying contextual information that visual or haptic channels might not easily communicate. This versatility makes audio feedback a valuable tool for enhancing user experience, particularly in tasks that require immediate and clear responses.

Even in acoustically complex environments with multiple audio sources and potential distractors, participants have demonstrated the ability to accurately localize sounds, as highlighted by the study conducted by Moraes et al. [130]. It evaluated how listeners perceive spatialized audio within a VE. Their findings revealed not only the effectiveness of spatial audio in guiding user attention but also that eye gaze can be correlated with varying levels of cognitive load and effort during sound localization tasks. This suggests that gaze patterns may offer insights into the cognitive demands placed on users when navigating acoustically rich environments, further emphasizing the potential of gaze–audio integration.

In the context of grasping tasks within a VE, audio feedback emerges as a promising alternative to haptic feedback, which is traditionally used to signal the successful grasp of a virtual object. In such tasks, recognizing when an object has been successfully grasped can be challenging without tactile cues. Canales and Jörg [131] conducted a study to explore the effectiveness of audio and visual feedback as substitutes for tactile feedback in virtual grasping interactions. Their results indicated that participants consistently preferred audio feedback over visual feedback, even though this preference seemed to come at the cost of a slight decrease in grasping performance. This preference for audio feedback highlights its potential to create more immersive and intuitive interaction experiences in VEs, especially when traditional tactile feedback is not available. Using audio cues for tasks like object manipulation, virtual environments can provide users with essential feedback, fostering a more engaging and responsive interaction. Although haptic feedback remains the gold standard for certain tasks, audio feedback presents a viable and sometimes preferred alternative, particularly in situations where physical touch is impractical or impossible.

In a virtual environment, speech generation can guide users through complex spaces, helping them reach their destination even without prior knowledge of the environment’s layout. By providing verbal cues, such as directional instructions or contextual hints, users can orient themselves more easily within the virtual space. This auditory feedback can mimic human-to-human type interaction for teaching and guiding. Moreover, it can be made even more effective by incorporating real-time gaze data into the process. When gaze information is integrated with speech generation, the system can adapt its verbal instructions based on the user’s current attention. For instance, if the user’s gaze is focused on an irrelevant part of the scene, the system could redirect their attention with more precise cues, such as “Look to your left” or “Focus on the red door”. This dynamic interaction personalizes the experience, making navigation more efficient and intuitive by responding to the user’s natural focus points. Studies by Staudte et al. and Garoufi et al. [132,133] have shown that exploiting gaze data for speech generation can significantly enhance the clarity and effectiveness of instructions in such environments, improving both user engagement and task performance. This combination of gaze tracking and audio feedback enables a more responsive system that tailors its guidance to individual user behavior, offering a more immersive and adaptive experience in virtual environments.

6.3. Visual Feedback

As previously discussed, visual feedback is highly effective for conveying detailed and specific information within a scenario. In a VE, visual cues can provide general context, such as indicating which objects can be interacted with, or more specific details, like identifying the type of error that has occurred. However, while visual feedback excels at delivering precise information, it tends to be less effective at immediately capturing the user’s attention, especially in dynamic or immersive environments. For this reason, visual feedback is often combined with other modalities, such as auditory or haptic feedback, to ensure that users are both aware of and can respond to critical changes or events in the environment. For instance, Zhang et al. [134] conducted a study in which participants were tasked with completing a “peg-in-a-hole” task—a classic motor skill exercise that involves navigating a small object through a confined space. The study evaluated the effectiveness of both audio and visual feedback during the task. The results revealed that while both feedback types were beneficial, participants expressed a clear preference for the combination of audio and visual feedback. This multimodal approach allowed users to benefit from the precision of visual feedback in understanding the task requirements and the immediacy of auditory feedback to alert them of their actions.

While visual feedback proves to be effective, its usefulness is also task-dependent. In a study by Kangas et al. [135], the researchers compared haptic, audio, visual, and no feedback in a task where participants were required to adjust the gray level using smooth pursuit eye movements. Interestingly, despite the diverse feedback modalities, there were no statistically significant differences in performance across the different conditions. However, participant preferences told a different story: both haptic and audio feedback were rated more favorably compared to visual and no feedback. This suggests that while visual feedback can provide valuable information, other feedback types may offer greater immediacy or intuitiveness in certain tasks. Similarly, another study conducted by Lankes and Haslinger [136] explored the use of visual, audio, and haptic feedback in an exploratory game where participants had to locate objects using the different feedback modalities. Their findings revealed a clear preference for visual and audio feedback, with haptic feedback receiving notably lower ratings. The participants found the combination of visual and auditory feedback to be the most engaging and effective, leading the researchers to conclude that further exploration of this multimodal approach could offer significant benefits for feedback systems in similar tasks.

These studies underscore the complexity of feedback design, particularly in gaze-based or interactive systems. While visual feedback can provide detailed and context-rich information, its effectiveness varies depending on the task at hand. Combining it with auditory or haptic feedback often enhances user experience by supporting the strengths of each modality (Figure 6). Therefore, a deeper investigation into how these feedback mechanisms interact is essential to optimize user performance and satisfaction across various types of virtual environments and applications.

These feedback mechanisms are beneficial, but they often either inform the user of actions that have already occurred or attempt to guide them toward desired behavior. However, systems that can accurately predict a user’s next likely action have the potential to create a more seamless and efficient user experience—much like how text prediction accelerates messaging. In this context, gaze-based human–system interaction stands to gain significantly from intent prediction, enabling more proactive and intuitive support.

7. Discussion

As gaze-based communication technology continues to advance, it is increasingly becoming a viable solution for interacting with AI systems while minimizing computational effort. Figure 7 illustrates a two-way communication model that emphasizes the dynamic exchange between the user and the AI system. This loop begins with the user’s input, which is captured through modalities such as gaze or head movements and contextualized with information about the scene. The system processes this data to interpret the user’s intention and provides feedback via visual, auditory, or haptic channels. This bi-directional interaction mirrors natural human communication, where both parties mutually learn based on each other’s responses. In this discussion, we explore the components underlying this communication model, including gaze data acquisition, challenges in gaze-based interaction, intention prediction, and feedback strategies. Finally, we address how the integration of these elements can promote a seamless and intuitive interaction experience that builds trust and improves usability.

Starting with the question: How is gaze data used in interactive systems, especially in immersive environments? Gaze data is typically collected by eye-tracking devices, which record eye movements and determine the direction of the user’s gaze. The raw gaze coordinates collected over time are then processed to provide meaningful insights into gaze behavior. For example, algorithms analyze the duration and stability of the gaze at specific coordinates to determine whether a user is fixating on an object, following a moving target, or directing their gaze to a new point of interest. These inferred behaviors, including fixations, smooth pursuits, and saccades, provide fundamental data that can be used to calculate higher-level metrics. Such interpretations of gaze data are essential for applications in immersive environments, as they enable interactive systems to respond to the user’s focus and intent, creating a more natural and intuitive communication interface. A summary of that pipeline is shown in Figure 1. Despite progress, many issues remain, such as identifying key features for interpreting gaze in unaffected environments and developing robust real-time analysis methods; increasingly, AI is being used to overcome these challenges and enable learning with minimal supervision and adaptability to real-world conditions [137].

Moving forward, we address the question: How can gaze be used effectively as an interaction tool, and what challenges does this pose? By combining gaze data with scene data, eye-tracking systems can accurately identify the user’s object of interest, enabling direct communication of it to the system. This capability opens up a wide range of applications: For example, gaze can facilitate tasks such as text entry, navigation through menus or interfaces or—especially in immersive environments—the selection of objects in augmented or virtual reality. However, the use of gaze as an interaction tool brings with it unique challenges. One well-known problem is the so-called Midas Touch problem, where unintended decisions are made simply because the user is looking at an object. To solve this problem, various approaches have been developed to ensure that gaze-based interaction is precise and intentional (see Figure 3). These solutions typically involve using gaze to select the target while requiring confirmation through an additional input, such as a head movement or an external control (e.g., a hand gesture or voice command). This multimodal approach helps to avoid accidental actions and makes gaze-based interaction more reliable and intuitive for users [69]. By utilizing these strategies, gaze-based systems can create responsive and engaging experiences for various applications, from hands-free control of devices to seamless interaction in immersive environments.

The next step in gaze-based communication is intention prediction: How can gaze be used to predict user intentions? In general, systems interpret gaze and scene data to classify likely intentions, often using machine learning models. Since intentions can be viewed as hidden states that influence user behavior, methods such as Hidden Markov Models (HMMs) are commonly applied to model the sequential nature of gaze data and predict the underlying intentions over time. However, predicting intentions based on gaze data alone can be challenging as accuracy decreases with the number of possible intentions, but when the set of possible intentions is small, high accuracies can be achieved [120]. Combining gaze-based interaction methods with intention prediction simplifies this process: Identifying the specific object of interest reduces the range of possible user intentions. This combined approach has shown success in predefined environments, where the limited set of intentions enables accurate predictions. Expanding this accuracy to more dynamic, open-ended settings remains an ongoing research challenge.

Once the system has interpreted the user’s intent, it needs to provide feedback to ensure that the user understands the system’s response, which leads us to the question: What types of feedback channels from the system to the user are most effective? There are three primary feedback channels: visual, haptic, and auditory, each with unique strengths and limitations (see Figure 6). Visual feedback can quickly confirm the point of interest, haptic feedback provides a more tactile confirmation, and audio feedback provides additional context or instructions. In practice, the effectiveness of a feedback channel depends heavily on the task. For example, visual cues can emphasize the point of interest, while audio feedback can provide relevant information without disrupting the visual focus. Multimodal feedback, where two or more channels are combined, often increases clarity and improves user interaction by reinforcing the message across multiple senses [138]. However, it is important to balance the richness of feedback with user comfort, as excessive feedback can lead to sensory overload, especially in immersive environments.

This brings us to the final question: How can these gaze-based techniques be integrated to create a system that encourages intuitive, responsive communication? For users, the process of expressing intent should be straightforward and natural, with clear and easy-to-interpret feedback from the AI. By enabling users to effectively communicate their intentions and receive meaningful feedback, the system creates a dynamic feedback loop that allows users to feel understood and correct any misinterpretations. This interactive flow mirrors human-to-human communication and promotes trust between the user and the AI system [139,140]. This review shows that gaze-based input has proven successful in various applications, particularly in immersive environments. While much of the research to date has focused on healthy users, gaze-based communication systems show promise for people with disabilities as they offer a more accessible way of interacting. Future research should aim to make these systems more accessible to a wider range of users, adapting to individual needs and optimizing user comfort and engagement. By incorporating these findings, emerging AI systems can become more inclusive and user-friendly and serve a wider range of users. However, it remains a challenge to achieve reliable performance in various real-world environments. To overcome these challenges, future vision-based systems must prioritize safety and ease of use, especially for people with mobility impairments. The advancement of technologies such as Apple Vision Pro and HoloLens 2 has the potential to transform everyday interactions, especially for users with special needs. By focusing on accessibility and usability, these devices could make everyday interactions more efficient, inclusive, and impactful, supporting a wider range of users in a meaningful way.

Moreover, gaze-based communication systems for real-world applications must take ethical and privacy considerations into account: With attention to both the input and the feedback mechanisms. Gaze is an intuitive and natural method for communicating intentions, allowing users to convey their focus and goals without text or spoken language. This makes it inherently private compared to voice or text input, as the risk of eavesdropping or unauthorized access is minimized [141]. Feedback channels also pose a particular challenge for data protection. Audio feedback is clear and specific but becomes a public channel unless headphones are used, potentially exposing sensitive information. Haptic feedback, such as vibrations, is very private but lacks the precision required for detailed communication. Visual feedback, delivered through devices such as AR glasses, offers a middle ground that provides clarity while maintaining privacy, although it requires specialized devices that are not always practical. Additionally, it is important to recognize that advanced data analytics can extract extensive information from gaze signals, including biometric identity, demographic details, personality traits, emotional states, interests, cognitive processes, and even indicators of physical or mental health conditions [142]. Balancing these factors is critical to developing systems that are not only effective but also in line with ethical principles and user trust.

8. Conclusions

In this overview, the main methods and applications for gaze-based communication have been analyzed and their potential to improve interaction with AI systems highlighted. Bi-directional, gaze-based communication provides users with a natural and intuitive way to communicate their intentions to the technology, mimicking familiar interactions. By using gaze as the primary input, users can express their intentions with minimal effort and are less reliant on complex interaction methods such as voice commands or manual input. This simplicity in design is one of the biggest advantages of gaze-based communication, as it works with minimal computer requirements and is a scalable solution for different platforms. As AI systems become an integral part of everyday life, two-way communication with gaze enables more dynamic and responsive interactions that improve both the accuracy of intent recognition and the user experience by providing immediate, clear feedback. This feedback loop not only builds user confidence but also promotes satisfaction by making the system feel more intuitive and engaging. Gaze-based communication holds promise for a range of applications, from augmented reality to assistive robots, as its intuitive nature and efficiency make it an attractive solution for improving human-AI interaction. Future research and development work should continue to focus on usability and accessibility to ensure that these systems serve a variety of users effectively. In this way, gaze-based communication can shape a more inclusive and seamless future for AI-driven technology.