1. Introduction
Immersive and interactive technologies are increasingly being used to attract more visitors to museums and exhibitions. Renowned museums all over the world (such as the National Gallery in London or the Smithsonian Institution in Washington D.C.) frequently use digital installations to display cultural and scientific knowledge in a more engaging way.
The scientific literature has extensively examined the possibilities of augmented and virtual reality (AR/VR) technologies applied to Cultural Heritage [
1,
2,
3,
4], as well as the usage of Serious Gaming [
5,
6], which is very effective for teaching young visitors. Virtual museums (i.e., museums or collections accessible online) have also been widely studied [
7], especially showing their importance during the COVID-19 pandemic [
8,
9].
In this paper, we focus on a particular, and little studied, type of digital installation, namely
gaze-based interactive applications. Traditionally, interactive installations for museums and exhibitions have mainly used gestural communication, especially exploiting the capabilities of Microsoft Kinect [
10,
11,
12]. Even if this kind of application is now well-established, museums are always interested in new approaches and new ways to effectively present their collections. Gaze-based interaction is relatively new in this field, but has great potential: it can provide a natural and intuitive interaction in which visitors can retrieve information by simply looking at the items on display; it can be a hygienic alternative to touchscreens (a very important aspect after the COVID-19 pandemic) and it can also enhance the accessibility of a museum or exhibition, since eye tracking technology is commonly used by motor-impaired people to communicate.
After some pioneering studies that date back to the late 1980s [
13], the first attempts to use eye tracking technology in museums were focused on studying visitors’ behavior [
14,
15,
16], their cognitive processes while observing artwork [
17,
18], and their emotional reactions [
19]. More recently, various museums have used gaze-based solutions to both collect data and engage visitors—for instance, by showing them, after the visit, how they observed the items on display and/or highlighting possible similarities of their eye paths with those of other visitors. In this regard, notable examples include the Cleveland Museum of Art (
https://mw18.mwconf.org/glami/gaze-tracker/ (accessed on 20 April 2023)) (United States), the ARoSArt Museum (
https://userexperienceawards.com/2017-submissions/aros-art-museum-aros-public/ (accessed on 20 April 2023)) (Denmark) and the M-Museum Leuven (
https://www.mleuven.be/en/research-support/research/eye-tracking-research-how-do-we-look-art (accessed on 20 April 2023)) (Belgium). These can mostly be considered “passive” uses of eye tracking technology, since there is not an actual, explicit interaction—gaze input is used for some kind of “a posteriori” analysis. There are also cases, such as the Science Museum of Trento (
https://www.srlabs.it/en/project/muse/ (accessed on 20 April 2023)) (Italy), in which the interaction based on eye tracking per se (therefore not necessarily connected with the exhibited works) is simply included in scientific museums for informational purposes, to make the existence of this technology known to the general public.
A particular case involves robot-guides, a relatively new but active research field [
20,
21,
22]. Gaze is sometimes used in this context, but mainly in an indirect way. For example, a robot may detect a visitor looking at a painting and start describing it [
21]. In these applications, however, gaze direction is only roughly estimated by considering the visitor’s body and head position and orientation, and gaze input does not have an active role in the interaction, apart from triggering the activation of the robot.
For the present literature review, we analyzed the state-of-the-art of gaze-based interaction for museums and exhibitions, with the purpose of identifying current and future trends, as well as of highlighting the advantages and limitations of this technology. We considered only active gaze input, and only those works where real eye tracking devices were used (not simply estimates of the direction of the head).
The paper is structured as follows:
Section 2 highlights the purposes and goals of this review;
Section 3 provides an overview of eye tracking technology;
Section 4 presents the methodology and choices we adopted for paper selection;
Section 5 and
Section 6 illustrate and discuss the results obtained;
Section 7, lastly, draws some conclusions.
2. Objectives
This review has two main goals. First, we summarize a relatively new application field of eye tracking not yet analyzed in other scientific literature reviews. We think that our work can be useful for researchers dealing with interactive applications in the Cultural Heritage field, since it discusses a possible alternative to the most common interactive approaches typically used in museums, namely, gestural and touch interfaces. Secondly, museums and exhibitions are among the few public contexts where gaze-based interaction has been used “in the wild”. Most gaze-based interactive applications have been developed as assistive solutions, to allow people with severe motor disabilities to interact with the computer, or for mere research purposes, to conduct experiments in controlled laboratory settings. However, very few general and practical applications of gaze-based communication are reported in the literature. Museums are a very good test field for eye tracking technology, since they are both public places visited by many people and relatively quiet environments.
We think that a review of gaze-based applications for museums and exhibitions can also be useful for those who need to develop gaze-based interactive applications for use in public places in general, which is a recent trend in eye tracking technology.
3. Eye Tracking Technology
Eye tracking is a technique for detecting and measuring eye movements and characteristics [
23]. An
eye tracker can sense a person’s gaze locations at a certain frequency. Finding gaze position allows identification of
fixations and
saccades. Fixations, which typically last between 100 and 600 ms [
24,
25], are time periods during which the eyes are almost still, with the gaze being focused on a specific element of the scene. On the other hand, saccades, which normally last less than 100 ms [
25], are very fast eye movements occurring between consecutive pairs of fixations, with the purpose of relocating the gaze on a different element in the visual scene.
Electro-oculography, scleral contact lens/search coil, photo-oculography, video-oculography and pupil center-corneal reflection are some of the eye-tracking technologies that have been developed over time [
23].
Electro-oculography (EOG), one of the oldest methods to record eye movements, measures the skin’s electrical potential differences through small electrodes placed around the eyes [
26]. This solution allows recording of eye movements even when the eyes are closed, but is generally more invasive and less accurate and precise than other approaches.
Scleral contact lens/search coil is another old method consisting of small coils of wire inserted in special contact lenses. The user’s head is then placed inside a magnetic field to generate an electrical potential that allows estimation of eye position [
27]. While this technique has a very high spatial and temporal resolution, it is also extremely invasive and uncomfortable, used practically only for physiological studies.
Photo- and video-oculography (POG and VOG) are generally video-based methods in which small cameras, incorporated in head-mounted devices, measure eye features such as pupil size, iris–sclera boundaries and possible corneal reflections. The assessment of these characteristics can occur both automatically and manually. However, these systems tend to be inaccurate and are mainly used for medical purposes [
23].
Pupil center-corneal reflection (PCCR) is the most used eye tracking technique nowadays. Its basic principle consists of using infrared (or near-infrared) light sources to illuminate the eyes and detect reflections on their surface (
Figure 1); this allows determination of the gaze direction [
23]. Infrared light is employed because it is invisible and also produces a better contrast between pupil and iris. The prices of these eye trackers range from a few hundreds to tens of thousands of euros, depending on their accuracy and gaze sampling frequency. All the works analyzed in the present review employ this technology.
There are two main kinds of eye trackers, namely,
remote and
wearable. Remote eye trackers (
Figure 2, left) are normally non-intrusive devices (often little “bars”) that are positioned at the bottom of standard displays. They are currently the most prevalent kind of eye trackers. Wearable eye trackers (
Figure 2, right), on the other hand, are frequently used to study viewing behavior in real-world settings. Recent wearable eye trackers look more and more like glasses, making them much more comfortable than in the past.
Psychology [
28], neuroscience [
29], marketing [
30], education [
31], usability [
32,
33] and biometrics [
34,
35] are all fields in which eye tracking technology has been applied, for instance, to determine the user’s gaze path while looking at something (e.g., an image or a web page) or to obtain information about the screen regions that are most frequently inspected. When using an eye tracker as an input tool (i.e., for interactive purposes, in an explicit way), gaze data must be evaluated in real-time, so that the computer can respond to specific gaze behaviors [
36]. Gaze input is also extremely beneficial as an assistive technology for people who are unable to use their hands. Several assistive solutions have been devised to date, including those for writing [
37,
38,
39], surfing the Web [
40,
41] and playing music [
42,
43].
Two common ways to provide gaze input are through
dwell time and
gaze gestures. Dwell time, which is the most used approach, consists of fixating a target element (e.g., a button) for a certain time (the dwell time), after which an action connected to that element is triggered. The duration of the dwell time can vary depending on the application, but it should be chosen so as to avoid the so called “Midas touch problem” [
44], i.e., involuntary selections occurring when simply looking at the elements of an interface.
Gaze gestures consist of gaze paths performed by the user to trigger specific actions. This approach can be fast and is immune to the Midas touch problem, but it is also generally less intuitive than the dwell time (since the user needs to memorize a set of gaze gestures, it may have a steep learning curve). For this reason, gaze gestures are recommended only for applications meant to be used multiple times, such as writing systems (e.g., [
45,
46]).
Hybrid approaches that mix dwell time and gaze gestures (e.g., for interacting with video games [
47]) have also been proposed, while other gaze input methods (such as those based on blinks [
48] or smooth pursuit [
39]) are currently less common.
6. Discussion
Although the first gaze-based interactive applications for museums date back more than a decade, the use of eye tracking as a technology to attract and engage visitors can still be considered relatively new and uncommon today. The few applications tested in real museums have only been in use for short periods, mainly as prototypes exploited for scientific purposes. However, our analysis has highlighted important aspects that we believe will characterize the future of eye tracking in museums and exhibitions.
6.1. Summary on the Use of Wearable Eye Trackers/AR Applications
To improve the visiting experience in museums, wearable eye trackers are being integrated into AR applications. By tracking the user’s gaze, these systems can identify elements of interest and deliver appropriate auditory information or augmentations, potentially resulting in a more informative, personalized and interactive experience. Mobile eye trackers and audio guide systems are being combined to create proactive and reactive versions of traditional mobile audio guides.
Eye tracking technology is also used to spatialize appropriate sounds in paintings or exhibits based on the visitor’s gaze. When a visitor focuses on a specific object or area, the related sound can be amplified while other sounds are reduced. Such gaze-based audio augmentation can increase the visitor’s focus and enhance their overall experience with the artwork.
The evolution of augmented reality applications in museums now includes multimodal interactions that exploit gaze-based engagement together with gestures, voice, music and general sounds. Voice commands allow visitors to interact with the narrative, and hand and eye gestures can be used to identify items and play music or instructions. Such multimodal strategy can produce an attractive, immersive and dynamic AR experience.
6.2. Summary on the Use of Remote Eye Trackers/Non-Immersive VR Applications
Like wearable devices, remote eye trackers can grab visitors’ attention by providing them with proper output based on their gaze; even a simple circle shown onto an observed image can act as a visual indicator of visitors’ focus and involvement. Visitors can engage with exhibits using their gaze, picking and examining particular photos or items, panning and zooming over them and obtaining contextual information on specific characters or objects. Studies aimed at comparing gaze-based interaction to traditional mouse-based interaction have shown that eye input can be better from an “emotional” point of view, and may also be perceived as more natural.
Multimodal interaction also plays an important role, mainly as a combination of gaze and voice input. For example, visitors can visually explore exhibits, gaze-select specific areas of interest and use voice commands to ask for further information. Moreover, visitor interaction with artwork can be facilitated using gamification principles. For instance, visitors can use their gaze to move around paintings by panning, zooming and triggering “active areas”, which then provide text, images and/or videos about specific elements. Through enhanced interaction and learning opportunities, this strategy can encourage visitors to view the complete piece of art.
Due to the COVID-19 pandemic, several of the experiments presented in the papers analyzed in this review were carried out in laboratories or in very specific exhibitions. Nevertheless, the results of user studies involving task completion and participant feedback show that gaze-based applications are generally appreciated by users.
Overall, the integration of non-immersive VR applications and remote eye trackers into museum settings open new opportunities for engaging visitors, personalizing interactions and improving their comprehension and enjoyment of exhibits. For a more engaging museum experience, gaze-based interaction and visual feedback can draw interest, encourage exploration and provide pertinent information.
6.3. General Considerations
All eye trackers, whether wearable or remote, have both advantages and disadvantages;
Table 5 summarizes the most distinctive features of the two types of devices, especially when used in museums.
Regarding wearable eye trackers, one problem is certainly their generally high cost compared to remote devices, but this issue will probably be overcome in the next years, due to the constant improvements in technology. However, the “hygienic” problem should be considered for wearable tools—especially after the COVID-19 pandemic. Sharing them among different visitors, without appropriate sanitation procedures, may represent both an actual risk and a psychological barrier. Of course, similar considerations apply to any wearable device, including those for augmented and virtual reality. On the contrary, remote eye trackers are generally cheaper and more hygienic, but lack the freedom of movement of wearable solutions.
A well-known limitation of eye tracking technology (common to both wearable and remote devices) is the need for an initial calibration procedure when a good precision is required, which may make the interaction less natural and, in some cases, may force the visitor to remain relatively still. Non-calibrated gaze-based applications have appeared in recent years (e.g., for writing [
39,
51] or for basic interaction in public places [
66,
67]), but they are still significantly slower than their calibrated counterpart.
Despite these limitations, eye tracking technology certainly has its advantages as well. For example, remote devices are a more hygienic solution than the much more popular touchscreens, and can also help improve accessibility by enabling people with mobility impairments to interact with the exhibit using their gaze only. In general, eye tracking can be used to provide hands-free input by simply focusing on places of interest, thus implementing a more natural way of interacting with exhibits in a museum (for example, showing visitors information about what they are looking at in a painting).
The review of the scientific literature has also highlighted that some research paths are not yet fully explored. In particular, gaze-based interactive applications for virtual museums are surely a less-studied topic. Similarly, we have not found any immersive VR applications for museums that actively use the gaze to interact with virtual objects. The gaze is mainly exploited to simply “align” the simulation, or, more rarely, for implicit interaction (e.g., for the creation of virtual artwork using the visitors’ gaze paths [
68]).
6.4. Future Perspectives
The above analysis has shown the potential of eye tracking technology for museums and exhibitions, but it has also highlighted some limitations of current approaches. Here, we suggest possible future research paths, also taking into account the current trends in related research fields.
Table 6 provides a brief summary of our analysis.
The first problem to address is surely the need for initial calibration. While calibration is a standard procedure in a laboratory setting, it may be problematic in a museum (and, more generally, in public places). Since calibration depends on the user’s position, a museum visitor may move too much (for example, because they are distracted by other people) and, thus, need to repeat the procedure. Calibration problems may occur not only with remote eye trackers, but also with wearable devices. For example, a visitor may adjust the position of eye tracking glasses (to make them more comfortable), losing the initial calibration during the process. Repeated calibrations may be perceived as a boring nuisance, especially if compared to other interactive devices, such as touch screens. The need for calibration is also the main reason why remote devices may require one to stay relatively still during their use. Removing this constraint would ensure greater freedom of movement for visitors when dealing with applications based on remote eye trackers. For all these reasons, researchers should focus on developing applications that do not need calibration (for example, because the gaze-based interaction occurs through big target graphical elements, which allow pointing errors to be tolerated). A sort of “hidden” calibration can also be considered. For example, the initial screen of an application may contain a large central element to be looked at to start the interaction. This element can be used to perform a so-called
one-point calibration, without the user being aware of it. While less precise than standard calibration procedures, this quick calibration method provides a rough initial reference that can then be used to adjust the user’s estimated gaze positions. Finally, Artificial Intelligence (AI) can also help. In fact, new algorithms for gaze estimation that exploit deep learning have been recently proposed [
69]. With the rapid evolution of Artificial Intelligence, it is probable that robust solutions that do not require calibration (or allow it to be achieved through very fast procedures) will soon be developed, thus guaranteeing a more immediate and natural interaction.
Eye tracking technology paired with AI could also offer real-time interpretation of exhibits or artwork. As visitors look at an exhibit, deep learning models could deliver rapid translations, audio descriptions or contextual information. This would enable a more engaging and instructive experience for a wider spectrum of visitors. Moreover, data on visitors’ gaze patterns and interests could be processed in real time to create custom experiences based on the preferences of each visitor. This would allow museums to provide specialized tours, recommend different itineraries or offer supplemental material tailored to a specific visitor’s attention patterns. Based on individual preferences, AI algorithms could evaluate gaze patterns and provide real-time guidance or suggestions on the most efficient routes or exhibits to explore. This “intelligent wayfinding” could help visitors save time and avoid missing important features.
Immersive VR can be another interesting path to explore for future research, especially for virtual museums. The COVID pandemic has shown the need to accelerate the digitization of artwork to make them available to people all around the world. Immersive VR is surely the best option for this scenario. Explicit gaze interaction in VR is a well-studied topic (e.g., [
70]), whose solutions could easily be extended to the exploration of virtual museums. As for gaze-based AR applications, there are already various examples in which gaze is used alone or in combination with voice and gestures (see
Table 2 and
Table 4). Future improvements will involve the consolidation of this technology, with more robust implementations of existing solutions, better responsiveness and a more realistic “merge” between virtual and real content. The availability of powerful HMDs with eye-tracking capabilities (such as Microsoft Hololens or the recently announced Apple Vision Pro) can surely ensure, in the near future, the possibility of more convenient gaze-based interaction for both AR and VR applications.
Finally, it should be noted that, while there are many gaze-based solutions designed to help people with mobility impairments, no trials have been proposed in museums until now. This is probably due to the fact that all the applications we have presented in our review have only been installed for a relatively short time, for experimental purposes or for specific events. However, museums are generally very interested in improving their accessibility and we think that eye tracking technology can be effectively used for this purpose. Research in this direction should include extensive testing with people with motor disabilities, to better understand how to properly design and contextualize gaze-based assistive applications in museums or temporary exhibitions.
7. Conclusions
In this literature review, we have analyzed the state-of-the-art of gaze-based interactive applications for museums and exhibitions. Eye tracking technology can allow these cultural settings to improve visitor engagement, personalize experiences, optimize exhibit layouts and create more inclusive environments. Museums may attract and fascinate visitors by skillfully utilizing modern technology, offering them a memorable and interactive experience.
The results of our analysis have shown that, in the last decade, eye tracking technology has been applied in museums in various ways for designing AR and non-immersive VR applications, using both wearable and remote devices. However, there is still room for improvement, with several interesting research paths, including non-calibrated algorithms, guided and personalized content, interactive VR for virtual museums and assistive installations for motor impaired visitors.
Of course, it is important to stress that, as with any technology, ethical considerations and visitor privacy should be prioritized when using eye tracking devices in museums and exhibitions. To preserve confidence and provide a positive user experience for all visitors, clear communication, consent and data anonymization should be established.
In conclusion, even if gaze-based interactive solutions for museums and exhibitions are still few, we think this is a promising research field, especially with the emergence of new AI algorithms that may further improve the precision and robustness of eye tracking devices and help in the creation of personalized visitor experiences.