Search Results (29)

Human–computer interaction (HCI) plays a crucial role across various fields, with eye-tracking technology emerging as a key enabler for intuitive and dynamic control in assistive systems like Assistive Robotic Arms (ARAs). By precisely tracking eye movements, this technology allows for more natural user interaction. However, current systems primarily rely on the single gaze-dependent interaction method, which leads to the “Midas Touch” problem. This highlights the need for real-time eye movement classification in dynamic interactions to ensure accurate and efficient control. This paper proposes a novel Gaussian Mixture Model–Hidden Markov Model (GMM-HMM) classification algorithm aimed at overcoming the limitations of traditional methods in dynamic human–robot interactions. By incorporating sum of squared error (SSE)-based feature extraction and hierarchical training, the proposed algorithm achieves a classification accuracy of 94.39%, significantly outperforming existing approaches. Furthermore, it is integrated with a robotic arm system, enabling gaze trajectory-based dynamic path planning, which reduces the average path planning time to 2.97 milliseconds. The experimental results demonstrate the effectiveness of this approach, offering an efficient and intuitive solution for human–robot interaction in dynamic environments. This work provides a robust framework for future assistive robotic systems, improving interaction intuitiveness and efficiency in complex real-world scenarios. Full article

This paper presents ITap, a novel interaction method utilizing hand tracking to create a virtual touchpad on a tabletop. ITap facilitates touch interactions such as tapping, dragging, and swiping using the index finger. The technique combines gaze-based object selection with touch gestures, while a pinch gesture performed with the opposite hand activates a manual mode, enabling precise cursor control independently of gaze direction. The primary purpose of this research is to enhance interaction efficiency, reduce user fatigue, and improve accuracy in gaze-based object selection tasks, particularly in complex and cluttered XR environments. Specifically, we addressed two research questions: (1) How does ITap’s manual mode compare with the traditional gaze + pinch method regarding speed and accuracy in object selection tasks across varying distances and densities? (2) Does ITap provide improved user comfort, naturalness, and reduced fatigue compared to the traditional method during prolonged scrolling and swiping tasks? To evaluate these questions, two studies were conducted. The first study compared ITap’s manual mode with the traditional gaze + pinch method for object selection tasks across various distances and in cluttered environments. The second study examined both methods for scrolling and swiping tasks, focusing on user comfort, naturalness, and fatigue. The findings revealed that ITap outperformed gaze + pinch in terms of object selection speed and error reduction, particularly in scenarios involving distant or densely arranged objects. Additionally, ITap demonstrated superior performance in scrolling and swiping tasks, with participants reporting greater comfort and reduced fatigue. The integration of gaze-based input and touch gestures provided by ITap offers a more efficient and user-friendly interaction method compared to the traditional gaze + pinch technique. Its ability to reduce fatigue and improve accuracy makes it especially suitable for tasks involving complex environments or extended usage in XR settings. Full article

Multimodal interaction is a transformative human-computer interaction (HCI) approach that allows users to interact with systems through various communication channels such as speech, gesture, touch, and gaze. With advancements in sensor technology and machine learning (ML), multimodal systems are becoming increasingly important in various applications, including virtual assistants, intelligent environments, healthcare, and accessibility technologies. This survey concisely overviews recent advancements in multimodal interaction, interfaces, and communication. It delves into integrating different input and output modalities, focusing on critical technologies and essential considerations in multimodal fusion, including temporal synchronization and decision-level integration. Furthermore, the survey explores the challenges of developing context-aware, adaptive systems that provide seamless and intuitive user experiences. Lastly, by examining current methodologies and trends, this study underscores the potential of multimodal systems and sheds light on future research directions. Full article

Nonverbal connection is an important aspect of everyday communication. For romantic partners, nonverbal connection is essential for establishing and maintaining feelings of closeness. EEG hyperscanning offers a unique opportunity to examine the link between nonverbal connection and neural synchrony among romantic partners. This current study used an EEG hyperscanning paradigm to collect frontal alpha asymmetry (FAA) signatures from 30 participants (15 romantic dyads) engaged in five different types of nonverbal connection that varied based on physical touch and visual contact. The results suggest that there was a lack of FAA while romantic partners were embracing and positive FAA (i.e., indicating approach) while they were holding hands, looking at each other, or doing both. Additionally, partners’ FAA synchrony was greatest at a four second lag while they were holding hands and looking at each other. Finally, there was a significant association between partners’ weekly negative feelings and FAA such that as they felt more negative their FAA became more positive. Taken together, this study further supports the idea that fleeting moments of interpersonal touch and gaze are important for the biological mechanisms that may underlie affiliative pair bonding in romantic relationships. Full article

Spatiotemporal models for the 3-D shape and motion of objects allowed large progress in the 1980s in visual perception of moving objects observed from a moving platform. Despite the successes demonstrated with several vehicles, the “4-D approach” has not been accepted generally. Its advantage is that only the last image of the sequence needs to be analyzed in detail to allow the full state vectors of moving objects, including their velocity components, to be reconstructed by the feedback of prediction errors. The vehicle carrying the cameras can, thus, together with conventional measurements, directly create a visualization of the situation encountered. In 1994, at the final demonstration of the project PROMETHEUS, two sedan vehicles using this approach were the only ones worldwide capable of driving autonomously in standard heavy traffic on three-lane Autoroutes near Paris at speeds up to 130 km/h (convoy driving, lane changes, passing). Up to ten vehicles nearby could be perceived. In this paper, the three-layer architecture of the perception system is reviewed. At the end of the 1990s, the system evolved from mere recognition of objects in motion, to understanding complex dynamic scenes by developing behavioral capabilities, like fast saccadic changes in the gaze direction for flexible concentration on objects of interest. By analyzing motion of objects over time, the situation for decision making was assessed. In the third-generation system “EMS-vision” behavioral capabilities of agents were represented on an abstract level for characterizing their potential behaviors. These maneuvers form an additional knowledge base. The system has proven capable of driving in networks of minor roads, including off-road sections, with avoidance of negative obstacles (ditches). Results are shown for road vehicle guidance. Potential transitions to a robot mind and to the now-favored CNN are touched on. Full article

It is still largely unknown to what extent domestication, ancestry, or recent functional selection are responsible for the behavioral differences in whether dogs look back to a human when presented with a difficult task. Here, we tested whether this ubiquitous human-related response of companion dogs would appear differently in subjects that were selected for either cooperative or independent work tasks. We tested N = 71 dogs from 18 cooperative and 18 independent breeds. Subjects learned in a five-trial warming-up phase that they could easily obtain the reward from a container. In trial six, the reward became impossible to take out from the locked container. When the task was easy, both breed groups behaved similarly, and their readiness to approach the container did not differ between the last ‘solvable’ and the subsequent ‘unsolvable’ trial. Task focus, looking at the container, touching the container for the first time, or interacting with the container with a paw or nose did not differ between the breed groups, indicating that their persistence in problem solving was similar. However, in the ‘unsolvable’ trial, cooperative dogs alternated their gaze more often between the container and the humans than the independent dogs did. The frequency of looking back was also higher in cooperative dogs than in the independent breeds. These are the first empirical results that suggest, in a balanced, representative sample of breeds, that the selection for different levels of cooperativity in working dogs could also affect their human-dependent behavior in a generic problem-solving situation. Full article

Some of the barriers preventing virtual reality (VR) from being widely adopted are the cost and unfamiliarity of VR systems. Here, we propose that in many cases, the specialized controllers shipped with most VR head-mounted displays can be replaced by a regular smartphone, cutting the cost of the system, and allowing users to interact in VR using a device they are already familiar with. To achieve this, we developed SmartVR Pointer, an approach that uses smartphones to replace the specialized controllers for two essential operations in VR: selection and navigation by teleporting. In SmartVR Pointer, a camera mounted on the head-mounted display (HMD) is tilted downwards so that it points to where the user will naturally be holding their phone in front of them. SmartVR Pointer supports three selection modalities: tracker based, gaze based, and combined/hybrid. In the tracker-based SmartVR Pointer selection, we use image-based tracking to track a QR code displayed on the phone screen and then map the phone’s position to a pointer shown within the field of view of the camera in the virtual environment. In the gaze-based selection modality, the user controls the pointer using their gaze and taps on the phone for selection. The combined technique is a hybrid between gaze-based interaction in VR and tracker-based Augmented Reality. It allows the user to control a VR pointer that looks and behaves like a mouse pointer by moving their smartphone to select objects within the virtual environment, and to interact with the selected objects using the smartphone’s touch screen. The touchscreen is used for selection and dragging. The SmartVR Pointer is simple and requires no calibration and no complex hardware assembly or disassembly. We demonstrate successful interactive applications of SmartVR Pointer in a VR environment with a demo where the user navigates in the virtual environment using teleportation points on the floor and then solves a Tetris-style key-and-lock challenge. Full article

Infant attention is a cognitive function that underlines sensory–motor integration processes at the interface between the baby and the surrounding physical and socio-relational environment, mainly with the caregivers. The investigation of the role of non-visual inputs (i.e., vocal and tactile) provided by the caregivers in shaping infants’ attention in the context of visual impairment is relevant from both a theoretical and clinical point of view. This study investigated the social attention (i.e., gaze orientation) skills in a group of visually impaired (VI) and age-matched sighted controls (SCs) between 9 and 12 months of age. Moreover, the role of VI severity and maternal vocalizations and touch in shaping the social attention were investigated. Overall, 45 infants and their mothers participated in a video-recorded 4 min interaction procedure, including a play and a still-face episode. The infants’ gaze orientation (i.e., mother-directed, object-directed, or unfocused) and the types of maternal vocalizations and touch (i.e., socio-cognitive, affective) were micro-analytically coded. Maternal vocalizations and touch were found to influence gaze orientation differently in VI infants compared SCs. Moreover, the group comparisons during the play episode showed that controls were predominantly oriented to the mothers, while VI infants were less socially oriented. Visual impairment severity did not emerge as linked with social attention. These findings contribute to our understanding of socio-cognitive developmental trajectories in VI infants and highlight the need for tailored interventions to promote optimal outcomes for VI populations. Full article

In this paper, we present an approach to improve pointing methods and target selection on tactile human–machine interfaces. This approach defines a two-level highlighting technique (TLH) based on the direction of gaze for target selection on a touch screen. The technique uses the orientation of the user’s head to approximate the direction of his gaze and uses this information to preselect the potential targets. An experimental system with a multimodal interface has been prototyped to assess the impact of TLH on target selection on a touch screen and compare its performance with that of traditional methods (mouse and touch). We conducted an experiment to assess the effectiveness of our proposition in terms of the rate of selection errors made and time for completion of the task. We also made a subjective estimation of ease of use, suitability for selection, confidence brought by the TLH, and contribution of TLH to improving the selection of targets. Statistical results show that the proposed TLH significantly reduces the selection error rate and the time to complete tasks. Full article

Eye gaze can be a potentially fast and ergonomic method for target selection in augmented reality (AR). However, the eye-tracking accuracy of current consumer-level AR systems is limited. While state-of-the-art AR target selection techniques based on eye gaze and touch (gaze-touch), which follow the “eye gaze pre-selects, touch refines and confirms” mechanism, can significantly enhance selection accuracy, their selection speeds are usually compromised. To balance accuracy and speed in gaze-touch grid menu selection in AR, we propose the Hand-Held Sub-Menu (HHSM) technique.tou HHSM divides a grid menu into several sub-menus and maps the sub-menu pointed to by eye gaze onto the touchscreen of a hand-held device. To select a target item, the user first selects the sub-menu containing it via eye gaze and then confirms the selection on the touchscreen via a single touch action. We derived the HHSM technique’s design space and investigated it through a series of empirical studies. Through an empirical study involving 24 participants recruited from a local university, we found that HHSM can effectively balance accuracy and speed in gaze-touch grid menu selection in AR. The error rate was approximately 2%, and the completion time per selection was around 0.93 s when participants used two thumbs to interact with the touchscreen, and approximately 1.1 s when they used only one finger. Full article

The emotion of disgust is thought to play a critical role in maintaining restrictive eating among individuals with anorexia nervosa. This exploratory cross-sectional study examined correlations between food-specific trait and state disgust, eating disorder psychopathology, illness severity (body mass index: BMI), and interactions with virtual foods in people with anorexia nervosa. Food-specific trait disgust and eating disorder symptoms were measured before exposure to virtual foods in one of three virtual reality (VR) kitchens to which participants were randomly allocated. Food interactions (eye gaze and reaching towards virtual foods) were measured during the VR exposure. Food-specific state disgust ratings were collected after the VR exposure. In the entire sample, eating disorder symptoms correlated positively with food-specific trait disgust (rs (68) = 0.45, p < 0.001). We also found a significant association between food-specific state disgust and eating disorder symptoms in each virtual kitchen scenario: virtual kitchen only (rs (22) = 0.40, p = 0.05), virtual kitchen plus pet (rs (22) = 0.80, p < 0.001), and virtual kitchen plus avatar (rs (20) = 0.78, p < 0.001). No significant correlation was observed for the link between food-specific disgust measures and food-related touch. Correlations between food-specific trait disgust and food-related eye gaze differed across scenarios. The current experimental paradigm needs to be improved to draw firm conclusions. Both food-specific trait and state disgust are associated with eating disorder psychopathology, and therefore, effective strategies are warranted to attenuate food-specific disgust. Full article

For greater efficiency, human–machine and human–robot interactions must be designed with the idea of multimodality in mind. To allow the use of several interaction modalities, such as the use of voice, touch, gaze tracking, on several different devices (computer, smartphone, tablets, etc.) and to integrate possible connected objects, it is necessary to have an effective and secure means of communication between the different parts of the system. This is even more important with the use of a collaborative robot (cobot) sharing the same space and very close to the human during their tasks. This study present research work in the field of multimodal interaction for a cobot using the MQTT protocol, in virtual (Webots) and real worlds (ESP microcontrollers, Arduino, IOT2040). We show how MQTT can be used efficiently, with a common publish/subscribe mechanism for several entities of the system, in order to interact with connected objects (like LEDs and conveyor belts), robotic arms (like the Ned Niryo), or mobile robots. We compare the use of MQTT with that of the Firebase Realtime Database used in several of our previous research works. We show how a “pick–wait–choose–and place” task can be carried out jointly by a cobot and a human, and what this implies in terms of communication and ergonomic rules, via health or industrial concerns (people with disabilities, and teleoperation). Full article

The principle of Fitts’ law explains that the difficulty of movement increases when targets are farther away and narrower in width, particularly when touching two parallel targets as quickly as possible. Understanding the differences in motor and gaze behaviors between extroverts and introverts when performing tasks that require speed and accuracy is crucial for the development of sensor-based interfaces for games and rehabilitation. This study aimed to investigate such differences in a computer task that assesses the speed–accuracy trade-off (Fitts’ task). Twenty introverts and seventeen extroverts wore an eye tracker and an accelerometer attached to their hand while performing 12 trials through six levels of difficulty presented on a computer screen. The results showed that introverts had longer visual fixations at the higher difficulty levels and reduced pupil diameter variability when difficulty was intermediate, suggesting that their gaze behavior may be different from that of extroverts. However, no significant differences were found in the speed and accuracy performance or kinematic variables between extroverts and introverts. These findings have important implications for the design of interventions that require both speed and accuracy in movement, such as in the development of virtual reality/games for rehabilitation purposes. It is important to consider individual differences in motor and gaze behaviors, particularly in those who may struggle with longer visual fixations, for the design of sensor-based applications and to promote successful interventions and recovery. Full article

When an electric wheelchair is operated using gaze motion, eye movements such as checking the environment and observing objects are also incorrectly recognized as input operations. This phenomenon is called the “Midas touch problem”, and classifying visual intentions is extremely important. In this paper, we develop a deep learning model that estimates the user’s visual intention in real time and an electric wheelchair control system that combines intention estimation and the gaze dwell time method. The proposed model consists of a 1DCNN-LSTM that estimates visual intention from feature vectors of 10 variables, such as eye movement, head movement, and distance to the fixation point. The evaluation experiments classifying four types of visual intentions show that the proposed model has the highest accuracy compared to other models. In addition, the results of the driving experiments of the electric wheelchair implementing the proposed model show that the user’s efforts to operate the wheelchair are reduced and that the operability of the wheelchair is improved compared to the traditional method. From these results, we concluded that visual intentions could be more accurately estimated by learning time series patterns from eye and head movement data. Full article

This paper investigates the usability of touch screens used in mass production road vehicles. Our goal is to provide a detailed comparison of conventional physical buttons and capacitive touch screens taking the human factor into account. The pilot test focuses on a specific Non-driving Related Task (NDRT): the control of the on-board climate system using a touch screen panel versus rotating knobs and push buttons. Psychological parameters, functionality, usability and, the ergonomics of In-Vehicle Information Systems (IVIS) were evaluated using a specific questionnaire, a system usability scale (SUS), workload assessment (NASA-TLX), and a physiological sensor system. The measurements are based on a wearable eye-tracker that provides fixation points of the driver’s gaze in order to detect distraction. The closed road used for the naturalistic driving study was provided by the ZalaZONE Test Track, Zalaegerszeg, Hungary. Objective and subjective results of the pilot study indicate that the control of touch screen panels causes higher visual, manual, and cognitive distraction than the use of physical buttons. The statistical analysis demonstrated that conventional techniques need to be complemented in order to better represent human behavior differences. Full article