Multimodal Technologies and Interaction

36 pages, 18792 KiB

Open AccessArticle

VICTORIOUS: A Visual Analytics System for Scoping Review of Document Sets

by Amir Haghighati, Amir Reza Haghverdi and Kamran Sedig

Multimodal Technol. Interact. 2025, 9(5), 37; https://doi.org/10.3390/mti9050037 - 22 Apr 2025

Scoping review is an iterative knowledge synthesis methodology concerned with broad questions about the nature of a research subject. The increasingly large number of published documents in scholarly domains poses challenges in conducting scoping reviews. Despite attempts to address these challenges, the specific [...] Read more.

Scoping review is an iterative knowledge synthesis methodology concerned with broad questions about the nature of a research subject. The increasingly large number of published documents in scholarly domains poses challenges in conducting scoping reviews. Despite attempts to address these challenges, the specific step of sensemaking in the context of scoping reviews is seldom addressed. We address sensemaking of a curated document collection by developing a VIsual analytiCs sysTem for scOping RevIew of dOcUment Sets (VICTORIOUS). Using known methods within the machine learning community, we propose and develop six modules within VICTORIOUS: Map, Summary, Skim, SemJump, BiblioNetwork, and Compare. To demonstrate the utility of VICTORIOUS, we describe three usage scenarios. We conclude by a qualitative comparison of VICTORIOUS and other available systems. While existing systems leave their users with singular information items regarding a document set and gaining an aggregated assessment in a scoping review is often a challenge, VICTORIOUS shows promise for making sense of documents in a scoping review process. Full article

► Show Figures

Figure 1

21 pages, 3847 KiB

Open AccessArticle

Evaluating Uses of XR in Fostering Art Students’ Learning

by Panagiotis Dafiotis, Stella Sylaiou, Efstratios Stylianidis, Dimitrios Koukopoulos and Christos Fidas

Multimodal Technol. Interact. 2025, 9(4), 36; https://doi.org/10.3390/mti9040036 - 14 Apr 2025

Abstract

This paper addresses the potential of extended reality (XR) to foster art students’ learning and creativity with specially developed applications for the creation of art exhibitions. This study is based on the EU-funded research project ‘Scaffolding Creativity of Arts Students: Framework, Toolchain, and [...] Read more.

This paper addresses the potential of extended reality (XR) to foster art students’ learning and creativity with specially developed applications for the creation of art exhibitions. This study is based on the EU-funded research project ‘Scaffolding Creativity of Arts Students: Framework, Toolchain, and Educational Material on how to Create their Own Virtual Exhibitions’ (CREAMS). CREAMS develops interconnected virtual reality (VR), augmented reality (AR), and mixed reality (MR) indoor and outdoor exhibition creation applications. The article puts emphasis on the issue of what evaluation methodology is more adequate to assess the efficacy of such applications, and pertinent challenges, as well as related research, are discussed. In this context, ways in which Fine Art School undergraduates can benefit from the creation of XR/VR exhibitions are explored. Such uses of XR are primarily examined in terms of their potential to enhance learning, and foster students’ skills. The potential of adding multimodal resources that contextualize exhibited artworks is examined to foster viewers’ meaningful engagement. Art students’ ability to communicate the underlying concepts/ideas of their art through XR apps is also addressed. Moreover, this paper investigates how XR technologies can foster the collaboration of tutors and students in Fine Art Schools through specially developed platforms embedded in the XR applications. The ability of the CREAMS VR application to foster such cooperation between students and their tutors is also evaluated. The scientific contribution of this paper relates to the evaluation methodology of XR art exhibition applications that have an educational role. Full article

(This article belongs to the Special Issue Innovative Theories and Practices for Designing and Evaluating Inclusive Educational Technology and Online Learning)

► Show Figures

Figure 1

18 pages, 1050 KiB

Open AccessArticle

The Use of 3D Printing and ICT in the Designing of Didactic Strategies to Foster Creative Thinking

by Lorena Cabrera-Frías, Diana-Margarita Córdova-Esparza, Julio-Alejandro Romero-González, Teresa García-Ramírez, Rocio-Edith López-Martínez, Juan Terven and Juan-José Rodríguez-Peña

Multimodal Technol. Interact. 2025, 9(4), 35; https://doi.org/10.3390/mti9040035 - 7 Apr 2025

Abstract

The use of iteration is fundamental in the field of design, as it allows for the exploration of multiple options and the development of innovative solutions. This research analyzes how iteration, combined with Information and Communications Technology (ICT), especially 3D printing, facilitates the [...] Read more.

The use of iteration is fundamental in the field of design, as it allows for the exploration of multiple options and the development of innovative solutions. This research analyzes how iteration, combined with Information and Communications Technology (ICT), especially 3D printing, facilitates the development of creativity in higher-level students, specifically Industrial Designers. A qualitative approach was employed, using an action-research methodological strategy with a field journal to document student activities during the iterative process. The results indicate that group dynamics enhance iteration by facilitating the exchange of ideas; therefore, the incorporation of activities that adopt constant feedback is necessary to refine initial concepts and integrate new perspectives. Additionally, the use of technology, such as 3D printing, drives the creative process by encouraging students to continue developing their ideas through physical iteration, allowing them to materialize and improve their concepts. The creative process only needs to be stimulated through different methodological strategies, both analog and digital, in and outside the classroom, with the use of technologies as essential learning tools. Full article

(This article belongs to the Special Issue Innovative Theories and Practices for Designing and Evaluating Inclusive Educational Technology and Online Learning)

► Show Figures

Figure 1

20 pages, 1862 KiB

Open AccessArticle

Evaluation of a Virtual Human in Delivering Relaxation Exercises for Wound Healing and Stress Reduction: A Randomised Controlled Trial

by Isabella Pickering, Mikaela Law, Kate Loveys, Mark Sagar, Nadine Skoluda, Urs M. Nater and Elizabeth Broadbent

Multimodal Technol. Interact. 2025, 9(4), 34; https://doi.org/10.3390/mti9040034 - 7 Apr 2025

Abstract

Objective: Relaxation delivered via audiotapes can reduce stress and improve wound healing. Virtual humans are a promising technology to deliver relaxation, but robust research is needed into their effectiveness. This randomised controlled trial investigated whether relaxation delivered by a virtual human could improve [...] Read more.

Objective: Relaxation delivered via audiotapes can reduce stress and improve wound healing. Virtual humans are a promising technology to deliver relaxation, but robust research is needed into their effectiveness. This randomised controlled trial investigated whether relaxation delivered by a virtual human could improve healing and reduce stress after an experimental wound. Methods: A total of 159 healthy adults underwent a tape-stripping wounding procedure and were randomly assigned to relaxation delivered by a virtual human, human audiotape, or a control condition. Skin barrier recovery (SBR) was measured by assessing changes in transepidermal water loss at baseline, post-tape-stripping, and post-intervention. Psychological and physiological variables were measured over the session. Participants’ perceptions of the interventions were assessed. Results: There were no significant differences in SBR between conditions. All conditions experienced significant improvements in the psychological variables, heart rate, and cortisol over time. After controlling for the baseline values, the virtual human and audiotape conditions were significantly more relaxed post-intervention than the control condition (p = 0.005), the audiotape condition had lower post-intervention anxiety than the control condition (p = 0.016), and alpha-amylase was significantly reduced in the virtual human group compared with the audiotape (p = 0.041). The audiotape received the highest satisfaction and engagement ratings, with qualitative results suggesting the appearance and lip-syncing of the virtual human could be improved. Conclusions: Relaxation instructions delivered by a virtual human increased participants’ relaxation levels with similar effects to traditional audiotapes. Furthermore, it reduced physiological stress indices. Further work with other wound types and stressed samples is needed. The voice and interactiveness of the virtual human should be improved to promote greater engagement and satisfaction. Full article

► Show Figures

Figure 1

21 pages, 2000 KiB

Open AccessArticle

Bloom: Scaffolding Multiple Positive Emotion Regulation Techniques to Enhance Casual Conversations and Promote the Subjective Well-Being of Emerging Adults

by Kyra Kozin, Sehar Mapara, Chajoong Kim and JungKyoon Yoon

Multimodal Technol. Interact. 2025, 9(4), 33; https://doi.org/10.3390/mti9040033 - 7 Apr 2025

Abstract

This paper presents Bloom, a low-tech interactive artifact designed to enhance in-person social interactions among emerging adults. Drawing upon emotion regulation theories, the development of Bloom explores how multiple Positive Emotion Regulation (PER) techniques can be incorporated into an artifact that can effectively [...] Read more.

This paper presents Bloom, a low-tech interactive artifact designed to enhance in-person social interactions among emerging adults. Drawing upon emotion regulation theories, the development of Bloom explores how multiple Positive Emotion Regulation (PER) techniques can be incorporated into an artifact that can effectively enhance users’ subjective well-being in their routines. By grounding on autobiographical research through design, we discuss the design methodology employed to develop a prototype that supports five PER techniques, along with key design features, and experiences of using Bloom. The chosen PER techniques are as follows: (1) being immersed and absorbed, (2) engaging in a collective, (3) creating a savoring atmosphere, (4) sharing the positive experience with others, and (5) infusing ordinary events with positive meaning. Through daily observations and reflections, we discuss key design features that have proven critical in engaging ourselves in PER, the benefits of utilizing diverse PER techniques to enhance our well-being, and implications for future behavioral intervention technologies. Full article

► Show Figures

Figure 1

15 pages, 734 KiB

Open AccessSystematic Review

Utilizing VR Visual Novels Incorporating Social Stories for Learning in Children with Autism Spectrum Disorder: A Systematic Literature Review

by Katerina Atsalaki and Ioannis Kazanidis

Multimodal Technol. Interact. 2025, 9(4), 32; https://doi.org/10.3390/mti9040032 - 4 Apr 2025

Abstract

Autism Spectrum Disorder (ASD) is a neurodevelopmental condition that impacts social, communication, and emotional skills, presenting significant challenges in learning and social interaction. Traditional teaching approaches often fail to engage children with ASD, highlighting the need for innovative solutions. This study investigates the [...] Read more.

Autism Spectrum Disorder (ASD) is a neurodevelopmental condition that impacts social, communication, and emotional skills, presenting significant challenges in learning and social interaction. Traditional teaching approaches often fail to engage children with ASD, highlighting the need for innovative solutions. This study investigates the potential of virtual reality (VR) visual novels, incorporating social stories, as a tool to enhance social skills in children with ASD Level 1. Through a comprehensive literature review, the research evaluates VR environments that blend the interactive, choice-based structure of visual novels with immersive social narratives. Key aspects such as empathy, communication, and emotional regulation are analyzed to assess whether VR-based social stories provide better learning outcomes compared to conventional 2D methods. The findings aim to inform about the application of VR technologies in educational interventions, demonstrating how immersive learning experiences can promote essential social competencies in children with ASD. Full article

► Show Figures

Figure 1

17 pages, 4523 KiB

Open AccessArticle

Predicting Activity in Brain Areas Associated with Emotion Processing Using Multimodal Behavioral Signals

by Lahoucine Kdouri, Youssef Hmamouche, Amal El Fallah Seghrouchni and Thierry Chaminade

Multimodal Technol. Interact. 2025, 9(4), 31; https://doi.org/10.3390/mti9040031 - 31 Mar 2025

Abstract

Artificial agents are expected to increasingly interact with humans and to demonstrate multimodal adaptive emotional responses. Such social integration requires both perception and production mechanisms, thus enabling a more realistic approach to emotional alignment than existing systems. Indeed, existing emotion recognition methods rely [...] Read more.

Artificial agents are expected to increasingly interact with humans and to demonstrate multimodal adaptive emotional responses. Such social integration requires both perception and production mechanisms, thus enabling a more realistic approach to emotional alignment than existing systems. Indeed, existing emotion recognition methods rely on behavioral signals, predominantly facial expressions, as well as non-invasive brain recordings, such as Electroencephalograms (EEGs) and functional Magnetic Resonance Imaging (fMRI), to identify humans’ emotions, but accurate labeling remains a challenge. This paper introduces a novel approach examining how behavioral and physiological signals can be used to predict activity in emotion-related regions of the brain. To this end, we propose a multimodal deep learning network that processes two categories of signals recorded alongside brain activity during conversations: two behavioral signals (video and audio) and one physiological signal (blood pulse). Our network enables (1) the prediction of brain activity from these multimodal inputs, and (2) the assessment of our model’s performance depending on the nature of interlocutor (human or robot) and the brain region of interest. Results demonstrate that the proposed architecture outperforms existing models in anterior insula and hypothalamus regions, for interactions with a human or a robot. An ablation study evaluating subsets of input modalities indicates that local brain activity prediction was reduced when one or two modalities are omitted. However, they also revealed that the physiological data (blood pulse) achieve similar levels of predictions alone compared to the full model, further underscoring the importance of somatic markers in the central nervous system’s processing of social emotions. Full article

► Show Figures

Figure 1

40 pages, 817 KiB

Open AccessArticle

The Impact of Dating Apps on the Mental Health of the LGBTIQA+ Population

by Laura Perić, Michel Rabar and Tihomir Orehovački

Multimodal Technol. Interact. 2025, 9(4), 30; https://doi.org/10.3390/mti9040030 - 27 Mar 2025

Abstract

This study explores the psychological impact of dating app usage on the mental health of LGBTIQA+ individuals in Southeast Europe, focusing on key factors such as anxiety, cyberbullying, perceived security, and self-confidence. Data were collected through an online survey administered to a representative [...] Read more.

This study explores the psychological impact of dating app usage on the mental health of LGBTIQA+ individuals in Southeast Europe, focusing on key factors such as anxiety, cyberbullying, perceived security, and self-confidence. Data were collected through an online survey administered to a representative sample of active dating app users within the LGBTIQA+ community, enabling a comprehensive analysis of how these factors influence users’ emotional states, attitudes, and behavioral intentions related to dating app use. The findings reveal that, while dating apps provide important opportunities for social connection and support, they also pose significant risks. Cyberbullying was identified as a prominent issue, strongly associated with heightened anxiety, reduced self-confidence, and increased fear. Anxiety also showed a negative association with perceived security. Among the examined factors, fear and pleasure emerged as significant predictors of LGBTIQA+ users’ attitudes toward dating apps, which in turn influenced their intention to continue using them. The results underscore the need for enhanced security features and inclusive design practices that prioritize the psychological well-being of LGBTIQA+ users. By addressing these challenges, dating app developers and policymakers can create environments that support healthier interactions and promote more positive experiences for marginalized users. Full article

► Show Figures

Figure 1

44 pages, 14851 KiB

Open AccessArticle

Physics-Based Tool Usage Simulations in VR

by Nikolaos Partarakis, Xenophon Zabulis, Dimitris Zourarakis, Ioanna Demeridou, Ines Moreno, Arnaud Dubois, Nikolaos Nikolaou, Peiman Fallahian, David Arnaud, Noël Crescenzo, Patricia Hee and Andriani Stamou

Multimodal Technol. Interact. 2025, 9(4), 29; https://doi.org/10.3390/mti9040029 - 24 Mar 2025

Abstract

The need for scalable, immersive training systems is universal and recently has been included in fields that rely on complex, hands-on processes, such as surgery operations, assembly operations, construction processes training, etc. This paper examines the potential to support immersive training via digital [...] Read more.

The need for scalable, immersive training systems is universal and recently has been included in fields that rely on complex, hands-on processes, such as surgery operations, assembly operations, construction processes training, etc. This paper examines the potential to support immersive training via digital tool manipulation in the domain of traditional handicrafts. The proposed methodology employs Finite Element Method simulations to compute material transformations and apply them to interactive virtual environments. The challenge is to accurately simulate human–tool interactions, which are critical to the acquisition of manual skills. Using Simulia Abaqus (v.2023HF2), crafting simulations are authored, executed, and exported as animation sequences. These are further refined in Blender (v3.6) and integrated into Unity to create reusable training components called Action Animators. Two software applications—Craft Studio (v1.0) and Apprentice Studio (v1.0)—are designed and implemented to enable instructors to create training lessons and students to practice and get evaluated in virtual environments. The methodology has wide-ranging applications beyond crafts, offering a solution for immersive training in skill-based activities. The validation and evaluation of the proposed approach suggest that it can significantly improve training effectiveness, scalability, and accessibility across various industries. Full article

(This article belongs to the Special Issue Innovative Theories and Practices for Designing and Evaluating Inclusive Educational Technology and Online Learning)

► Show Figures

Figure 1

28 pages, 7066 KiB

Open AccessSystematic Review

A Systematic Review on Artificial Intelligence-Based Multimodal Dialogue Systems Capable of Emotion Recognition

by Luis Bravo, Ciro Rodriguez, Pedro Hidalgo and Cesar Angulo

Multimodal Technol. Interact. 2025, 9(3), 28; https://doi.org/10.3390/mti9030028 - 14 Mar 2025

Abstract

In the current context, the use of technologies in applications for multimodal dialogue systems with computers and emotion recognition through artificial intelligence continues to grow rapidly. Consequently, it is challenging for researchers to identify gaps, propose new models, and increase user satisfaction. The [...] Read more.

In the current context, the use of technologies in applications for multimodal dialogue systems with computers and emotion recognition through artificial intelligence continues to grow rapidly. Consequently, it is challenging for researchers to identify gaps, propose new models, and increase user satisfaction. The objective of this study is to explore and analyze potential applications based on artificial intelligence for multimodal dialogue systems incorporating emotion recognition. The methodology used in selecting papers is in accordance with PRISMA and identifies 13 scientific articles whose research proposals are generally focused on convolutional neural networks (CNNs), Long Short-Term Memory (LSTM), GRU, and BERT. The research results identify the proposed models as Mindlink-Eumpy, RHPRnet, Emo Fu-Sense, 3FACRNNN, H-MMER, TMID, DKMD, and MatCR. The datasets used are DEAP, MAHNOB-HCI, SEED-IV, SEDD-V, AMIGOS, and DREAMER. In addition, the metrics achieved by the models are presented. It is concluded that emotion recognition models such as Emo Fu-Sense, 3FACRNNN, and H-MMER obtain outstanding results, with their accuracy ranging from 92.62% to 98.19%, and multimodal dialogue models such as TMID and the scene-aware model with BLEU4 metrics obtain values of 51.59% and 29%, respectively. Full article

(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)

► Show Figures

Figure 1

24 pages, 4499 KiB

Open AccessArticle

Human Attitudes in Robotic Path Programming: A Pilot Study of User Experience in Manual and XR-Controlled Robotic Arm Manipulation

by Oscar Escallada, Nagore Osa, Ganix Lasa, Maitane Mazmela, Fatih Doğangün, Yigit Yildirim, Serdar Bahar and Emre Ugur

Multimodal Technol. Interact. 2025, 9(3), 27; https://doi.org/10.3390/mti9030027 - 10 Mar 2025

Abstract

Extended reality (XR) and collaborative robots are reshaping human–robot interaction (HRI) by introducing novel control methods that enhance user experience (UX). However, human factors such as cognitive workload, usability, trust, and task performance are often underexplored. This study evaluated UX during robotic manipulation [...] Read more.

Extended reality (XR) and collaborative robots are reshaping human–robot interaction (HRI) by introducing novel control methods that enhance user experience (UX). However, human factors such as cognitive workload, usability, trust, and task performance are often underexplored. This study evaluated UX during robotic manipulation tasks under three interaction modalities: manual control, XR-based control at real-time speed (RS), and XR-based control at reduced safety speed (SS). Twenty-one participants performed a series of tasks across three scenarios, where we measured usability, workload, flow state, trust, and agency using a subjective questionnaire adapted from SUS, NASA-TLX, FSS, SoAS, and Trust in Industrial Human–Robot Collaboration Questionnaire, and objective task metrics (completion time, errors, and attempts). Our results reveal that RS-based control modes significantly reduced physical workload and improved usability compared to manual control. RS control at real-time speed enhanced task efficiency but increased error rates during complex tasks, while SS mode mitigated errors at the cost of prolonged completion times. Trust and agency remained stable across all modalities, indicating extended reality technologies do not undermine user confidence. These findings contribute to the field of human–robot collaboration by offering insights regarding efficiency, accuracy, and UX. The results are particularly relevant for industries seeking to optimize safety, productivity, and human-centric robotic systems. Full article

► Show Figures

Graphical abstract

25 pages, 2317 KiB

Open AccessArticle

diaLogic: A Multi-Modal Framework for Automated Team Behavior Modeling Based on Speech Acquisition

by Ryan Duke and Alex Doboli

Multimodal Technol. Interact. 2025, 9(3), 26; https://doi.org/10.3390/mti9030026 - 10 Mar 2025

Abstract

This paper presents diaLogic, a humans-in-the-loop system for modeling the behavior of teams during collective problem solving. Team behavior is modeled using multi-modal data about cognition, social interactions, and emotions acquired from speech inputs. The system includes methods for speaker diarization, speaker interaction [...] Read more.

This paper presents diaLogic, a humans-in-the-loop system for modeling the behavior of teams during collective problem solving. Team behavior is modeled using multi-modal data about cognition, social interactions, and emotions acquired from speech inputs. The system includes methods for speaker diarization, speaker interaction characterization, speaker emotion recognition, and speech-to-text conversion. Hypotheses about the invariant and differentiated aspects of teams are extracted using the similarities and dissimilarities of their behavior over time. Hypothesis extraction, a novel contribution of this work, uses a method to identify the clauses and concepts in each spoken sentence. Experiments present system performance for a broad set of cases of team behavior during problem solving. The average errors of the various methods are between 6% and 21%. The system can be used in a broad range of applications, from education to team research and therapy. Full article

(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)

► Show Figures

Figure 1

25 pages, 2851 KiB

Open AccessArticle

Self-Created Film as a Resource in a Multimodal Conversational Narrative

by Mascha Legel, Stijn R. J. M. Deckers, Gloria Soto, Nicola Grove, Annalu Waller, Hans van Balkom, Ronald Spanjers, Christopher S. Norrie and Bert Steenbergen

Multimodal Technol. Interact. 2025, 9(3), 25; https://doi.org/10.3390/mti9030025 - 10 Mar 2025

Abstract

When access to natural speech is limited or challenging, as is the case for people with complex communication needs, self-created digital film can be practical to use as one of the resources within a multimodal conversation about a personal experience. The detailed and [...] Read more.

When access to natural speech is limited or challenging, as is the case for people with complex communication needs, self-created digital film can be practical to use as one of the resources within a multimodal conversation about a personal experience. The detailed and contextual information such audiovisual media offers with today’s available technology may assist the utility of other communication modes, such as (computerized) spoken, written or signed language, to foster mutual understanding and story growth. To promote the use of self-created film, here named a personal-video-scene (PVS), in the practice of augmentative and alternative communication (AAC), a greater understanding is required of how such media can operate as a resource within social interactions, such as daily conversations. This study therefore introduces a multimodal coding model developed to study the employment of a PVS within a film-elicited conversational narrative, relating to four aspects of conversational control: (a) topic development, (b) conversational structure, (c) conversational repair and (d) conversational maintenance. A case study illustrates how the use of a PVS in story-sharing was instrumental in establishing common ground between narrators, boosting the frequency of comments and questions, mitigating instances of conversational repair and expanding topic development. Full article

► Show Figures

Graphical abstract

20 pages, 1416 KiB

Open AccessArticle

Effects of Flight Experience or Simulator Exposure on Simulator Sickness in Virtual Reality Flight Simulation

by Alexander Somerville, Keith Joiner and Graham Wild

Multimodal Technol. Interact. 2025, 9(3), 24; https://doi.org/10.3390/mti9030024 - 6 Mar 2025

Abstract

The use of virtual reality (VR) for flight simulation, particularly in the earliest stages of pilot training, is gaining attention in both research and industry. The use of the technology for this ab initio training requires suitable consideration of the risks of simulator [...] Read more.

The use of virtual reality (VR) for flight simulation, particularly in the earliest stages of pilot training, is gaining attention in both research and industry. The use of the technology for this ab initio training requires suitable consideration of the risks of simulator sickness—risks that are heightened relative to conventional simulators. If simulator sickness results in the development of compensatory skills, or otherwise disrupts the training process, the benefits of the technology may be negated. Enabling the effective integration of VR within flight training requires that, to the extent that simulator sickness is an issue, practical mechanisms are developed to manage the occurrence without disrupting existing training structures. The primary objective of this research is, thus, to evaluate an intervention and a nuisance factor in relation to the reduction of simulator sickness, considering their practicality within existing flight training syllabi. The Total Severity (TS) of the Simulator Sickness Questionnaire (SSQ) was evaluated within a quasi-experimental, non-equivalent pre-test–post-test design, incorporating three groups: a prior flight experience nuisance factor group, a prior personal computer aviation training device (PCATD) exposure intervention group, and a control group with neither prior experience nor prior simulator exposure. The results indicated that the TS was significantly reduced for the prior flight experience nuisance factor (r_rb = 0.375), but that the PCATD exposure intervention produced no such reduction (r_rb = 0.016). The findings suggest that VR flight simulation is likely best used as a supplemental tool, introduced after initial airborne experience. Notwithstanding this finding, the relatively low median TS scores (<20) for all groups suggest that the technology may still be used with caution earlier in the training process. No other published research has examined this important effect in the context of the new VR situation. Full article

► Show Figures

Figure 1

15 pages, 910 KiB

Open AccessBrief Report

Real-Time Norwegian Sign Language Recognition Using MediaPipe and LSTM

by Md. Zia Uddin, Costas Boletsis and Pål Rudshavn

Multimodal Technol. Interact. 2025, 9(3), 23; https://doi.org/10.3390/mti9030023 - 3 Mar 2025

Abstract

The application of machine learning models for sign language recognition (SLR) is a well-researched topic. However, many existing SLR systems focus on widely used sign languages, e.g., American Sign Language, leaving other underrepresented sign languages such as Norwegian Sign Language (NSL) relatively underexplored. [...] Read more.

The application of machine learning models for sign language recognition (SLR) is a well-researched topic. However, many existing SLR systems focus on widely used sign languages, e.g., American Sign Language, leaving other underrepresented sign languages such as Norwegian Sign Language (NSL) relatively underexplored. This work presents a preliminary system for recognizing NSL gestures, focusing on numbers 0 to 10. Mediapipe is used for feature extraction and Long Short-Term Memory (LSTM) networks for temporal modeling. This system achieves a testing accuracy of 95%, aligning with existing benchmarks and demonstrating its robustness to variations in signing styles, orientations, and speeds. While challenges such as data imbalance and misclassification of similar gestures (e.g., Signs 3 and 8) were observed, the results underscore the potential of our proposed approach. Future iterations of the system will prioritize expanding the dataset by including additional gestures and environmental variations as well as integrating additional modalities. Full article

► Show Figures

Figure 1

23 pages, 2067 KiB

Open AccessArticle

Choice Vectors: Streamlining Personal AI Alignment Through Binary Selection

by Eleanor Watson, Minh Nguyen, Sarah Pan and Shujun Zhang

Multimodal Technol. Interact. 2025, 9(3), 22; https://doi.org/10.3390/mti9030022 - 3 Mar 2025

Abstract

Value alignment for AI is not “one-size-fits-all”: even polite and friendly models can still fail to represent individual user contexts and preferences, and local cultural norms. This paper presents a modular workflow for personal fine-tuning, synthesizing four core components from our previous research: [...] Read more.

Value alignment for AI is not “one-size-fits-all”: even polite and friendly models can still fail to represent individual user contexts and preferences, and local cultural norms. This paper presents a modular workflow for personal fine-tuning, synthesizing four core components from our previous research: (1) robust vectorization of user values and preferences, (2) a binary choice user interface (UI) approach to capturing those preferences with minimal cognitive load, (3) contrastive activation methods for steering large language models (LLMs) via difference vectors, and (4) knowledge graph integration for more auditable and structured alignment. Our approach—descended from past research on “Towards an End-to-End Personal Fine-Tuning Framework”—demonstrates how these elements can be combined to create personalized, context-rich alignment solutions. We report on user studies for the forced-choice UI, describe an experimental pipeline for deriving “control vectors”, and propose a “moral graph” method for bridging symbolic and vector-based alignment. Our findings suggest that multi-pronged personalization can significantly reduce user annotation fatigue, improve alignment fidelity, and allow for more flexible, interpretable AI behaviors. Full article

(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)

► Show Figures

Figure 1

17 pages, 3021 KiB

Open AccessArticle

Perceptions of Parents and Children About Videogame Use

by Michela Franzò, Gaia Maria Olivieri, Anna Salerni and Marco Iosa

Multimodal Technol. Interact. 2025, 9(3), 21; https://doi.org/10.3390/mti9030021 - 28 Feb 2025

Abstract

This study aims to investigate the gap in perceptions of parents and children on the use of videogames in childhood. Methods: A survey was conducted with 75 pairs formed by a son or daughter and one parent. The data collected contradict the prejudice [...] Read more.

This study aims to investigate the gap in perceptions of parents and children on the use of videogames in childhood. Methods: A survey was conducted with 75 pairs formed by a son or daughter and one parent. The data collected contradict the prejudice that playing video games reduces study time and leads to lower grades at school (R < 0.13). Our results support the idea that playing together fosters bonding and facilitates conversation. The impact of videogames on mood showed the most substantial differences in perception, with parents mainly reporting negative mood changes, while children reported similar frequencies of negative, neutral, and positive ones. In relation to the educational and informative potential of videogames, children had slightly more positive opinions than their parents (p < 0.001). Finally, more than half of the participants potentially agreed with the possibility of using videogames as academic tools. In conclusion, there is a gap between parents’ and children’s perceptions about videogaming, especially concerning their effects on children’s mood. Playing together and developing deeper knowledge about videogames could enhance positive effects on children’s development as well as their relationships with peers, parents, and at school. Full article

► Show Figures

Graphical abstract

19 pages, 1582 KiB

Open AccessArticle

Designing Digital Escape Rooms with Generative AI in University Contexts: A Qualitative Study

by Paula Rodríguez-Rivera, José M. Rodríguez-Ferrer and Ana Manzano-León

Multimodal Technol. Interact. 2025, 9(3), 20; https://doi.org/10.3390/mti9030020 - 27 Feb 2025

Cited by 1

Abstract

The rapid evolution of technology in education highlights the need for methodologies that enhance student engagement and skill development. This study examines students’ perceptions of designing educational escape rooms using ICT tools and generative AI (GenAI) as a learning methodology. A total of [...] Read more.

The rapid evolution of technology in education highlights the need for methodologies that enhance student engagement and skill development. This study examines students’ perceptions of designing educational escape rooms using ICT tools and generative AI (GenAI) as a learning methodology. A total of 47 students participated in creating digital escape rooms with GenAI, Genially, and HeroForge in the course “Mediation in Conflicts and Situations of Violence” within a Social Education degree. A qualitative approach was used, analyzing focus group discussions conducted after the activity. Results indicate that students valued the experience, emphasizing its impact on digital competence, creativity, and problem-solving skills. Collaborative learning helped overcome initial technical challenges, and students recognized the practical applicability of escape room design in mediation contexts. However, they identified areas for improvement, such as the need for more initial training, extended development time, and better access to digital tools. This study contributes to game-based learning and AI-enhanced education research, positioning students as active designers rather than passive users. Future research should explore the long-term impact on knowledge retention and transferable skills in professional settings. Full article

(This article belongs to the Special Issue Innovative Theories and Practices for Designing and Evaluating Inclusive Educational Technology and Online Learning)

► Show Figures

Graphical abstract

14 pages, 981 KiB

Open AccessArticle

Sensory Perception During Partial Pseudo-Haptics Applied to Adjacent Fingers

by Satoshi Saga and Kotaro Sakae

Multimodal Technol. Interact. 2025, 9(3), 19; https://doi.org/10.3390/mti9030019 - 26 Feb 2025

Abstract

Pseudo-haptics, the phenomenon of creating a simulated tactile sensation by introducing a discrepancy between a voluntary movement and its visual feedback, is well known. Typically, when inducing pseudo-haptics, the same control-display ratio (C/D ratio) is applied to all effectors. However, with the aim [...] Read more.

Pseudo-haptics, the phenomenon of creating a simulated tactile sensation by introducing a discrepancy between a voluntary movement and its visual feedback, is well known. Typically, when inducing pseudo-haptics, the same control-display ratio (C/D ratio) is applied to all effectors. However, with the aim of expanding the potential illusions that can be presented with pseudo-haptics, we investigated how perceived sensations change when partial pseudo-haptics are applied to adjacent body parts. In this research, we examined how perceived sensations change when pseudo-haptic stimuli are applied to adjacent body parts. Specifically, we investigated the correlation between finger states and the magnitude of illusory perception during both quasi-static and dynamic movements and identified the finger that experienced discomfort during dynamic movements with pseudo-haptics. Our findings revealed that: First, the magnitude of the illusion varied based on the contact state of adjacent fingers. Second, the illusion was more pronounced during dynamic movements compared to quasi-static movements. Third, regardless of the finger receiving the pseudo-haptic stimulus, the discomfort was primarily experienced in the finger exhibiting an overall inhibitory movement. The findings contribute to the practical application of pseudo-haptics as a virtual haptic display technology. Full article

(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)

► Show Figures

Figure 1

Journal Description

Multimodal Technologies and Interaction

Latest Articles

Journal Menu

Journal Browser

Highly Accessed Articles

Latest Books

E-Mail Alert

News

Topics

Conferences

Special Issues

Further Information

Guidelines

MDPI Initiatives

Follow MDPI