Multimodal Technologies and Interaction doi: 10.3390/mti8030023
Authors: Colin Dodds Ahmed Kharrufa
In this paper, we explore an approach to feedback which could allow those learning creative digital media practices in remote and asynchronous environments to receive rich, multi-modal, and interactive feedback upon their creative artefacts. We propose the show-and-tell feedback interface which couples graphical user interface changes (the show) to a text-based explanations (the tell). We describe the rationale behind the design and offer a tentative set of design criteria. We report the implementation and deployment into a real-world educational setting using a prototype interface developed to allow either traditional text-only feedback or our proposed show-and tell feedback across four sessions. The prototype was used to provide formative feedback upon music students’ coursework resulting in a total of 103 pieces of feedback. Thematic analysis was used to analyse the data obtained through interviews and focus groups with both educators and students (i.e., feedback givers and receivers). Recipients considered show-and-tell feedback to possess greater clarity and detail in comparison with the single modality text-only feedback they are used to receiving. We also report interesting emergent issues around control and artistic vision, and we discuss how these issues could be mitigated in future iterations of the interface.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8030022
Authors: Amal Abdulrahman Katherine Hopman Deborah Richards
Virtual agents (VAs) have been used effectively for psychoeducation. However, getting the VA’s design right is critical to ensure the user experience does not become a barrier to receiving and responding to the intended message. The study reported in this paper seeks to help first-year psychology students to develop knowledge and confidence to recommend emotion regulation strategies. In previous work, we received negative feedback concerning the VA’s lip-syncing, including creepiness and visual overload, in the case of stroke patients. We seek to test the impact of the removal of lip-syncing on the perception of the VA and its ability to achieve its intended outcomes, also considering the influence of the visual features of the avatar. We conducted a 2 (lip-sync/no lip-sync) × 2 (human-like/cartoon-like) experimental design and measured participants’ perception of the VA in terms of eeriness, user experience, knowledge gain and participants’ confidence to practice their knowledge. While participants showed a tendency to prefer the cartoon look over the human look and the absence of lip-syncing over its presence, all groups reported no significant increase in knowledge but significant increases in confidence in their knowledge and ability to recommend the learnt strategies to others, concluding that realism and lip-syncing did not influence the intended outcomes. Thus, in future designs, we will allow the user to switch off the lip-sync function if they prefer. Further, our findings suggest that lip-syncing should not be a standard animation included with VAs, as is currently the case.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8030021
Authors: Achraf Othman Khansa Chemnad Aboul Ella Hassanien Ahmed Tlili Christina Yan Zhang Dena Al-Thani Fahriye Altınay Hajer Chalghoumi Hend S. Al-Khalifa Maisa Obeid Mohamed Jemni Tawfik Al-Hadhrami Zehra Altınay
The following article investigates the Metaverse and its potential to bolster digital accessibility for persons with disabilities. Through qualitative analysis, we examine responses from eleven experts in digital accessibility, Metaverse development, disability advocacy, and policy formulation. This exploration uncovers key insights into the Metaverse’s current state, its inherent principles, and the challenges and opportunities it presents in terms of accessibility. The findings reveal a mixed state of inclusivity within the Metaverse, highlighting significant advancements along with notable gaps, especially in integrating assistive technologies and ensuring interoperability across different virtual environments. This study emphasizes the Metaverse’s potential to revolutionize experiences for individuals with disabilities, provided that accessibility is embedded in its foundational design. Ethical and legal considerations, such as privacy, non-discrimination, and evolving legal frameworks, are identified as critical factors that shape an inclusive Metaverse. We propose a comprehensive framework that emphasizes technological adaptation and innovation, user-centric design, universal access, social and economic considerations, and global standards. This framework aims to guide future research and policy interventions to foster an inclusive digital environment in the Metaverse. This paper contributes to the emerging discourse on the Metaverse and digital accessibility, offering a nuanced understanding of its complexities and a roadmap for future exploration and development. This underscores the necessity of a multi-faceted approach that incorporates technological innovation, user-centered design, ethical considerations, legal compliance, and continuous research to create an inclusive and accessible Metaverse.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8030020
Authors: Setareh Zafari Jesse de Pagter Guglielmo Papagni Alischa Rosenstein Michael Filzmoser Sabine T. Koeszegi
This article reports on a longitudinal experiment in which the influence of an assistive system’s malfunctioning and transparency on trust was examined over a period of seven days. To this end, we simulated the system’s personalized recommendation features to support participants with the task of learning new texts and taking quizzes. Using a 2 × 2 mixed design, the system’s malfunctioning (correct vs. faulty) and transparency (with vs. without explanation) were manipulated as between-subjects variables, whereas exposure time was used as a repeated-measure variable. A combined qualitative and quantitative methodological approach was used to analyze the data from 171 participants. Our results show that participants perceived the system making a faulty recommendation as a trust violation. Additionally, a trend emerged from both the quantitative and qualitative analyses regarding how the availability of explanations (even when not accessed) increased the perception of a trustworthy system.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8030019
Authors: Logan Pinter Mohammad Faridul Haque Siddiqui
In the realm of collegiate education, calculus can be quite challenging for students. Many students struggle to visualize abstract concepts, as mathematics often moves into strict arithmetic rather than geometric understanding. Our study presents an innovative solution to this problem: an immersive, interactive VR graphing tool capable of standard 2D graphs, solids of revolution, and a series of visualizations deemed potentially useful to struggling students. This tool was developed within the Unity 3D engine, and while interaction and expression parsing rely on existing libraries, core functionalities were developed independently. As a pilot study, it includes qualitative information from a survey of students currently or previously enrolled in Calculus II/III courses, revealing its potential effectiveness. This survey primarily aims to determine the tool’s viability in future endeavors. The positive response suggests the tool’s immediate usefulness and its promising future in educational settings, prompting further exploration and consideration for adaptation into an Augmented Reality (AR) environment.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8030018
Authors: Carina Liebers Pranav Megarajan Jonas Auda Tim C. Stratmann Max Pfingsthorn Uwe Gruenefeld Stefan Schneegass
Robot training often takes place in simulated environments, particularly with reinforcement learning. Therefore, multiple training environments are generated using domain randomization to ensure transferability to real-world applications and compensate for unknown real-world states. We propose improving domain randomization by involving human application experts in various stages of the training process. Experts can provide valuable judgments on simulation realism, identify missing properties, and verify robot execution. Our human-in-the-loop workflow describes how they can enhance the process in five stages: validating and improving real-world scans, correcting virtual representations, specifying application-specific object properties, verifying and influencing simulation environment generation, and verifying robot training. We outline examples and highlight research opportunities. Furthermore, we present a case study in which we implemented different prototypes, demonstrating the potential of human experts in the given stages. Our early insights indicate that human input can benefit robot training at different stages.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8030017
Authors: Mathieu Raynal Julie Ducasse Marc J.-M. Macé Bernard Oriola Christophe Jouffrais
Over the last decade, several projects have demonstrated how interactive tactile graphics and tangible interfaces can improve and enrich access to information for people with vision impairments. While the former can be used to display a relatively large amount of information, they cannot be physically updated, which constrains the type of tasks that they can support. On the other hand, tangible interfaces are particularly suited for the (re)construction and manipulation of graphics, but the use of physical objects also restricts the type and amount of information that they can convey. We propose to bridge the gap between these two approaches by investigating the potential of tactile and tangible graphics for people with vision impairments. Working closely with special education teachers, we designed and developed the FlexiBoard, an affordable and portable system that enhances traditional tactile graphics with tangible interaction. In this paper, we report on the successive design steps that enabled us to identify and consider technical and design requirements. We thereafter explore two domains of application for the FlexiBoard: education and board games. Firstly, we report on one brainstorming session that we organized with four teachers in order to explore the application space of tangible and tactile graphics for educational activities. Secondly, we describe how the FlexiBoard enabled the successful adaptation of one visual board game into a multimodal accessible game that supports collaboration between sighted, low-vision and blind players.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8030016
Authors: Jakob Peintner Bengt Escher Henrik Detjen Carina Manger Andreas Riener
Currently, a significant gap exists between academic and industrial research in automated driving development. Despite this, there is common sense that cooperative control approaches in automated vehicles will surpass the previously favored takeover paradigm in most driving situations due to enhanced driving performance and user experience. Yet, the application of these concepts in real driving situations remains unclear, and a holistic approach to driving cooperation is missing. Existing research has primarily focused on testing specific interaction scenarios and implementations. To address this gap and offer a contemporary perspective on designing human–vehicle cooperation in automated driving, we have developed a three-part taxonomy with the help of an extensive literature review. The taxonomy broadens the notion of driving cooperation towards a holistic and application-oriented view by encompassing (1) the “Cooperation Use Case”, (2) the “Cooperation Frame”, and (3) the “Human–Machine Interface”. We validate the taxonomy by categorizing related literature and providing a detailed analysis of an exemplar paper. The proposed taxonomy offers designers and researchers a concise overview of the current state of driver cooperation and insights for future work. Further, the taxonomy can guide automotive HMI designers in ideation, communication, comparison, and reflection of cooperative driving interfaces.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8030015
Authors: Bram van Deurzen Gustavo Alberto Rovelo Ruiz Daniël M. Bot Davy Vanacken Kris Luyten
Buttons are everywhere and are one of the most common interaction elements in both physical and digital interfaces. While virtual buttons offer versatility, enhancing them with realistic haptic feedback is challenging. Achieving this requires a comprehensive understanding of the tactile perception of physical buttons and their transferability to virtual counterparts. This research investigates tactile perception concerning button attributes such as shape, size, and roundness and their potential generalization across diverse button types. In our study, participants interacted with each of the 36 buttons in our search space and provided a response to which one they thought they were touching. The findings were used to establish six substitute buttons capable of effectively emulating tactile experiences across various buttons. In a second study, these substitute buttons were validated against virtual buttons in VR. Highlighting the potential use of the substitute buttons as haptic proxies for applications such as encountered-type haptics.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8020014
Authors: Nianmei Zhou Steven Devleminck Luc Geurts
This study investigates accessible and sensitive electrode solutions for detecting touches and squeezes on soft interfaces based on commercially available conductive polyurethane foam. Various electrode materials and configurations are explored, and for electrodes made of conductive threads, the static and dynamic electrical behaviors are studied in depth. In contrast to existing approaches that aim to minimize or stabilize contact resistance, we propose leveraging contact resistance to significantly enhance sensing sensitivity. Suggestions for future researchers and developers when building squeeze sensors based on this material are provided. Our findings offer insights for DIY enthusiasts and researchers, enabling them to develop sensitive soft interfaces for touch and squeeze interactions in an affordable and accessible manner and provide a completely soft user experience.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8020013
Authors: Gvidas Razevicius Anne Roudaut Abhijit Karnik
Deployable kinematic structures can transform themselves from a small closed configuration to a large deployed one. These structures are widely used in many engineering fields including aerospace, architecture, robotics and to some extent within HCI. In this paper, we investigate the use of a symmetric spherical deployable structure and its application to interface control. We present HoberUI, a bimanual symmetric tangible interface with 7 degrees of freedom and explore its use for manipulating 3D environments. We base this on the toy version of the deployable structure called the Hoberman sphere, which consists of pantographic scissor mechanisms and is capable of homogeneous shrinkage and expansion. We first explore the space for designing and implementing interactions through such kinematic structures and apply this to 3D object manipulation. We then explore HoberUI’s usability through a user evaluation that shows the intuitiveness and potential of using instrumented kinematic structures as input devices for bespoke applications.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8020012
Authors: Miah Dawes Katherine Rackliffe Amanda Lee Hughes Derek L. Hansen
This paper identifies subgenres of asymmetric virtual reality (AVR) games and proposes the AVR Game Genre (AVRGG) framework for developing AVR games. We examined 66 games “in the wild” to develop the AVRGG and used it to identify 5 subgenres of AVR games including David(s) vs. Goliath, Hide and Seek, Perspective Puzzle, Order Simulation, and Lifeline. We describe these genres, which account for nearly half of the 66 games reviewed, in terms of the AVRGG framework that highlights salient asymmetries in the mechanics, dynamics, and aesthetics categories. To evaluate the usefulness of the AVRGG framework, we conducted four workshops (two with the AVRGG framework and two without) with novice game designers who generated 16 original AVR game concepts. Comparisons between the workshop groups, observations of the design sessions, focus groups, and surveys showed the promise and limitations of the AVRGG framework as a design tool. We found that novice designers were able to understand and apply the AVRGG framework after only a brief introduction. The observations indicated two primary challenges that AVR designers face: balancing the game between VR and non-VR player(s) and generating original game concepts. The AVRGG framework helped overcome the balancing concerns due to its ability to inspire novice game designers with example subgenres and draw attention to the asymmetric mechanics and competitive/cooperative nature of games. While half of those who used the AVRGG framework to design with created games that fit directly into existing subgenres, the other half viewed the subgenres as “creative constraints” useful in jumpstarting novel game designs that combined, modified, or purposefully avoided existing subgenres. Additional benefits and limitations of the AVRGG framework are outlined in the paper.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8020011
Authors: Abdu Arslanyilmaz Margaret L. Briley Gregory V. Boerio Katie Petridis Ramlah Ilyas
There is a limited amount of research dedicated to designing and developing computing curricula specifically tailored for students with autism spectrum disorder (ASD), and thus far, no study has examined the effectiveness of an accessible computing curriculum designed specifically for students with ASD. The goal of this study is to evaluate the effectiveness of an accessible curriculum in improving the learning of computational thinking concepts (CTCs) such as sequences, loops, parallelism, conditionals, operators, and data, as well as the development of proficiency in computational thinking practices (CTPs) including experimenting and iterating, testing and debugging, reusing and remixing, and abstracting and modularizing. The study involved two groups, each comprising twenty-four students. One group received instruction using the accessible curriculum, while the other was taught with the original curriculum. Evaluation of students’ CTCs included the analysis of pretest and posttest scores for both groups, and their CTPs were assessed through artifact-based interview scores. The results indicated improvement in both groups concerning the learning of CTCs, with no significant difference between the two curricula. However, the accessible computing curriculum demonstrated significant enhancements in students’ proficiency in debugging and testing, iterating and experimenting, modularizing and abstracting, as well as remixing and reusing. The findings suggest the effectiveness of accessible computing curricula for students with ASD.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8020010
Authors: Taneli Nyyssönen Seppo Helle Teijo Lehtonen Jouni Smed
This paper presents two gesture-based user interfaces which were designed for a 3D design review in virtual reality (VR) with inspiration drawn from the shipbuilding industry’s need to streamline and make their processes more sustainable. The user interfaces, one focusing on single-hand (unimanual) gestures and the other focusing on dual-handed (bimanual) usage, are tested as a case study using 13 tasks. The unimanual approach attempts to provide a higher degree of flexibility, while the bimanual approach seeks to provide more control over the interaction. The interfaces were developed for the Meta Quest 2 VR headset using the Unity game engine. Hand-tracking (HT) is utilized due to potential usability benefits in comparison to standard controller-based user interfaces, which lack intuitiveness regarding the controls and can cause more strain. The user interfaces were tested with 25 test users, and the results indicate a preference toward the one-handed user interface with little variation in test user categories. Additionally, the testing order, which was counterbalanced, had a statistically significant impact on the preference and performance, indicating that learning novel interaction mechanisms requires an adjustment period for reliable results. VR sickness was also strongly experienced by a few users, and there were no signs that gesture controls would significantly alleviate it.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8020009
Authors: Tjaša Kermavnar Pieter M. A. Desmet
Existing studies of technology supporting meditation habit formation mainly focus on mobile applications which support users via reminders. A potentially more effective source of motivation could be contextual cues provided by meaningful objects in meaningful locations. This longitudinal mixed-methods 8-week study explored the effectiveness of such an object, Prana, in supporting forming meditation habits among seven novice meditators. First, the Meditation Intentions Questionnaire-24 and the Determinants of Meditation Practice Inventory-Revised were administered. The self-report habit index (SrHI) was administered before and after the study. Prana recorded meditation session times, while daily diaries captured subjective experiences. At the end of the study, the system usability scale, the ten-item personality inventory, and the brief self-control scale were completed, followed by individual semi-structured interviews. We expected to find an increase in meditation frequency and temporal consistency, but the results failed to confirm this. Participants meditated for between 16% and 84% of the study. The frequency decreased with time for four, decreased with subsequent increase for two, and remained stable for one of them. Daily meditation experiences were positive, and the perceived difficulty to start meditating was low. No relevant correlation was found between the perceived difficulty in starting to meditate and meditation experience overall; the latter was only weakly associated with the likelihood of meditating the next day. While meditation became more habitual for six participants, positive scores on SrHI were rare. Despite the inconclusive results, this study provides valuable insights into challenges and benefits of using a meditation device, as well as potential methodological difficulties in studying habit formation with physical devices.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8020008
Authors: Maximilian Rosilius Martin Spiertz Benedikt Wirsing Manuel Geuen Volker Bräutigam Bernd Ludwig
Even though assistance systems offer more potential due to the increasing maturity of the inherent technologies, Automatic Speech Recognition faces distinctive challenges in the industrial context. Speech recognition enables immersive assistance systems to handle inputs and commands hands-free during two-handed operative jobs. The results of the conducted study (with n = 22 participants) based on the counterbalanced within-subject design demonstrated the performance (word error rate and information transfer rate) of the HMD HoloLens 2 as a function of the sound pressure level of industrial noise. The negative influence of industrial noise was higher on the word error rate of dictation than on the information transfer rate of the speech command. Contrary to expectations, no statistically significant difference in performance was found between the stationary and non-stationary noise. Furthermore, this study confirmed the hypothesis that user acceptance was negatively influenced by erroneous speech interactions. Furthermore, the erroneous speech interaction had no statistically significant influence on the workload or physiological parameters (skin conductance level and heart rate). It can be summarized that Automatic Speech Recognition is not yet a capable interaction paradigm in an industrial context.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8020007
Authors: Apostolos Vrontos Verena Nitsch Christopher Brandl
This paper presents a thorough review of electrical muscle stimulation (EMS) in the context of augmented reality (AR) and virtual reality (VR), specifically focusing on its application in providing kinesthetic feedback. Our systematic review of 17 studies reveals the growing interest and potential of EMS in this domain, as evidenced by the growing body of literature and citations. The key elements presented in our review encompass a catalog of the applications developed to date, the specifics of the stimulation parameters used, the participant demographics of the studies, and the types of measures used in these research efforts. We discovered that EMS offers a versatile range of applications in AR/VR, from simulating physical interactions like touching virtual walls or objects to replicating the sensation of weight and impact. Notably, EMS has shown effectiveness in areas such as object handling and musical rhythm learning, indicating its broader potential beyond conventional haptic feedback mechanisms. However, our review also highlights major challenges in the research, such as inconsistent reporting of EMS parameters and a lack of diversity in study participants. These issues underscore the need for improved reporting standards and more inclusive research approaches to ensure wider applicability and reproducibility of results.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8020006
Authors: Clemens Reitelbach Kiemute Oyibo
Brain–computer interfaces (BCIs) based on steady-state visually evoked potentials (SSVEPs) have been well researched due to their easy system configuration, little or no user training and high information transfer rates. To elicit an SSVEP, a repetitive visual stimulus (RVS) is presented to the user. The properties of this RVS (e.g., frequency, luminance) have a significant influence on the BCI performance and user comfort. Several studies in this area in the last one-and-half decades have focused on evaluating different stimulus parameters (i.e., properties). However, there is little research on the synthesis of the existing studies, as the last review on the subject was published in 2010. Consequently, we conducted a scoping review of related studies on the influence of stimulus parameters on SSVEP response and user comfort, analyzed them and summarized the findings considering the physiological and neurological processes associated with BCI performance. In the review, we found that stimulus type, frequency, color contrast, luminance contrast and size/shape of the retinal image are the most important stimulus properties that influence SSVEP response. Regarding stimulus type, frequency and luminance, there is a trade-off between the best SSVEP response quality and visual comfort. Finally, since there is no unified measuring method for visual comfort and a lack of differentiation in the high-frequency band, we proposed a measuring method and a division of the band. In summary, the review highlights which stimulus properties are important to consider when designing SSVEP BCIs. It can be used as a reference point for future research in BCI, as it will help researchers to optimize the design of their SSVEP stimuli.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8010005
Authors: Maria Luisa Lorusso Francesca Borasio Paola Panetto Mariangela Curioni Giada Brotto Giulio Pons Alex Carsetti Massimo Molteni
Previous research has shown the importance of font type, size, and spacing to facilitate text reading in dyslexia. Great heterogeneity in the population of readers with specific learning disorders suggests that personalized parameters should be preferable compared to one-fits-all ones. A special automatized procedure was designed to select the most favorable parameters for both text visualization and text-to-speech conversion. A total of 78 primary and middle school students (29 typical readers, 49 children with atypical reading skills, either diagnosed as specific reading disorder or as special learning needs) took part in this study, which included the application of the procedure and a validation of its outcomes through a systematic comparison of the use of the personalized versus standard fonts and voices in reading and writing tests. The results show a significant advantage for the personalized parameters. Moreover, in the case of text-to-speech personalization, the advantage is significantly larger for dyslexic readers than for typical readers. These results confirm the usefulness of a personalization approach in providing support to facilitate learning in dyslexic students.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8010004
Authors: Vincenzo Ferrari Nadia Cattari Sara Condino Fabrizio Cutolo
Head-mounted displays (HMDs) are hands-free devices particularly useful for guiding near-field tasks such as manual surgical procedures. See-through HMDs do not significantly alter the user’s direct view of the world, but the optical merging of real and virtual information can hinder their coherent and simultaneous perception. In particular, the coherence between the real and virtual content is affected by a viewpoint parallax-related misalignment, which is due to the inaccessibility of the user-perceived reality through the semi-transparent optical combiner of the OST Optical See-Through (OST) display. Recent works demonstrated that a proper selection of the collimation optics of the HMD significantly mitigates the parallax-related registration error without the need for any eye-tracking cameras and/or for any error-prone alignment-based display calibration procedures. These solutions are either based on HMDs that projects the virtual imaging plane directly at arm’s distance, or they require the integration on the HMD of additional lenses to optically move the image of the observed scene to the virtual projection plane of the HMD. This paper describes and evaluates the pros and cons of both the suggested solutions by providing an analytical estimation of the residual registration error achieved with both solutions and discussing the perceptual issues generated by the simultaneous focalization of real and virtual information.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8010003
Authors: Amaryllis-Chryssi Malegiannaki Evangelia Garefalaki Nikolaos Pellas Mary H. Kosmidis
Early detection is crucial for addressing attention deficits commonly associated with Traumatic brain injury (TBI), informing effective rehabilitation planning and intervention. While traditional neuropsychological assessments have been conventionally used to evaluate attention deficits, their limited ecological validity presents notable challenges. This study explores the efficacy and validity of a novel virtual reality test, the Computerized Battery for the Assessment of Attention Disorders (CBAAD), among a cohort of TBI survivors (n = 20), in comparison to a healthy control group (n = 20). Participants, ranging in age from 21 to 62 years, were administered a comprehensive neuropsychological assessment, including the CBAAD and the Attention Related Cognitive Errors Scale. While variations in attentional performance were observed across age cohorts, the study found no statistically significant age-related effects within either group. The CBAAD demonstrated sensitivity to attentional dysfunction in the TBI group, establishing its value as a comprehensive test battery for assessing attention in this specific population. Regression analyses demonstrated the CBAAD’s effectiveness in predicting real-life attentional errors reported by TBI patients. In summary, the CBAAD demonstrates sensitivity to attentional dysfunction in TBI patients and the ability to predict real-world attentional errors, establishing its value as a comprehensive test battery for assessing attention in this specific population. Its implementation holds promise for enhancing the early identification of attentional impairments and facilitating tailored rehabilitation strategies for TBI patients.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8010002
Authors: Nikolaos Partarakis Xenophon Zabulis
Craft education and training are important for preserving cultural heritage and fostering artisanal skills. However, the pedagogical challenges in this domain are numerous. This research paper presents a comprehensive framework for applying Cognitive Load Theory to enhance craft education and training via eLearning platforms. In this study, practical guidelines based on CLT principles are provided to optimize the instructional design and content delivery. These guidelines scaffold craft learning experiences within eLearning platforms and encompass strategies to manage cognitive load, promote active learning, and facilitate gradual transition. Subsequently, the paper details the implementation of these guidelines within a popular eLearning platform, Moodle, emphasizing its adaptability and utility for craft education. It discusses the customization of Moodle courses to align with the cognitive load management principles, providing a practical blueprint for educators and instructional designers. The research culminates in a case study, wherein the guidelines are applied to a craft eLearning course using Moodle.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti8010001
Authors: Francesco Ganis Andrea Gulli Federico Fontana Stefania Serafin
Sensory substitution and augmentation are pivotal concepts in multi-modal perception, particularly when confronting the challenges associated with impaired or missing sense rehabilitation. The present systematic review investigates the role of haptics for the hearing impaired in training or gamified activities. We applied a set of keywords to the Scopus® and PubMed® databases, obtaining a collection of 35 manuscripts spanning 23 years. Each article has been categorized following a documented procedure and thoroughly analyzed. Our findings reveal a rising number of studies in this field in the last five years, mostly testing the effectiveness of the developed rehabilitative method (77.14%). Despite a wide variety in almost every category we analyzed, such as haptic devices, body location, and data collection, we report a constant difficulty in recruitment, reflected in the low number of hearing-impaired participants (mean of 8.31). This review found that in all six papers reporting statistically significant positive results, the vibrotactile device in use generated vibrations starting from a sound, suggesting that some perceptual aspects connected to sound are transmittable through touch. This fact provides evidence that haptics and vibrotactile devices could be viable solutions for hearing-impaired rehabilitation and training.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7120118
Authors: Jill A. Dosso Jaya N. Kailley Susanna E. Martin Julie M. Robillard
Social robots have the potential to support health and quality of life for children experiencing anxiety. We engaged families with lived experiences of pediatric anxiety in social robot development to explore desired design features, application areas, and emotion functionalities of social robots in anxiety care. We conducted 10 online co-creation workshops with (1) children with anxiety aged 7–13 (n = 24) with their family members (n = 20), and (2) youth with anxiety aged 14–18 (n = 12). Workshop participation included a validated robot expectations scale, anonymous polls, and discussion. Transcripts and text responses were subjected to content analysis. A lived experience expert group provided feedback throughout the research. Participants desired a pet-like robot with a soft texture, expressive eyes, and emotion detection to support activities of daily living. Specific anxiety-related applications included breathing exercises, managing distressing thoughts, and encouragement. Emotional alignment, the design of a robot’s emotional display, and the emotional impacts of an interaction were discussed. Privacy and the replacement of human interaction were concerns. We identify pediatric anxiety-specific design features, applications, and affective considerations for existing and future social robots. Our findings highlight the need for customizability and robust emotional functionality in social robot technologies intended to support the health and care of children living with anxiety.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7120117
Authors: Andrés N. Vargas González Brian Williamson Joseph J. LaViola
Creating digital twins of real objects is becoming more popular, with smartphones providing 3D scanning capabilities. Adding semantics to the reconstructed virtual objects is important to possibly reproduce training scenarios that otherwise could demand significant resources or happen in dangerous scenarios in some cases. The aim of this work is to evaluate the usability of authoring object component behaviors in immersive and non-immersive approaches. Therefore, we present an evaluation of the perceived ease of use to author moving parts of objects under three different conditions: desktop, augmented reality (AR) and virtual reality (VR). This paper provides insights into the perceived benefits and issues that domain experts might encounter when authoring geometrical component behaviors across each interface. A within-subject study is the major contribution of this work, from which is presented an analysis based on the usability, workload and user interface preferences of participants in the study. To reduce confounding variables in the study, we ensured that the virtual objects and the environment used for the evaluation were digital twins of the real objects and the environment that the experiment took place in. Results show that the desktop interface was perceived as more efficient and easier to use based on usability and workload measures. The desktop was preferred for performing component selection but no difference was found in the preference for defining a behavior and visualizing it. Based on these results, a set of recommendations and future directions are provided to achieve a more usable, immersive authoring experience.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7120116
Authors: Manjeet Singh Shaun Bangay Atul Sajjanhar
There are various ways that teachers manage student disengagement levels during their class lessons, and managing disengagement can be both stressful and challenging, especially since each student is unique. Methods and techniques utilised are specific to teachers’ own experience level, subject knowledge, and teaching styles. We report on the techniques and methods teachers utilise to identify, mitigate, and measure student disengagement during class lessons; the paper presents the results of a mixed-methods, multisession study design comprising gathered qualitative and quantitative data to enable a greater understanding. Eight educators who were full-time educators with varying years of experience from three different schools, who taught or had taught English, maths, and science subjects at the primary school level, participated in this study. The study also observed that teachers used three AR applications and collected valuable feedback on their perspectives by using analytics generated by AR applications to help manage student disengagement. A postsession survey tool was used to gather the perceived importance and ranking of the techniques and methods discussed by the teachers during the previous sessions. The results showed that the majority of teachers deemed spending “Time on Tasks” and giving “Feedback/Reflections” most suited for measuring disengagement, and encouraging “Movement” and use of “Technology” emerged as the most favoured for mitigating disengagement. For utilising AR enhanced analytics in mitigating and measuring student disengagement, the data suggested a difference in perspectives based on teachers’ teaching levels, especially concerning conversations and the use of technology devices. The study did not find conclusive evidence of differences based on teachers’ teaching subjects and there was a notable distinction in building positive relationships among English teachers. This leads to the suggestion that subject-specific pedagogy might influence the perceived effectiveness of using AR-generated analytics in mitigating and measuring student disengagement.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7120115
Authors: Matteo Al Kalak Lorenzo Baraldi
The article aims to reflect on the Lodovico media library, a digital repository preserving the digitised cultural heritage of the Emilia-Romagna region. The first part covers the project’s history and the challenges encountered during its setup phase, and we also explore the co-creation approach employed in defining the metadata architecture. The discussion extends by outlining the key features of shared metadata, illustrating their application to diverse digital objects within the Lodovico media library. Following a concise examination of the methodology for collecting/creating data and the initial research findings, the article concludes by highlighting the project’s potential in the realm of automatic handwriting recognition processes.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7120114
Authors: Kalliopi Apostolou Filip Škola Fotis Liarokapis
This study explores the conventional ‘funneling’ method by introducing two extra locations beyond the virtual reality (VR) controller boundaries, terming it the extended funneling technique. Thirty-two participants engaged in a localization task, with their responses recorded using eye-tracking technology. They were tasked with localizing a virtual ping-pong ball as it bounced both within and outside their virtual hands on a virtual board. Both the experimental and control groups received simultaneous spatial audio and vibrotactile feedback. The experimental group received vibrotactile feedback with extended funneling, while the control group received vibrotactile feedback without funneling for comparison. The results indicate that the experimental group, benefiting from the extended funneling technique, demonstrated a significantly higher accuracy rate (41.79%) in localizing audio–vibrotactile stimuli compared to the control group (28.21%). No significant differences emerged in embodiment or workload scores. These findings highlight the effectiveness of extended funneling for enhancing the localization of sensory stimuli in VR.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7120113
Authors: Olivia Zechner Daniel García Guirao Helmut Schrom-Feiertag Georg Regal Jakob Carl Uhl Lina Gyllencreutz David Sjöberg Manfred Tscheligi
Mixed reality (MR) technology has the potential to enhance the disaster preparedness of medical first responders in mass-casualty incidents through new training methods. In this manuscript, we present an MR training solution based on requirements collected from experienced medical first responders and technical experts, regular end-user feedback received through the iterative design process used to develop a prototype and feedback from two initial field trials. We discuss key features essential for an effective MR training system, including flexible scenario design, added realism through patient simulator manikins and objective performance assessment. Current technological challenges such as the responsiveness of avatars and the complexity of smart scenario control are also addressed, along with the future potential for integrating artificial intelligence. Furthermore, an advanced analytics and statistics tool that incorporates complex data integration, machine learning for data analysis and visualization techniques for performance evaluation is presented.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7120112
Authors: Miriam Mendoza López Petronila Mireia Alcaraz Artero Carlos Truque Díaz Manuel Pardo Ríos Juan José Hernández Morante Rafael Melendreras Ruiz
This study focuses on the development and assessment of a serious game for health (SGH) aimed at educating children about cardiopulmonary resuscitation (CPR). A video game was created using the Berkeley Snap platform, which uses block programming. Eye-tracking technology was utilized to validate the graphic design. To assess the tool’s effectiveness, a pre-post analytical study was conducted with primary education children to measure the knowledge acquired. The study involved 52 participants with a mean age of 9 years. The results from a custom questionnaire used to measure their theoretical CPR knowledge showed significant improvements in CPR knowledge after the use of the videogame, and their emotional responses improved as well. The assessment of the knowledge acquired through the video game obtained an average score of 5.25 out of 6. Ten video segments consisting of 500 frames each (20 s of video per segment) were analyzed. Within these segments, specific areas that captured the most relevant interaction elements were selected to measure the child’s attention during game play. The average number of gaze fixations, indicating the points in which the child’s attention was placed within the area of interest, was 361.5 out of 500. In conclusion, the utilization of SGH may be an effective method for educating kids about CPR, to provide them with fundamental knowledge relevant to their age group.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7120111
Authors: Marco Romano Alessandro Frolli Alessandro Aloisio Claudio Russello Angelo Rega Francesco Cerciello Fabio Bisogni
In recent years, there has been a surge of interest in affordable and accessible extended reality devices. Big tech companies like Apple and Meta have announced advanced devices expected to become more prevalent in everyday life. As younger generations embrace immersive digital realities for socialization, entertainment, and information retrieval, there is a need to explore immersive digital technologies that support experiential learning and reevaluate educational approaches. In Italy, the COVID-19 pandemic has sparked a growing interest in immersive virtual reality (VR) and the metaverse for distance education. However, the integration of VR in Italian schools could be faster, primarily due to cost and teacher knowledge challenges. Our study aims to involve high school teachers in a practical workshop to assess their knowledge, skills, and intention to use VR in their teaching after brief training. The focus is on evaluating the acceptability of VR for educational purposes among Italian high school teachers. The workshop involved up to 16 teachers at once and was repeated eight times to reach 120 teachers. Participants received VR training and explored three educational VR applications. The results show that teachers are interested in learning and integrating VR into their lessons. They believe it can enhance teaching practices by actively engaging students and enabling experiential learning. This work provides an overview of the current state of VR in education, describes the workshop with high school teachers, and presents the obtained results.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7120110
Authors: Derek Moskowitz Diana R. Sanchez Brian Trinh
Video vignettes are one form of virtualized vignettes that may build upon traditional text vignettes and enable research participants to see and experience a unique scenario that is better translated visually than through a written text. This study examined using video vignettes to study perceptions of leaders. Participants watched virtualized, video vignettes depicting a male leader expressing either a masculine, gender-conforming expression or a feminine, gender-nonconforming expression. Participants evaluated these leaders on measures of leadership likability and leadership effectiveness. Results demonstrated that the videos of the masculine male leader were perceived as more likable and more effective than the videos of the feminine male leader. This relationship was not moderated by gender-related expectations the participants had. This finding reveals that there is a prototypical expectation that male leaders who depicted masculine expressions are more likable and effective. Additionally, we also found that the participant’s individual ideologies of gender-related expectations and conformity expectations were related to the results in unique ways. When the participant believed an ideal leader should have higher versus lower feminine traits, those participants also rated both leaders more positively regardless of the type of gender expression that was depicted in the video vignettes. In contrast, participants with strong expectations that others should conform to gender norms in social situations generally rated both leaders more negatively, regardless of the gender expression in the video vignette. The results are discussed in detail and the benefits of using virtualized and video vignettes are discussed.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7120109
Authors: Adrià Gabarnet Guillem Feixas Adrián Montesano
Several studies have researched how people can use the anonymity of the Internet to explore different aspects of their identity. There are many different platforms where one can create a “virtual self” by actively choosing how one presents themselves to others, and each one is designed with different characteristics depending on their purpose: from socialization to professional networking or even entertainment. Different studies have usually focused on specific platforms, but there is no one comparing how people construe their virtual self across different online environments. In this review, we aimed to synthesize research studies carried out about the construal of one’s identity within online platforms and how it can relate to different aspects of participants’ offline identity, such as their self-esteem or self-concept clarity. Data were gathered from 34 publications that researched this topic across different kinds of online platforms. We conducted a quality assessment and a narrative synthesis, reporting and comparing the main findings, as well as identifying possible gaps in the literature. Many studies have explored the idea of people creating either an idealized or accurate version of themselves when construing their online identities. Others have also contemplated the possibility of exploring identities that diverge from both the actual and the ideal self or even an actively negative one. The latter was studied more in the context of video games and other avatar-mediated worlds. We found that people with low self-esteem create a more idealized virtual identity rather than a virtual self that is similar to their offline self. Other variables such as the purpose for using the online platform or self-concept clarity also had a role in virtual self construal, although the latter was only studied in the context of social media.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7110108
Authors: Federico Morosi Niccolò Becattini Giandomenico Caruso Gaetano Cascini
In recent years, research reached a very high level of development and validation of augmented prototyping systems in support of collaborative design activities. However, there is still great scepticism in companies when it comes to integrating these new technologies within a consolidated working model. Among others, the main barrier to overcome concerns the lack of understanding of the impact of AR systems on the key objectives of a business, such as improving its efficiency and revenue. For this reason, this paper aims to quantify these indicators by observing the technological impact not on a single design session but on an entire product development process, during which the aspects related to its integration are also considered. Thanks to the collaboration with a design agency, it was possible to compare parameters such as the lead time, number of iterations, person-hours and costs between two similar and realistic projects in which only one was supported by projection-based AR technology.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7110107
Authors: Pia von Terzi Sarah Diefenbach
Although many of our interactions with technology nowadays take place in public places (e.g., using a mobile phone in public transportation), research and design on Human-Computer Interaction (HCI) has paid little attention to how this kind of technology usage affects others present—and vice versa. To illustrate the perspective of the attendant, i.e., a person who is not interacting with technology themselves but co-experiencing it as listener or viewer, we developed the so-called Attendant Card Set (ACS). In two studies, an expert survey and a student workshop, we tested its practical applicability and usefulness. It showed not only that experts assess the cards positively, i.e., helpful, informative, and relevant, but also that the cards can be used with laypersons for perspective-taking, creative ideation, and discussions. Thus, analyzing and/or comparing the experience of different types with the help of the ACS provides a unique approach to the consideration of the attendant perspective in the research and development process. Limitations of the present research and opportunities for future tool applications are discussed. In addition to establishing this concept in HCI, we also see potential in the transferability to other areas and contexts such as the design of public space or non-technological products.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7110106
Authors: Ningyuan Sun Jean Botev
Interacting with virtual agents in immersive settings is becoming increasingly common thanks to the proliferation of dedicated media devices, such as consumer-grade virtual and augmented reality headsets. These technologies offer many advantages, e.g., in terms of presence and engagement, and can impact user behavior and attitudes toward virtual agents. Recognizing and understanding these effects is essential, especially in critical contexts involving the delegation of high-stake decisions to virtual agents. This article presents two experiments that explore users’ delegatory behavior toward virtual agents experienced via different media devices that vary in their technological immersion, i.e., a device’s technical capacity to deliver immersive experiences. The experiments’ results suggest that technological immersion is not a significant factor in users’ delegation decisions. Thus, for virtual agents designed to carry out critical tasks, developers may focus on other relevant factors, such as agents’ trustworthiness or performance.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7110105
Authors: Tamara J. Skootsky Diana R. Sanchez Kentaro Kawasaki
The relationship between workplace communication and affective outcomes, specifically connectedness at work and affective organizational commitment, is one that warrants further investigation for practical usage in the increasingly multimodal workplace. This study considers the frequency of use across five communication modalities, that being face-to-face, email, phone calls, instant messaging, and video calls, in relation to affective outcomes, as well as their relationships with communication meaningfulness. Employed participants (n = 516) completed an online survey in which they self-reported weekly communication tendencies, experienced connectedness, and affective organizational commitment. The final sample consisted of participants across 20 different industries in the United States. The most respondents worked in Health Care and Social Assistance or Professional, Scientific, and Technical Services (15% each), followed closely by respondents from Educational Services or Finance and Insurance (13% each). Data collection took place in between October 2021 and January 2022. Generally, participants who reported more frequent communication at work reported higher levels of connectedness and higher affective organizational commitment. Employees who found their communication more meaningful (irrespective of frequency) felt the most emotionally connected. Unique benefits of different communication modalities, as well as implications for hybrid and remote organizations, are discussed.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7110104
Authors: Andrew C. Pooley Andrew May Val Mitchell
This paper extends current research on Human–Robot Interaction (HRI) within Human–Computer Interaction (HCI), focusing on how future virtual agents and communication robots can support the temporal structures and routines within the home. We recruited representatives from 15 households with varied compositions, ranging from single inhabitants to full nest families. Drawing upon P.G. Wodehouse’s The Inimitable Jeeves as an inspiration, the methodology sought to imitate the relationship between a recruitment agency worker (researcher) and an employer (participant) seeking to hire a personal assistant (e.g., a virtual agent or communication robot device) for their home. A ‘household audit’ comprising a guided household tour and an architectural survey was conducted to ascertain the nuanced spatiotemporal routines within the home. The study analysed the responses of participants using the Labovian narrative schema, a traditional method in linguistics research. The findings were then examined through the lens of Reddy et al.’s temporal features of work to understand how domestic work unfolds within the home from a temporal perspective. We argue that the temporal concepts discussed by Reddy et al. provided valuable insights into the temporal dynamics of everyday activities and could inform the design of virtual agents and communication robotic devices to fulfil their roles as domesticated ‘personal assistants’.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7110103
Authors: Dimitrios Ramandanis Stelios Xinogalos
A chatbot, or else a conversational agent (CA), is a technology that is used in order to imitate the process of a conversation between a human being and a software application for supporting specific services. The utilization of this technology has been increasing considerably over the past five years, particularly in education where CAs are mostly utilized as teaching assistants that provide educational content. This paper aims to contribute to the existing body of knowledge by systematically reviewing the support provided by chatbots both to educational institutions and their students, investigating their capabilities in further detail, and highlighting the various ways that this technology could and should be used in order to maximize its benefits. Emphasis is given to analyzing and synthesizing the emerging roles of CAs, usage recommendations and suggestions, student’s desires, and challenges recorded in the literature. For this reason, a systematic literature review (SLR) was carried out using the PRISMA framework in order to minimize the common biases and limitations of SLRs. However, we must note that the SLR presented has specific limitations, namely using only Scopus as a search engine, utilizing a general search query, and selecting only journal articles published in English in the last five years.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7110102
Authors: Raffaella Trocchianesi Letizia Bollini
In this essay, we are interested in investigating some of the possible relations between design and digital humanities. In particular, we analyze the contribution that communication and interface design can bring to digital humanities. In a scene currently characterized by a heterogeneous set of activities and humanistic, technological, and cultural studies, the involvement of design seems confined to the development of digital instruments in accessing, exploring, and manipulating cultural data. How can design and the humanities work in an interdisciplinary way in order to shape new digital means to explore humanistic content? This essay presents four case studies (three of them developed by the authors), each of which suggests some methods and tools focused on the interdisciplinary relationships of scholars. The findings are both models of collaboration and models of digital architecture (data visualization) and showcase applied digital interactive platforms that present several paths to discovering different levels of content in the fields of art, psychology, literature, and history. In conclusion, this essay presents a manifesto focusing on ten points of virtuous relation between design humanities and the field of information visualization.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7110101
Authors: Chih-Kuan Lin Chien-Hsiung Chen Kai-Shuan Shen
This study explored drivers’ emotion-based impressions of car-borne central control platforms (CBCCPs) for personal-use vehicles. Thus, this preference-based study examined experts’ and drivers’ opinions regarding the appeal of CBCCPs from the perspective of Miryoku engineering. To this end, this study analyzed data via the EGM (evaluation grid method (EGM) and quantification theory type I. Results: Drivers’ preferences for specific CBCCP design characteristics were categorized into the factors “legible”, convenient”, and “tasteful”, which comprised the core of the EGM semantic hierarchical diagram. In addition, the importance of CBCCPs’ appeal factors and characteristics was assessed through quantification theory type I. The findings of this study provide valuable insights for designers, manufacturers, and researchers interested in the design of CBCCPs. Additionally, the results of this study can contribute to research on applied psychology, human–computer interactions, and car interface design.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7100100
Authors: Thomas Spielhofer Renate Motschnig
This research pursues the question of how the computer-generated analysis and visualization of communication can foster collaboration in teams that work together online. The audio data of regular online video meetings of three different teams were analyzed. Structural information regarding their communication was visualized in a communication report, and then, discussed with the teams in so-called digitally supported coaching (DSC) sessions. The aim of the DSC is to improve team collaboration by discerning helpful and less helpful patterns in the teams’ communication. This report allows us to recognize individual positions within the teams, as well as communication structures, such as conversational turn taking, that are relevant for group intelligence, as other research has shown. The findings pertaining to the team members during the DSC were gathered via questionnaires. These qualitative data were then matched with the quantitative data derived from the calls, particularly social network analysis (SNA). The SNA was inferred using the average number of interactions between the participants as measured in the calls. The qualitative findings of the teams were then cross-checked with the quantitative analysis. As a result, the assessment of team members’ roles was highly coherent with the SNA. Furthermore, all teams managed to derive concrete measures for improving their collaboration based on the reflection in the DSC.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7100099
Authors: Patrick Coe Grigori Evreinov Mounia Ziat Roope Raisamo
In this paper, we report a method of implementing a universal volumetric haptic actuation platform which can be adapted to fit a wide variety of visual displays with flat surfaces. This platform aims to enable the simulation of the 3D features of input interfaces. This goal is achieved using four readily available stepper motors in a diagonal cross configuration with which we can quickly change the position of a surface in a manner that can render these volumetric features. In our research, we use a Microsoft Surface Go tablet placed on the haptic enhancement actuation platform to replicate the exploratory features of virtual keyboard keycaps displayed on the touchscreen. We ask seven participants to explore the surface of a virtual keypad comprised of 12 keycaps. As a second task, random key positions are announced one at a time, which the participant is expected to locate. These experiments are used to understand how and with what fidelity the volumetric feedback could improve performance (detection time, track length, and error rate) of detecting the specific keycaps location with haptic feedback and in the absence of visual feedback. Participants complete the tasks with great success (p < 0.05). In addition, their ability to feel convex keycaps is confirmed within the subjective comments.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7100098
Authors: Ana Beatriz Marques Vasco Branco Rui Costa Nina Costa
Immersive Unit Visualization is an emergent form of visualization that arose from Immersive Analytics where, unlike traditional visualizations, each data point is represented by an individual visual mark in an immersive virtual environment. This practice has focused almost exclusively on virtual reality, excluding augmented reality (AR). This article develops and tests a prototype of an Immersive Unit Visualization (Floating Companies II) with two AR devices: head-mounted display (HMD) and hand-held display (HHD). Results from the testing sessions with 20 users were analyzed through qualitative research analysis and thematic coding indicating that, while the HHD enabled a first contact with AR visualization on a familiar device, HMD improved the perception of hybrid space by supporting greater stability of virtual content, wider field of view, improved spatial perception, increased sense of immersion, and more realistic simulation, which had an impact on information reading and sense-making. The materialization of abstract quantitative values into concrete reality through its simulation in the real environment and the ludic dimension stand out as important opportunities for this type of visualization. This paper investigates the aspects distinguishing two experiences regarding data visualization in hybrid space, and characterizes ways of seeing information with AR, identifying opportunities to advance information design research.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7100097
Authors: Maryam Aziz Achraf Othman
Sign language (SL) avatar systems aid communication between the hearing and deaf communities. Despite technological progress, there is a lack of a standardized avatar development framework. This paper offers a systematic review of SL avatar systems spanning from 1982 to 2022. Using PRISMA guidelines, we shortlisted 47 papers from an initial 1765, focusing on sign synthesis techniques, corpora, design strategies, and facial expression methods. We also discuss both objective and subjective evaluation methodologies. Our findings highlight key trends and suggest new research avenues for improving SL avatars.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7100095
Authors: Emmanouil Lionakis Konstantinos Karampidis Giorgos Papadourakis
The field of brain–computer interface (BCI) enables us to establish a pathway between the human brain and computers, with applications in the medical and nonmedical field. Brain computer interfaces can have a significant impact on the way humans interact with machines. In recent years, the surge in computational power has enabled deep learning algorithms to act as a robust avenue for leveraging BCIs. This paper provides an up-to-date review of deep and hybrid deep learning techniques utilized in the field of BCI through motor imagery. It delves into the adoption of deep learning techniques, including convolutional neural networks (CNNs), autoencoders (AEs), and recurrent structures such as long short-term memory (LSTM) networks. Moreover, hybrid approaches, such as combining CNNs with LSTMs or AEs and other techniques, are reviewed for their potential to enhance classification performance. Finally, we address challenges within motor imagery BCIs and highlight further research directions in this emerging field.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7100096
Authors: Alberto Sanchez-Acedo Alejandro Carbonell-Alcocer Manuel Gertrudix Jose Luis Rubio-Tamayo
Immersive journalism is a new form of media communication that uses extended reality systems to produce its content. Despite the possibilities it offers, its use is still limited in the media due to the lack of systematised and scientific knowledge regarding its application. This is a problem because it is a very powerful technology that changes the way audiences receive information and can be used both for new forms of storytelling that generate greater user engagement and for very sophisticated disinformation, which is why it is really important to study it. This study analyses articles published in the last 5 years that cover the use of extended technologies and the metaverse applied to immersive journalism. A systematic literature review applying PRISMA was carried out to identify literature within Web of Science, Scopus and Google Scholar (n = 61). Quantitative and qualitative analyses were conducted on the data collection techniques, the type of the data and the analysis techniques used. The results show a low level of methodological maturity, with research that is fundamentally descriptive and not very formalised, which limits the scope of its results and, therefore, the transfer of knowledge for its application in the configuration of new immersive journalistic products. The metaverse and extended technologies are considered independently and with distinct applications. It is concluded that research in this area is still in an initial exploratory and generalist stage that offers results that are not yet applicable to the promotion of this type of media format.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7100094
Authors: Sari Yli-Kauhaluoma Milt Statheropoulos Anne Zygmanowski Osmo Anttalainen Hanna Hakulinen Maria Theodora Kontogianni Matti Kuula Johannes Pernaa Paula Vanninen
Public warning systems are an essential element of safe cities. However, the functionality of neither traditional nor digital emergency warnings is understood well enough from the perspective of citizens. This study examines smart city development from the perspective of safety by exploring citizens’ viewpoints. It investigates people’s perceptions of the ways in which they obtain warnings and information about emergencies involving health risks. Data were collected in the form of focus group interviews and semi-structured interviews in Finland, Germany, and Greece. The results suggest that people place a lot of trust in their social network, receiving text messages, and their ability to use web-based search engines in order to obtain public warnings. The study discusses the challenges identified by citizens in the use of conventional radio and television transmissions and sirens for public warnings. Based on the results, citizens demonstrate informed ignorance about existing mobile emergency applications. Our results imply that it is not sufficient to build emergency communication infrastructure: the development of smart, safe cities requires continuous work and the integration of both hard and soft infrastructure-oriented strategies, i.e., technological infrastructure development including digitalisation and education, advancement of knowledge, and participation of people. Both strategic aspects are essential to enable people to take advantage of novel digital applications in emergency situations.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7100093
Authors: Sana Salman Deborah Richards Mark Dras
Relational cues are extracts from actual verbal dialogues that help build the therapist–patient working alliance and stronger bond through the depiction of empathy, respect and openness. ECAs (Embodied conversational agents) are human-like virtual agents that exhibit verbal and non-verbal behaviours. In the digital health space, ECAs act as health coaches or experts. ECA dialogues have previously been designed to include relational cues to motivate patients to change their current behaviours and encourage adherence to a treatment plan. However, there is little understanding of who finds specific relational cues delivered by an ECA helpful or not. Drawing the literature together, we have categorised relational cues into empowering, working alliance, affirmative and social dialogue. In this study, we have embedded the dialogue of Alex, an ECA, to encourage healthy behaviours with all the relational cues (empathic Alex) or with none of the relational cues (neutral Alex). A total of 206 participants were randomly assigned to interact with either empathic or neutral Alex and were also asked to rate the helpfulness of selected relational cues. We explore if the perceived helpfulness of the relational cues is a good predictor of users’ intention to change the recommended health behaviours and/or development of a working alliance. Our models also investigate the impact of individual factors, including gender, age, culture and personality traits of the users. The idea is to establish whether a certain group of individuals having similarities in terms of individual factors found a particular cue or group of cues helpful. This will establish future versions of Alex and allow Alex to tailor its dialogue to specific groups, as well as help in building ECAs with multiple personalities and roles.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7090092
Authors: Rima Shishakly Mohammed Amin Almaiah Shaha Al-Otaibi Abdalwali Lutfi Mahmaod Alrawad Ahmed Almulhem
Mobile learning has become increasingly important for higher education due to its numerous advantages and transformative potential. The aim of this study is to investigate how students perceive and utilize mobile learning (m-learning) services in universities. To achieve this objective, a conceptual model was developed, combining the TAM with additional new determinants, including perceived security, perceived trust, perceived risk, and service quality. The primary goal of this model is to assess the adoption of m-learning apps among users in university settings. To evaluate the proposed model, SEM was utilized to test the research model. The findings of the study highlight the critical roles of perceived security, perceived trust, and service quality in promoting the adoption of m-learning apps. Moreover, the results indicate that perceived risk negatively impacts both students’ trust and their attitudes towards using mobile learning services. The study reveals that the perceived trust, and service quality factors positively influence students’ attitudes towards adopting m-learning apps. These research findings hold significant implications for universities and academia, offering valuable insights to devise effective strategies for increasing the utilization of m- learning services among students. By gaining a deeper understanding of students’ perceptions and acceptance, universities can optimize their m-learning offerings to cater to students’ needs and preferences more effectively.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7090091
Authors: Mouadh Guesmi Mohamed Amine Chatti Lamees Kadhim Shoeb Joarder Qurat Ul Ain
The fast growth of data in the academic field has contributed to making recommendation systems for scientific papers more popular. Content-based filtering (CBF), a pivotal technique in recommender systems (RS), holds particular significance in the realm of scientific publication recommendations. In a content-based scientific publication RS, recommendations are composed by observing the features of users and papers. Content-based recommendation encompasses three primary steps, namely, item representation, user modeling, and recommendation generation. A crucial part of generating recommendations is the user modeling process. Nevertheless, this step is often neglected in existing content-based scientific publication RS. Moreover, most existing approaches do not capture the semantics of user models and papers. To address these limitations, in this paper we present a transparent Recommendation and Interest Modeling Application (RIMA), a content-based scientific publication RS that implicitly derives user interest models from their authored papers. To address the semantic issues, RIMA combines word embedding-based keyphrase extraction techniques with knowledge bases to generate semantically-enriched user interest models, and additionally leverages pretrained transformer sentence encoders to represent user models and papers and compute their similarities. The effectiveness of our approach was assessed through an offline evaluation by conducting extensive experiments on various datasets along with user study (N = 22), demonstrating that (a) combining SIFRank and SqueezeBERT as an embedding-based keyphrase extraction method with DBpedia as a knowledge base improved the quality of the user interest modeling step, and (b) using the msmarco-distilbert-base-tas-b sentence transformer model achieved better results in the recommendation generation step.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7090090
Authors: Ronald Böck
Humans are considered to be communicative, usually interacting in dyads or groups. In this paper, we investigate group interactions regarding performance in a rather formal gathering. In particular, a collection of ten performance indicators used in social group sciences is used to assess the outcomes of the meetings in this manuscript, in an automatic, machine learning-based way. For this, the Parking Lot Corpus, comprising 70 meetings in total, is analysed. At first, we obtain baseline results for the automatic prediction of performance results on the corpus. This is the first time the Parking Lot Corpus is tapped in this sense. Additionally, we compare baseline values to those obtained, utilising bidirectional long-short term memories. For multiple performance indicators, improvements in the baseline results are able to be achieved. Furthermore, the experiments showed a trend that the acoustic material of the remaining group should use for the prediction of team performance.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7090089
Authors: Tamlyn Young Mark T. Marshall
Augmented reality offers many artistic possibilities when it comes to the creation of place-based public artworks. In this paper, we present a series of works around the topic of augmented reality (AR) art and place-based storytelling, including the use of walking as a creative method, a series of workshops with emerging artists, public AR art collaborations and a study to examine user experience when interacting with such artworks. Our findings from these works show the potential of integrating augmented reality with public physical artworks and offer guidance to artists and AR developers on how to expand this potential. For artists, we show the importance of the space in which the artwork will be placed and provide guidance on how to work with the space. For developers, we find that there is a need to create tools that work with artists’ existing practices and to investigate how to expand augmented reality past the limitations of site- or piece-specific apps.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7090088
Authors: Georg Regal Daniele Pretolesi Helmut Schrom-Feiertag Jaison Puthenkalam Massimo Migliorini Elios De Maio Francesca Scarrone Marina Nadalin Massimiliano Guarneri Grace P. Xerri Daniele Di Giovanni Paola Tessari Federica Genna Andrea D’Angelo Markus Murtinger
The contemporary geopolitical environment and strategic uncertainty shaped by asymmetric and hybrid threats urge the future development of hands-on training in realistic environments. Training in immersive, virtual environments is a promising approach. Immersive training can support training for contexts that are otherwise hard to access, dangerous, or have high costs. This paper discusses the challenges for virtual reality training in the CBRN (chemical, biological, radioactive, nuclear) domain. Based on initial considerations and a literature review, we conducted a survey and three workshops to gather requirements for CBRN training in virtual environments. We structured the gathered insights into four overarching themes—the future of CBRN training, ethical and safety requirements, evaluation and feedback, and tangible objects and tools. We provide insights on these four themes and discuss recommendations.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7090087
Authors: Natacha Fernandes Joana Casteleiro-Pitrez
Augmented Reality (AR) is increasingly present in several fields, including the museological space, where the challenges of presenting objects interactively and attractively are constant, especially with the sociocultural changes of recent decades. Although there are numerous studies on AR in museums, the perspective of museum professionals on the technology still needs to be explored. Thus, in this study, we use a qualitative design and conduct in-depth interviews with professionals from 10 Portuguese museums involved in creating or applying AR within these environments. Applying the grounded theory, the researchers propose a framework to understand Portuguese museum professionals’ practices, perceptions, and experiences with AR in museum environments. The findings allow the creation of a theoretical framework divided into four levels, namely the perceptions of museum professionals on the role and use of AR, the understanding of departments, museum teams, and digital strategies, the perceived challenges, limitations, and advantages in the use of augmented reality technologies, and the future perspectives of AR in museums. The theory resulting from this study may also contribute suggestions for the design and implementation of AR in museums, which both museum professionals and designers can use.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7090086
Authors: Christos Batsaras Stelios Xinogalos
This article presents a comparative analysis of four low or no-code location-based game (LBG) authoring tools, namely Taleblazer, Aris, Actionbound, and Locatify. Each tool is examined in detail, with an emphasis on the functions and capabilities it provides for the development of LBGs. The article builds on the history and purpose of LBGs, their characteristics, as well as basic concepts and previous applications, placing emphasis both on the technological and pedagogical dimensions of these games. The evaluation of the tools is based on certain criteria, or metrics, recorded in the literature and empirical data collected through the development of prototype games for each tool. The tools are comparatively analyzed in terms of the LBG’s constituent features they incorporate, the fundamental and additional functionality provided to the developer, as well as the existence or absence of features that captivate players in the game experience. Moreover, feedback is provided based on the practical use of the platforms for developing LBGs in order to support prospective developers in making an informed choice of an LBG platform for implementing a specific game. The games were created by taking advantage of as many features of the tools as possible in order to have a more fair and complete evaluation. This study aims to highlight the affordances and limitations of the investigated low or no-code LBG authoring tools, enabling anyone interested in developing an LBG to choose the most appropriate tool taking into account their needs and technological background or designing their own LBG authoring tools.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7090085
Authors: Vid Podpečan
This retrospective study presents and summarizes our long-term efforts in the popularization of robotics, engineering, and artificial intelligence (STEM) using the NAO humanoid robot. By a conservative estimate, over a span of 8 years, we engaged at least a couple of thousand participants: approximately 70% were preschool children, 15% were elementary school students, and 15% were teenagers and adults. We describe several robot applications that were developed specifically for this task and assess their qualitative performance outside a controlled research setting, catering to various demographics, including those with special needs (ASD, ADHD). Five groups of applications are presented: (1) motor development activities and games, (2) children’s games, (3) theatrical performances, (4) artificial intelligence applications, and (5) data harvesting applications. Different cases of human–robot interactions are considered and evaluated according to our experience, and we discuss their weak points and potential improvements. We examine the response of the audience when confronted with a humanoid robot featuring intelligent behavior, such as conversational intelligence and emotion recognition. We consider the importance of the robot’s physical appearance, the emotional dynamics of human–robot engagement across age groups, the relevance of non-verbal cues, and analyze drawings crafted by preschool children both before and after their interaction with the NAO robot.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7080084
Authors: Megan L. Hutchcraft Robert C. Wallon Shanna M. Fealy Donovan Jones Roberto Galvez
Integration of technology within problem-based learning curricula is expanding; however, information regarding student experiences and attitudes about the integration of such technologies is limited. This study aimed to evaluate pre-clinical medical student perceptions and use patterns of the “Road to Birth” (RtB) software, a novel program designed to support human maternal anatomy and physiology education. Second-year medical students at a large midwestern American university participated in a prospective, mixed-methods study. The RtB software is available as a mobile smartphone/tablet application and in immersive virtual reality. The program was integrated into problem-based learning activities across a three-week obstetrics teaching period. Student visuospatial ability, weekly program usage, weekly user satisfaction, and end-of-course focus group interview data were obtained. Survey data were analyzed and summarized using descriptive statistics. Focus group interview data were analyzed using inductive thematic analysis. Of the eligible students, 66% (19/29) consented to participate in the study with 4 students contributing to the focus group interview. Students reported incremental knowledge increases on weekly surveys (69.2% week one, 71.4% week two, and 78.6% week three). Qualitative results indicated the RtB software was perceived as a useful educational resource; however, its interactive nature could have been further optimized. Students reported increased use of portable devices over time and preferred convenient options when using technology incorporated into the curriculum. This study identifies opportunities to better integrate technology into problem-based learning practices in medical education. Further empirical research is warranted with larger and more diverse student samples.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7080083
Authors: Víctor Martínez-Sánchez Iván Villalón-Turrubiates Francisco Cervantes-Álvarez Carlos Hernández-Mejía
This research explores a novel Mexican Sign Language (MSL) lexicon video dataset containing the dynamic gestures most frequently used in MSL. Each gesture consists of a set of different versions of videos under uncontrolled conditions. The MX-ITESO-100 dataset is composed of a lexicon of 100 gestures and 5000 videos from three participants with different grammatical elements. Additionally, the dataset is evaluated in a two-step neural network model as having an accuracy greater than 99% and thus serves as a benchmark for future training of machine learning models in computer vision systems. Finally, this research provides an inclusive environment within society and organizations, in particular for people with hearing impairments.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7080082
Authors: Robert Dongas Kazjon Grace Samuel Gillespie Marius Hoggenmueller Martin Tomitsch Stewart Worrall
In this study, we propose the use of virtual urban field studies (VUFS) through context-based interface prototypes for evaluating the interaction design of auditory interfaces. Virtual field tests use mixed-reality technologies to combine the fidelity of real-world testing with the affordability and speed of testing in the lab. In this paper, we apply this concept to rapidly test sound designs for autonomous vehicle (AV)–pedestrian interaction with a high degree of realism and fidelity. We also propose the use of psychometrically validated measures of presence in validating the verisimilitude of VUFS. Using mixed qualitative and quantitative methods, we analysed users’ perceptions of presence in our VUFS prototype and the relationship to our prototype’s effectiveness. We also examined the use of higher-order ambisonic spatialised audio and its impact on presence. Our results provide insights into how VUFS can be designed to facilitate presence as well as design guidelines for how this can be leveraged.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7080081
Authors: Fiona French David Levi Csaba Maczo Aiste Simonaityte Stefanos Triantafyllidis Gergo Varda
Educators and students have shown significant interest in the potential for generative artificial intelligence (AI) technologies to support student learning outcomes, for example, by offering personalized experiences, 24 h conversational assistance, text editing and help with problem-solving. We review contemporary perspectives on the value of AI as a tool in an educational context and describe our recent research with undergraduate students, discussing why and how we integrated OpenAI tools ChatGPT and Dall-E into the curriculum during the 2022–2023 academic year. A small cohort of games programming students in the School of Computing and Digital Media at London Metropolitan University was given a research and development assignment that explicitly required them to engage with OpenAI. They were tasked with evaluating OpenAI tools in the context of game development, demonstrating a working solution and reporting on their findings. We present five case studies that showcase some of the outputs from the students and we discuss their work. This mode of assessment was both productive and popular, mapping to students’ interests and helping to refine their skills in programming, problem-solving, critical reflection and exploratory design.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7080080
Authors: Yiannis Georgiou Andreas Ch. Hadjichambis Demetra Paraskeva-Hadjichambi Anastasia Adamou
As the global environmental crisis intensifies, there has been a significant interest in behavior change games (BCGs), as a viable venue to empower players’ pro-environmentalism. This pro-environmental empowerment is well-aligned with the notion of environmental citizenship (EC), which aims at transforming citizens into “environmental agents of change”, seeking to achieve more sustainable lifestyles. Despite these arguments, studies in this area are thinly spread and fragmented across various research domains. This article is grounded on a systematic review of empirical articles on BCGs for EC covering a time span of fifteen years and published in peer-reviewed journals and conference proceedings, in order to provide an understanding of the scope of empirical research in the field. In total, 44 articles were reviewed to shed light on their methodological underpinnings, the gaming elements and the persuasive strategies of the deployed BCGs, the EC actions facilitated by the BCGs, and the impact of BCGs on players’ EC competences. Our findings indicate that while BCGs seem to promote pro-environmental knowledge and attitudes, such an assertion is not fully warranted for pro-environmental behaviors. We reflect on our findings and provide future research directions to push forward the field of BCGs for EC.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7080079
Authors: Zlatin Zlatev Julieta Ilieva Daniela Orozova Galya Shivacheva Nadezhda Angelova
This paper presents a device that converts sound wave frequencies into colors to assist people with hearing problems in solving accessibility and communication problems in the hearing-impaired community. The device uses a precise mathematical apparatus and carefully selected hardware to achieve accurate conversion of sound to color, supported by specialized automatic processing software suitable for standardization. Experimental evaluation shows excellent performance for frequencies below 1000 Hz, although limitations are encountered at higher frequencies, requiring further investigation into advanced noise filtering and hardware optimization. The device shows promise for various applications, including education, art, and therapy. The study acknowledges its limitations and suggests future research to generalize the models for converting sound frequencies to color and improving usability for a broader range of hearing impairments. Feedback from the hearing-impaired community will play a critical role in further developing the device for practical use. Overall, this innovative device for converting sound to color represents a significant step toward improving accessibility and communication for people with hearing challenges. Continued research offers the potential to overcome challenges and extend the benefits of the device to a variety of areas, ultimately improving the quality of life for people with hearing impairments.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7080078
Authors: José Rouillard Jean-Marc Vannobel
For greater efficiency, human–machine and human–robot interactions must be designed with the idea of multimodality in mind. To allow the use of several interaction modalities, such as the use of voice, touch, gaze tracking, on several different devices (computer, smartphone, tablets, etc.) and to integrate possible connected objects, it is necessary to have an effective and secure means of communication between the different parts of the system. This is even more important with the use of a collaborative robot (cobot) sharing the same space and very close to the human during their tasks. This study present research work in the field of multimodal interaction for a cobot using the MQTT protocol, in virtual (Webots) and real worlds (ESP microcontrollers, Arduino, IOT2040). We show how MQTT can be used efficiently, with a common publish/subscribe mechanism for several entities of the system, in order to interact with connected objects (like LEDs and conveyor belts), robotic arms (like the Ned Niryo), or mobile robots. We compare the use of MQTT with that of the Firebase Realtime Database used in several of our previous research works. We show how a “pick–wait–choose–and place” task can be carried out jointly by a cobot and a human, and what this implies in terms of communication and ergonomic rules, via health or industrial concerns (people with disabilities, and teleoperation).
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7080077
Authors: Tahani Jaser Alahmadi Atta Ur Rahman Hend Khalid Alkahtani Hisham Kholidy
Vision impairment affects an individual’s quality of life, posing challenges for visually impaired people (VIPs) in various aspects such as object recognition and daily tasks. Previous research has focused on developing visual navigation systems to assist VIPs, but there is a need for further improvements in accuracy, speed, and inclusion of a wider range of object categories that may obstruct VIPs’ daily lives. This study presents a modified version of YOLOv4_Resnet101 as backbone networks trained on multiple object classes to assist VIPs in navigating their surroundings. In comparison to the Darknet, with a backbone utilized in YOLOv4, the ResNet-101 backbone in YOLOv4_Resnet101 offers a deeper and more powerful feature extraction network. The ResNet-101’s greater capacity enables better representation of complex visual patterns, which increases the accuracy of object detection. The proposed model is validated using the Microsoft Common Objects in Context (MS COCO) dataset. Image pre-processing techniques are employed to enhance the training process, and manual annotation ensures accurate labeling of all images. The module incorporates text-to-speech conversion, providing VIPs with auditory information to assist in obstacle recognition. The model achieves an accuracy of 96.34% on the test images obtained from the dataset after 4000 iterations of training, with a loss error rate of 0.073%.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7080076
Authors: Ioannis Paspatis Aggeliki Tsohou Spyros Kokolakis
What influences Information Communications and Technology (ICT) users’ privacy behavior? Several studies have shown that users state to care about their personal data. Contrary to that though, they perform unsafe privacy actions, such as ignoring to configure privacy settings. In this research, we present the results of an in-depth literature review on the factors affecting privacy behavior. We seek to investigate the underlying factors that influence individuals’ privacy-conscious behavior in the digital domain, as well as effective interventions to promote such behavior. Privacy decisions regarding the disclosure of personal information may have negative consequences on individuals’ lives, such as becoming a victim of identity theft, impersonation, etc. Moreover, third parties may exploit this information for their own benefit, such as targeted advertising practices. By identifying the factors that may affect SNS users’ privacy awareness, we can assist in creating methods for effective privacy protection and/or user-centered design. Examining the results of several research studies, we found evidence that privacy behavior is affected by a variety of factors, including individual ones (e.g., demographics) and contextual ones (e.g., financial exchanges). We synthesize a framework that aggregates the scattered factors that have been found in the literature to affect privacy behavior. Our framework can be beneficial to academics and practitioners in the private and public sectors. For example, academics can utilize our findings to create specialized information privacy courses and theoretical or laboratory modules.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7080075
Authors: Shivangi Surati Himani Trivedi Bela Shrimali Chintan Bhatt Carlos M. Travieso-González
With the widespread of Monkeypox and increase in the weekly reported number of cases, it is observed that this outbreak continues to put the human beings in risk. The early detection and reporting of this disease will help monitoring and controlling the spread of it and hence, supporting international coordination for the same. For this purpose, the aim of this paper is to classify three diseases viz. Monkeypox, Chikenpox and Measles based on provided image dataset using trained standalone DL models (InceptionV3, EfficientNet, VGG16) and Squeeze and Excitation Network (SENet) Attention model. The first step to implement this approach is to search, collect and aggregate (if require) verified existing dataset(s). To the best of our knowledge, this is the first paper which has proposed the use of SENet based attention models in the classification task of Monkeypox and also targets to aggregate two different datasets from distinct sources in order to improve the performance parameters. The unexplored SENet attention architecture is incorporated with the trunk branch of InceptionV3 (SENet+InceptionV3), EfficientNet (SENet+EfficientNet) and VGG16 (SENet+VGG16) and these architectures improve the accuracy of the Monkeypox classification task significantly. Comprehensive experiments on three datasets depict that the proposed work achieves considerably high results with regard to accuracy, precision, recall and F1-score and hence, improving the overall performance of classification. Thus, the proposed research work is advantageous in enhanced diagnosis and classification of Monkeypox that can be utilized further by healthcare experts and researchers to confront its outspread.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7070074
Authors: Reham Salhab Wajeeh Daher
As technology has explosively and globally revolutionized the teaching and learning processes at educational institutions, enormous and innovative technological developments, along with their tools and applications, have recently invaded the education system. Using mobile learning (m-learning) employs wireless technologies for thinking, communicating, learning, and sharing to disseminate and exchange knowledge. Consequently, assessing the learning attitudes of students toward mobile learning is crucial, as learning attitudes impact their motivation, performance, and beliefs about mobile learning. However, mobile learning seems under-researched and may require additional efforts from researchers, especially in the context of the Middle East. Hence, this study’s contribution is enhancing our knowledge about students’ attitudes towards mobile-based learning. Therefore, the study goal was to investigate m-learning’s effect on the learning attitudes among technology education students. An explanatory sequential mixed approach was utilized to examine the attitudes of 50 students who took an educational technology class. A quasi-experiment was conducted and a phenomenological approach was adopted. Data from the experimental group and the control group were gathered. Focus group discussions with three groups and 25 semi-structured interviews were performed with students who experienced m-learning in their course. ANCOVA was conducted and revealed the impact of m-learning on the attitudes and their components. An inductive and deductive content analysis was conducted. Eleven subthemes stemmed out of three main themes. These subthemes included: personalized learning, visualization of learning motivation, less learning frustration, enhancing participation, learning on familiar devices, and social interaction, which emerged from the data. The researchers recommended that higher education institutions adhere to a set of guiding principles when creating m-learning policies. Additionally, they should customize the m-learning environment with higher levels of interactivity to meet students’ needs and learning styles to improve their attitudes towards m-learning.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7070073
Authors: Champika Ranasinghe Auriol Degbelo
Data physicalisations, or physical visualisations, represent data physically, using variable properties of physical media. As an emerging area, Data physicalisation research needs conceptual foundations to support thinking about and designing new physical representations of data and evaluating them. Yet, it remains unclear at the moment (i) what encoding variables are at the designer’s disposal during the creation of physicalisations, (ii) what evaluation criteria could be useful, and (iii) what methods can be used to evaluate physicalisations. This article addresses these three questions through a narrative review and a systematic review. The narrative review draws on the literature from Information Visualisation, HCI and Cartography to provide a holistic view of encoding variables for data. The systematic review looks closely into the evaluation criteria and methods that can be used to evaluate data physicalisations. Both reviews offer a conceptual framework for researchers and designers interested in designing and evaluating data physicalisations. The framework can be used as a common vocabulary to describe physicalisations and to identify design opportunities. We also proposed a seven-stage model for designing and evaluating physical data representations. The model can be used to guide the design of physicalisations and ideate along the stages identified. The evaluation criteria and methods extracted during the work can inform the assessment of existing and future data physicalisation artefacts.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7070072
Authors: Alessandra Miano Marco Borsotti
The paper presents an existing scenario related to the advanced integration of digital technologies in the field of house museums, based on the critical literature and applied experimentation. House museums are a particular type of heritage site, in which is highlighted the tension between the evocative capacity of the spaces and the requirements for preservation. In this dimension, the use of a seamless approach amplifies the atmospheric component of the space, superimposing, through hybrid digital technologies, an interactive, context-driven layer in an open dialogue between digital and physical. The methodology moves on the one hand from the literature review, framing the macro themes of research, and on the other from the overview of case studies, selected on the basis of the experiential value of the space. The analysis of the selected cases followed as criteria: the formal dimension of the technology; the narrative plot, as storytelling of socio-cultural atmosphere or identification within the intimate story; and the involvement of visitors as individual immersion or collective rituality. The paper aimed at outlining a developmental panorama in which the integration of hybrid technologies points to a new seamless awareness within application scenarios as continuous and work-in-progress challenges.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7070071
Authors: Svenja Y. Schött Patricia Capsi-Morales Steeven Villa Andreas Butz Cristina Piazza
Recent technological advances have enabled the development of sophisticated prosthetic hands, which can help their users to compensate lost motor functions. While research and development has mostly addressed the functional requirements and needs of users of these prostheses, their broader societal perception (e.g., by external observers not affected by limb loss themselves) has not yet been thoroughly explored. To fill this gap, we investigated how the physical design of artificial hands influences the perception by external observers. First, we conducted an online study (n = 42) to explore the emotional response of observers toward three different types of artificial hands. Then, we conducted a lab study (n = 14) to examine the influence of design factors and depth of interaction on perceived trust and usability. Our findings indicate that some design factors directly impact the trust individuals place in the system’s capabilities. Furthermore, engaging in deeper physical interactions leads to a more profound understanding of the underlying technology. Thus, our study shows the crucial role of the design features and interaction in shaping the emotions around, trust in, and perceived usability of artificial hands. These factors ultimately impact the overall perception of prosthetic systems and, hence, the acceptance of these technologies in society.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7070070
Authors: Katharina Krösl Marina Lima Medeiros Marlene Huber Steven Feiner Carmine Elvezio
To create a truly accessible and inclusive society, we need to take the more than 2.2 billion people with vision impairments worldwide into account when we design our cities, buildings, and everyday objects. This requires sympathy and empathy, as well as a certain level of understanding of the impact of vision impairments on perception. In this study, we explore the potential of an extended version of our vision-impairment simulation system XREye to increase sympathy and empathy and evaluate its educational value in an expert study with 56 educators and education students. We include data from a previous study in related work on sympathy and empathy as a baseline for comparison with our data. Our results show increased sympathy and empathy after experiencing XREye and positive feedback regarding its educational value. Hence, we believe that vision-impairment simulations, such as XREye, have merit to be used for educational purposes in order to increase awareness for the challenges people with vision impairments face in their everyday lives.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7070069
Authors: Duy Duc Pham Trong Hieu Luu Le Trung Chanh Tran Hoai Trang Nguyen Thi Hoang-Long Cao
Measuring learning curves allows for the inspection of the rate of learning and competency threshold for each individual, training lesson, or training method. In this work, we investigated learning curves in acupuncture needle manipulation training with continuous performance measurement using a vision-based needle training system. We tracked the needle insertion depth of 10 students to investigate their learning curves. The results show that the group-level learning curve was fitted with the Thurstone curve, indicating that students were able to improve their needle insertion skills after repeated practice. Additionally, the analysis of individual learning curves revealed valuable insights into the learning experiences of each participant, highlighting the importance of considering individual differences in learning styles and abilities when designing training programs.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7070068
Authors: Anika van der Klis Frans Adriaans Mengru Han René Kager
There is a large interest in the annotation of speech addressed to infants. Infant-directed speech (IDS) has acoustic properties that might pose a challenge to automatic speech recognition (ASR) tools developed for adult-directed speech (ADS). While ASR tools could potentially speed up the annotation process, their effectiveness on this speech register is currently unknown. In this study, we assessed to what extent open-source ASR tools can successfully transcribe IDS. We used speech data from 21 Dutch mothers reading picture books containing target words to their 18- and 24-month-old children (IDS) and the experimenter (ADS). In Experiment 1, we examined how the ASR tool Kaldi-NL performs at annotating target words in IDS vs. ADS. We found that Kaldi-NL only found 55.8% of target words in IDS, while it annotated 66.8% correctly in ADS. In Experiment 2, we aimed to assess the difficulties in annotating IDS more broadly by transcribing all IDS utterances manually and comparing the word error rates (WERs) of two different ASR systems: Kaldi-NL and WhisperX. We found that WhisperX performs significantly better than Kaldi-NL. While there is much room for improvement, the results show that automatic transcriptions provide a promising starting point for researchers who have to transcribe a large amount of speech directed at infants.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7070067
Authors: Ruzaliev Rakhmiddin KangYoon Lee
Accurate and timely diagnosis is a pillar of effective healthcare. However, the challenge lies in gathering extensive training data while maintaining patient privacy. This study introduces a novel approach using federated learning (FL) and a cross-device multimodal model for clinical event classification based on vital signs data. Our architecture employs FL to train several machine learning models including random forest, AdaBoost, and SGD ensemble models on vital signs data. The data were sourced from a diverse clientele at a Boston hospital (MIMIC-IV dataset). The FL structure trains directly on each client’s device, ensuring no transfer of sensitive data and preserving patient privacy. The study demonstrates that FL offers a powerful tool for privacy-preserving clinical event classification, with our approach achieving an impressive accuracy of 98.9%. These findings highlight the significant potential of FL and cross-device ensemble technology in healthcare applications, especially in the context of handling large volumes of sensitive patient data.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7070066
Authors: Mark Reybrouck
This paper takes a dynamic interactive stance to music listening. It revolves around the focal concept of entrainment as an operational tool for the description of fine-grained dynamics between the music as an entraining stimulus and the listener as an entrained subject. Listeners, in this view, can be “entrained” by the sounds at several levels of processing, dependent on the degree of attunement and alignment of their attention. The concept of entrainment, however, is somewhat ill-defined, with distinct conceptual labels, such as external vs. mutual, symmetrical vs. asymmetrical, metrical vs. non-metrical, within-persons and between-person, and physical vs. cognitive entrainment. The boundaries between entrainment, resonance, and synchronization are also not always very clear. There is, as such, a need for a broadened approach to entrainment, taking as a starting point the concept of oscillators that interact with each other in a continuous and ongoing way, and relying on the theoretical framework of interaction dynamics and the concept of adaptation. Entrainment, in this broadened view, is seen as an adaptive process that accommodates to the music under the influence of both the attentional direction of the listener and the configurations of the sounding stimuli.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7070065
Authors: Detlef Gerhard Matthias Neges Jan Luca Siewert Mario Wolf
While Industrial Augmented Reality (IAR) has many applications across the whole product lifecycle, most IAR applications today are custom-built for specific use-cases in practice. This contribution builds upon a scoping literature review of IAR data representations to present a modern, modular IAR architecture. The individual modules of the presented approach are either responsible for user interface and user interaction or for data processing. They are use-case neutral and independent of each other, while communicating through a strictly separated application layer. To demonstrate the architecture, this contribution presents an assembly process that is supported once with a pick-to-light system and once using in situ projections. Both are implemented on top of the novel architecture, allowing most of the work on the individual models to be reused. This IAR architecture, based on clearly separated modules with defined interfaces, particularly allows small companies with limited personnel resources to adapt IAR for their specific use-cases more easily than developing single-use applications from scratch.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7070064
Authors: Mohamed Zaifri Hamza Khalloufi Fatima Zahra Kaghat Ahmed Azough Khalid Alaoui Zidani
Augmented reality has emerged as a transformative technology, with the potential to revolutionize the tourism industry. Nonetheless, there is a scarcity of studies tracing the progression of AR and its application in tourism, from early exploration to recent advancements. This study aims to provide a comprehensive overview of the evolution, contexts, and design elements of AR in tourism over the period (2002–2022), offering insights for further progress in this domain. Employing a dual-method approach, a bibliometric analysis was conducted on 861 articles collected from the Scopus and Web of Science databases, to investigate the evolution of AR research over time and across countries, and to identify the main contexts of the utilization of AR in tourism. In the second part of our study, a systematic content analysis was conducted, focusing on a subset of 57 selected studies that specifically employed AR systems in various tourism situations. Through this analysis, the most commonly utilized AR design components, such as tracking systems, AR devices, tourism settings, and virtual content were summarized. Furthermore, we explored how these components were integrated to enhance the overall tourism experience. The findings reveal a growing trend in research production, led by Europe and Asia. Key contexts of AR applications in tourism encompass cultural heritage, mobile AR, and smart tourism, with emerging topics such as artificial intelligence (AI), big data, and COVID-19. Frequently used AR design components comprise mobile devices, marker-less tracking systems, outdoor environments, and visual overlays. Future research could involve optimizing AR experiences for users with disabilities, supporting multicultural experiences, integrating AI with big data, fostering sustainability, and remote virtual tourism. This study contributes to the ongoing discourse on the role of AR in shaping the future of tourism in the post COVID-19 era, by providing valuable insights for researchers, practitioners, and policymakers in the tourism industry.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7070063
Authors: Vera Remizova Antti Sand I. Scott MacKenzie Oleg Špakov Katariina Nyyssönen Ismo Rakkolainen Anneli Kylliäinen Veikko Surakka Yulia Gizatdinova
Projected walk-through fogscreens have been created, but there is little research on the evaluation of the interaction performance with fogscreens. The present study investigated mid-air hand gestures for interaction with a large fogscreen. Participants (N = 20) selected objects from a fogscreen using tapping and dwell-based gestural techniques, with and without vibrotactile/haptic feedback. In terms of Fitts’ law, the throughput was about 1.4 bps to 2.6 bps, suggesting that gestural interaction with a large fogscreen is a suitable and effective input method. Our results also suggest that tapping without haptic feedback has good performance and potential for interaction with a fogscreen, and that tactile feedback is not necessary for effective mid-air interaction. These findings have implications for the design of gestural interfaces suitable for interaction with fogscreens.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7060062
Authors: Doreen Schwarze Frederik Diederichs Lukas Weiser Harald Widlroither Rolf Verhoeven Matthias Rötting
Higher levels of automated driving may offer the possibility to sleep in the driver’s seat in the car, and it is foreseeable that drivers will voluntarily or involuntarily fall asleep when they do not need to drive. Post-sleep performance impairments due to sleep inertia, a brief period of impaired cognitive performance after waking up, is a potential safety issue when drivers need to take over and drive manually. The present study assessed whether sleep inertia has an effect on driving and cognitive performance after different sleep durations. A driving simulator study with n = 13 participants was conducted. Driving and cognitive performance were analyzed after waking up from a 10–20 min sleep, a 30–60 min sleep, and after resting without sleep. The study’s results indicate that a short sleep duration does not reliably prevent sleep inertia. After the 10–20 min sleep, cognitive performance upon waking up was decreased, but the sleep inertia impairment faded within 15 min. Although the driving parameters showed no significant difference between the conditions, participants subjectively felt more tired after both sleep durations compared to resting. The small sample size of 13 participants, tested in a within-design, may have prevented medium and small effects from becoming significant. In our study, take-over was offered without time pressure, and take-over times ranged from 3.15 min to 4.09 min after the alarm bell, with a mean value of 3.56 min in both sleeping conditions. The results suggest that daytime naps without previous sleep deprivation result in mild and short-term impairments. Further research is recommended to understand the severity of impairments caused by different intensities of sleep inertia.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7060061
Authors: Sofia Pescarin Vanessa Bonanno Alessandra Marasco
Presently, social cohesion (SC) is a priority at different levels. Cultural heritage is an ideal context to promote SC through interactive digital technologies designed to engage groups of visitors. The purpose of the present study is to identify how to design digital heritage applications for SC and how to measure it. The results are based on the design of a cultural probe kit used to identify the design elements on top of which a collaborative and hybrid prototype, the Brancacci POV, was developed. Here, we analysed the results of this prototype, which included 107 visitors with respective groups of 5 participants and guided by an expert. From this analysis, the possibility of strengthening SC when collaborative tasks are included emerged. Additionally, it appeared to be possible to shorten the distance between citizens and cultural institutions if “mediated dialogue” approaches were adopted and if focus, motivation, trust and “in-group” perception of inclusion emerge when digital heritage experiences were set in intimate and quiet environments.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7060060
Authors: Emma Buchanan Giuseppe Loporcaro Stephan Lukosch
This article reports on a study that assessed whether Virtual Reality (VR) can be used to display Building Information Modelling (BIM) metadata alongside spatial data in a virtual environment, and by doing so determine if it increases the effectiveness of the design review by improving participants’ understanding of the design. Previous research has illustrated the potential for VR to enhance design reviews, especially the ability to convey spatial information, but there has been limited research into how VR can convey additional BIM metadata. A user study with 17 healthcare professionals assessed participants’ performances and preferences for completing design reviews in VR or using a traditional design review system of PDF drawings and a 3D model. The VR condition had a higher task completion rate, a higher SUS score and generally faster completion times. VR increases the effectiveness of a design review conducted by healthcare professionals.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7060059
Authors: Hiroyuki Mitsuhara Chie Tanimura Junko Nemoto Masami Shishibori
Participation in evacuation training can aid survival in the event of an unpredictable disaster, such as an earthquake. However, conventional evacuation training is not well designed for provoking critical thinking in participants regarding the processes involved in a successful evacuation. To realize thought-provoking evacuation training, we developed a location-based game that presents digital materials that express disaster situations corresponding to locations or times preset in a scenario and providing scenario-based multi-ending as the game element. The developed game motivates participants to take decisions by providing high situational and audiovisual realism. In addition, the game encourages the participants to think objectively about the evacuation process by working together with a reflection-support system. We practiced thought-provoking evacuation training with fifth-grade students, focusing on tsunami evacuation and lifesaving-related moral dilemmas. In this practice, we observed that the participants took decisions as if they were dealing with actual disaster situations and objectively thought about the evacuation process by reflecting on their decisions. Meanwhile, we found that lifesaving-related moral dilemmas are difficult to address in evacuation training.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7060058
Authors: Rute Silva Rui Jesus Pedro Jorge
Tourism on the island of Santa Maria, Azores, has been increasing due to its characteristics in terms of biodiversity and geodiversity. This island has several hiking trails; the available information can be consulted in pamphlets and physical placards, whose maintenance and updating is difficult and expensive. Thus, the need to improve the visitors’ experience arises, in this case, by using the technological means currently available to everyone: a smartphone. This paper describes the development and evaluation of the user experience of a mobile application for guiding visitors on said hiking trails, as well as the design principles and main issues observed during this process. The application is based on an augmented reality interaction model providing visitors with an interactive and recreational experience through Augmented Reality in outdoor environments (without additional marks in the physical space and using georeferenced information), helping in navigation during the route and providing updated information with easy maintenance. For the design and evaluation of the application, two studies were carried out with users on-site (Santa Maria, Azores). The first had 77 participants, to analyze users and define the application’s characteristics, and the second had 10 participants to evaluate the user experience. The feedback from participants was obtained through questionnaires. In these questionnaires, an average SUS (System Usability Scale) score of 83 (excellent) and positive results in the UEQ (User Experience Questionnaire) were obtained.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7060057
Authors: Katherine Hopman Deborah Richards Melissa M. Norberg
There is growing awareness that effective emotion regulation is critical for health, adjustment and wellbeing. Emerging evidence suggests that interventions that promote flexible emotion regulation may have the potential to reduce the incidence and prevalence of mental health problems in specific at-risk populations. The challenge is how best to engage with at risk populations, who may not be actively seeking assistance, to deliver this early intervention approach. One possible solution is via digital technology and development, which has rapidly accelerated in this space. Such rapid growth has, however, occurred at the expense of developing a deep understanding of key elements of successful program design and specific mechanisms that influence health behavior change. This paper presents a detailed description of the design, development and evaluation of an emotion regulation intervention conversational agent (ERICA) who acts as a digital coach. ERICA uses interactive conversation to encourage self-reflection and to support and empower users to learn a range of cognitive emotion regulation strategies including Refocusing, Reappraisal, Planning and Putting into Perspective. A pilot evaluation of ERICA was conducted with 138 university students and confirmed that ERICA provided a feasible and highly usable method for delivering an emotion regulation intervention. The results also indicated that ERICA was able to develop a therapeutic relationship with participants and increase their intent to use a range of cognitive emotion regulation strategies. These findings suggest that ERICA holds potential to be an effective approach for delivering an early intervention to support mental health and wellbeing. ERICA’s dialogue, embedded with interactivity, therapeutic alliance and empathy cues, provide the basis for the development of other psychoeducation interventions.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7060056
Authors: Daniel Hepperle Matthias Wölfel
In the broader field of human behavior studies, there are several trade-offs for on-site experiments. To be tied to a specific location can limit both the availability and diversity of participants. However, current and future technological advances make it possible to replicate real-world scenarios in a virtual environment up to a certain level of detail. How these differences add up and affect the cross-media validity of findings remains a topic of debate. How a virtual world is accessed, through a computer screen or a head-mounted display, may have a significant impact. Not surprisingly, the literature has presented various comparisons. However, while previous research has compared the different devices for a specific research question, a systematic review is lacking. To fill this gap, we conducted this review. We identified 1083 articles in accordance with the PRISMA guidelines. Following screening, 56 articles remained and were compared for a qualitative synthesis to provide the reader a summary of current research on the differences between head-mounted displays (HMDs), computer screens, and the real world. Overall, the data show that virtual worlds presented in an HMD are more similar to real-world situations than to computer screens. This supports the thesis that HMDs are more suitable than computer screens for conducting experiments in the field of human behavioral studies.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7060055
Authors: Jennifer Challenor David White David Murphy
With the rapid expansion of technology and hardware availability within the field of Augmented Reality, building and deploying Augmented Reality learning environments has become more logistically viable than ever before. In this paper, we focus on the development of a new mobile learning experience for a museum by combining multiple technologies to provide additional Human–computer interaction possibilities. This is both to reduce barriers to entry for end-users as well as provide natural interaction methods. Using our method, we implemented a new approach to gesture-based interactions for Augmented Reality interactions by combining two devices, a Leap Motion and a Microsoft HoloLens (1st Generation), via an intermediary device with the use of local-area networking. This was carried out with the intention of comparing this method against alternative forms of Augmented Reality to determine which implementation has the largest impact on adult learners’ ability to retain information. A control group has been used to establish data on memory retention without the use of Augmented Reality technology, along with three focus groups to explore the different methods and locations. Results found that adult learners retain the most overall information when being educated through a traditional lecture, with a statistically significant difference between the methods; however, the use of Augmented Reality resulted in a slower rate of knowledge decay between testing intervals. This contrasts with existing research as adult learners did not respond to the technology in the same way that child and teenage audiences previously have, which suggests that prior research may not be generalisable to all audiences.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7060054
Authors: Lisa Schadelbauer Stephan Schlögl Aleksander Groth
Throughout the last years, Intelligent Virtual Assistants (IVAs), such as Alexa and Siri, have increasingly gained in popularity. Yet, privacy advocates raise great concerns regarding the amount and type of data these systems collect and consequently process. Among many other things, it is technology trust which seems to be of high significance here, particularly when it comes to the adoption of IVAs, for they usually provide little transparency as to how they function and use personal and potentially sensitive data. While technology trust is influenced by many different socio-technical parameters, this article focuses on human personality and its connection to respective trust perceptions, which in turn may further impact the actual adoption of IVA products. To this end, we report on the results of an online survey (n=367). Findings show that on a scale from 0 to 100%, people trust IVAs 51.59% on average. Furthermore, the data point to a significant positive correlation between people’s propensity to trust in general technology and their trust in IVAs. Yet, they also show that those who exhibit a higher propensity to trust in technology tend to also have a higher affinity for technology interaction and are consequently more likely to adopt IVAs.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7050053
Authors: David Weinberg Healy Dwyer Sarah E. Fox Nikolas Martelaro
Sidewalk delivery robots are being deployed as a form of last-mile delivery. While many such robots have been deployed on college campuses, fewer have been piloted on public sidewalks. Furthermore, there have been few observational studies of robots and their interactions with pedestrians. To better understand how sidewalk robots might integrate into public spaces, the City of Pittsburgh, Pennsylvania conducted a pilot of sidewalk delivery robots to understand possible uses and the challenges that could arise in interacting with people in the city. Our team conducted ethnographic observations and intercept interviews to understand how residents perceived of and interacted with sidewalk delivery robots over the course of the public pilot. We found that people with limited knowledge about the robots crafted stories about their purpose and function. We observed the robots causing distractions and obstructions with different sidewalk users (including children and dogs), witnessed people helping immobilized robots, and learned about potential accessibility issues that the robots may pose. Based on our findings, we contribute a set of recommendations for future pilots, as well as questions to guide future design for robots in public spaces.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7050052
Authors: Vaishnavi N. Panjeti-Madan Prakash Ranganathan
Technology has become integral to children’s lives, impacting many aspects, from academic to socialization. Children of today’s generation are growing up with digital devices, such as mobile phones, iPads, computers, video games, and smart gadgets; therefore, screen time has become ubiquitous in children’s daily routines. This paper provides a review of screen time usage and its impact in children across multiple developmental domains: cognitive, language, physical, and socio-emotional domain of children under eight years of age. The cognitive domain considers factors such as attention span and memory; language domain examines vocabulary, speech, and language development; physical domain focuses on motor development, exercise, sleep, and diet; and social-emotional domain considers relationships, self-identity, and emotional behaviors/regulation. Our findings are mixed, as there are both benefits and drawbacks in technology use, but screen time in children requires controlled observation and monitoring for sustainable improved progress across developmental domains. Specific recommendations advise that children’s screen time per day should be limited to zero minutes (min) (0–2 years), <60 min (3–5-years), and 60 min (6–8 years).
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7050051
Authors: Dimitra Petousi Akrivi Katifori Maria Boile Lori Kougioumtzian Christos Lougiakis Maria Roussou Yannis Ioannidis
Over the past decades, 360-degree virtual tours have been used to provide the public access to accurate representations of cultural heritage sites and museums. The COVID-19 pandemic has contributed to a rise in the popularity of virtual tours as a means of engaging with locations remotely and has raised an interesting question: How could we use such experiences to bring the public closer to locations that are otherwise unreachable in real life or not considered to be tourist destinations? In this study, we examine the effectiveness of promoting engagement with a city through the virtual presentation of unknown and possibly also inaccessible points of interest through a 360-degree panoramic virtual tour. The evaluation of the experience with 31 users through an online questionnaire confirms its potential to spark curiosity, promote engagement, foster reflection, and motivate users to explore the location and its attractions at their leisure, thus enabling them to experience it from their personal point of view. The outcomes highlight the need for further research to explore this potential and identify best practices for virtual experience design.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7050050
Authors: Diana R. Sanchez Amanda Rueda Kentaro Kawasaki Saar Van Lysebetten Daniel Diaz
Organizations have maintained a commitment to using simulation technology for training purposes because it prepares employees for realistic work scenarios they may encounter and provides a relevant method for teaching hands-on skills. One challenge that simulation technology has faced is the persistent threat of obsolescence, where investment in an up-to-date solution can rapidly become irrelevant in a matter of months or years as technology progresses. This can be particularly challenging for organizations who seek out the best solutions to help develop and train employees while facing the constraints of limited resources and lengthy acquisition times for tools and equipment. Some industries and organizations may benefit from anticipating which technologies might best serve employees and stakeholders in the future. In this manuscript, we took a historical approach, looking at the history of training and the use of simulation-like experiences over time, which helps us identify historical themes in workplace training. Next, we carried out a systematic review of the recent training research using simulation technology to understand how these recent findings help us understand the identified historical themes. Lastly, we summarized the research literature on simulation technology used for training, and highlighted future directions and made recommendations for practitioners and researchers.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7050049
Authors: Ellenor J. Brown Kyoko Fujimoto Bennett Blumenkopf Andrea S. Kim Kimberly L. Kontson Heather L. Benz
Augmented reality (AR) head-mounted displays (HMDs) are an increasingly popular technology. For surgical applications, the use of AR HMDs to display medical images or models may reduce invasiveness and improve task performance by enhancing understanding of the underlying anatomy. This technology may be particularly beneficial in open surgeries and interventional procedures for which the use of endoscopes, microscopes, or other visualization tools is insufficient or infeasible. While the capabilities of AR HMDs are promising, their usability for surgery is not well-defined. This review identifies current trends in the literature, including device types, surgical specialties, and reporting of user demographics, and provides a description of usability assessments of AR HMDs for open surgeries and interventional procedures. Assessments applied to other extended reality technologies are included to identify additional usability assessments for consideration when assessing AR HMDs. The PubMed, Web of Science, and EMBASE databases were searched through September 2022 for relevant articles that described user studies. User assessments most often addressed task performance. However, objective measurements of cognitive, visual, and physical loads, known to affect task performance and the occurrence of adverse events, were limited. There was also incomplete reporting of user demographics. This review reveals knowledge and methodology gaps for usability of AR HMDs and demonstrates the potential impact of future usability research.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7050048
Authors: Diana R. Sanchez Amanda Rueda Hana R. Zimman Reese Haydon Daniel Diaz Kentaro Kawasaki
The virtuality of organizational teams have gained interest and popularity in recent years, and have become more prevalent amidst the COVID-19 pandemic. Organizational productivity and team relationship-building may suffer certain pitfalls in virtual communication and support without the understanding of the dynamics of short-term, project-based virtual teams. The manuscript aimed to expand what is currently known about short-term virtual team dynamics related to types of effective leadership behaviors. The present study employed a mixed method approach to understanding the dynamics of these teams at both the individual and team level. Small teams were formed and instructed to collaborate on a virtual survival task. Team-related outcomes were measured at the individual level, such as team coordination, team support, and team success. Additionally, distinct latent profiles of leadership behaviors were developed and analyzed at the team level. Team support, more so than team coordination, significantly predicted team success at the individual level, with instrumental support having the strongest effect. Distinct leadership behaviors emerged in teams and were classified through a latent profile analysis, but none of the profiles were significantly related to team performance scores. Demonstrating instrumental support in short-term virtual teams may improve team success. It is important to understand that distinct leadership behaviors exist and future research should explore the impact of these leadership behaviors on other team-related outcomes.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7050047
Authors: Daniel Page Robert W. Lindeman Stephan Lukosch
As Virtual Reality (VR) technology has improved in hardware, accessibility of development and availability of applications, its interest has increased. However, the problem of Cybersickness (CS) still remains, causing uncomfortable symptoms in users. Therefore, this research seeks to identify and understand new CS mitigation strategies that can contribute to developer guidelines. Three hypotheses for strategies were devised and tested in an experiment. This involved a physical travel interface for flying through a Virtual Environment (VE) as a Control (CT) condition. On top of this, three manipulation conditions referred to as Gaze-tracking Vignette (GV), First-person Perspective with members representation (FP) and Fans and Vibration (FV) were applied. The experiment was between subjects, with 37 participants randomly allocated across conditions. According to the Simulator Sickness Questionnaire (SSQ) scores, significant evidence was found that GV and FP made CS worse. Evidence was also found that FV did not have an effect on CS. However, from the physiological data recorded, an overall lowering of heart rate for FV indicated that it might have some effect on the experience, but cannot be strongly linked with CS. Additionally, comments from some participants identified that they experienced symptoms consistent with CS. Amongst these, dizziness was the most common, with a few having issues with the usability of the travel interface. Despite some CS symptoms, most participants reported little negative impact of CS on the overall experience and feelings of immersion.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7050046
Authors: Manjeet Singh Shaun Bangay Henry Grossek Atul Sajjanhar
The transition from kindergarten to primary school involves preparing students for a more structured classroom-based learning environment, which is typically different from the play-based model in kindergartens. Building on the Forest Room concept, which connects restless and disengaged students to nature as a calming medium, this case study describes the design of a combined storybook and augmented reality application to provide a literacy primer that integrates this concept. The design case study is presented relative to three frameworks that review the support for educational content, motivation and engagement mechanisms, and features of the AR application. This serves to validate the design process relative to these criteria and identifies opportunities for enhancement, including opportunities for meaningful interaction. The resulting application demonstrates appropriate design strategies to support its target age group and focus. It provides a stimulating and flexible learning activity that can be readily integrated into the classroom and that supports the kindergarten transition to appropriate classroom behaviour by encouraging active engagement and collaboration, blending aspects of both outdoor and classroom-based activities.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7050045
Authors: Peter A. Kara Aniko Simon
Light field displays offer glasses-free 3D visualization, which means that multiple individuals may observe the same content simultaneously from a virtually infinite number of perspectives without the need of viewing devices. The practical utilization of such visualization systems include various passive and active use cases. In the case of the latter, users often engage with the utilized system via human–computer interaction. Beyond conventional controls and interfaces, it is also possible to use advanced solutions such as motion tracking, which may seem seamless and highly convenient when paired with glasses-free 3D visualization. However, such solutions may not necessarily outperform conventional controls, and their true potentials may fundamentally depend on the use case in which they are deployed. In this paper, we provide a review on the 3D interaction of light field displays. Our work takes into consideration the different requirements posed by passive and active use cases, discusses the numerous challenges, limitations and potentials, and proposes research initiatives that could progress the investigated field of science.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7050044
Authors: Waleed Alsabhan Basil Dudin
This article discusses the design, development, and usability assessment of a mobile system for producing hydrological predictions and sending flood warnings in response to the desire for human-centered technology to better the management of flood occurrences. Our work acts as a bibliographic reference for understanding what others have attempted and found, as well as gives an integrated set of recommendations. Furthermore, our guidelines offer guidance to aid in the design of mobile GIS-based hydrological models for mobile devices. We concentrate on the full design of a human–computer interaction framework for an effective flood prediction and warning system. In addition, we analyze and address the current user needs and requirements for building a user interface for mobile real-time flood forecasting in a methodical manner. Although a functional prototype was created, the primary objective of this research was to comprehend the complexity of possible users’ demands and actual use situations in order to solve the problem of comparable systems being difficult to use. After consulting with possible consumers, application design standards were established and implemented in the initial prototype. Focusing on user demands and attitudes, special consideration was given to the usability of the mobile interface. To develop the application, a variety of assessment methods are added. The conclusion of the examination was that the system is efficient and effective.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7050043
Authors: Sharon Yavo-Ayalon Swapna Joshi Yuzhen (Adam) Zhang Ruixiang (Albert) Han Narges Mahyar Wendy Ju
Situated and shared experiences can motivate community members to plan shared action, promoting community engagement. We deployed and evaluated a communal extended-reality (CXR) bus tour that depicts the possible impacts of flooding and climate change. This paper describes the results of seven community engagement sessions with a total of N = 74 members of the Roosevelt Island community. We conducted pre- and post-bus tour focus groups to understand how the tour affected these community members’ awareness and motivation to take action. We found that the unique qualities of immersive, situated, and geo-located virtual reality (VR) on a bus made climate change feel real, brought the consequences of climate change closer to home, and highlighted existing community resources to address the issue. Our results showed that the CXR experience helped to simulate a physical emergency state, which empowered the community to translate feelings of hopelessness into creative and actionable ideas. Our finding exemplifies that geo-located VR on a bus can be a powerful tool to motivate innovations and collective action. Our work is a first-of-its-kind empirical contribution showing that CXR experiences can inspire action. It offers a proof-of-concept of a large-scale community engagement process featuring simulated communal experiences, leading to creative ideas for a bottom-up community resiliency plan.
]]>Multimodal Technologies and Interaction doi: 10.3390/mti7050042
Authors: Abdallah Namoun Mohammad Ali Humayun Oussama BenRhouma Burhan Rashid Hussein Ali Tufail Abdullah Alshanqiti Waqas Nawaz
Students with special needs should be empowered to use assistive technologies and services that suit their individual circumstances and environments to maximize their learning attainment. Fortunately, modern distributed computing paradigms, such as the Internet of Things (IoT), cloud computing, and mobile computing, provide ample opportunities to create and offer a multitude of digital assistive services and devices for people with disabilities. However, choosing the appropriate services from a pool of competing services while satisfying the unique requirements of disabled learners remains a challenging research endeavor. In this article, we propose an ensemble meta-learning model that ranks and selects the best IoT services while considering the diverse needs of disabled students within the educational context. We train and test our deep ensemble meta-learning model using two synthetically generated assistive services datasets. The first dataset incorporates 50,000 records representing the possible use of 12 learning activities, fulfilled by 60 distinct assistive services. The second dataset includes a range of 120,000 service ratings of seven quality features, including response, availability, successibility, latency, cost, quality of service, and accessibility. Our deep learning model uses an ensemble of multiple input learners fused using a meta-classification network shared by all the outputs representing individual assistive services. The model achieves significantly better results than traditional machine learning models (i.e., support vector machine and random forest) and a simple feed-forward neural network model without the ensemble technique. Furthermore, we extended our model to utilize the accessibility rating of services to suggest appropriate educational services for disabled learners. The empirical results show the acceptability of our assistive service recommender for learners with disabilities.
]]>