Multimodal Technol. Interact., Volume 7, Issue 7 (July 2023) – 12 articles

As technology has explosively and globally revolutionized the teaching and learning processes at educational institutions, enormous and innovative technological developments, along with their tools and applications, have recently invaded the education system. Using mobile learning (m-learning) employs wireless technologies for thinking, communicating, learning, and sharing to disseminate and exchange knowledge. Consequently, assessing the learning attitudes of students toward mobile learning is crucial, as learning attitudes impact their motivation, performance, and beliefs about mobile learning. However, mobile learning seems under-researched and may require additional efforts from researchers, especially in the context of the Middle East. Hence, this study’s contribution is enhancing our knowledge about students’ attitudes towards mobile-based learning. Therefore, the study goal was to investigate m-learning’s effect on the learning attitudes among technology education students. An explanatory sequential mixed approach was utilized to examine the attitudes of 50 students who took an educational technology class. A quasi-experiment was conducted and a phenomenological approach was adopted. Data from the experimental group and the control group were gathered. Focus group discussions with three groups and 25 semi-structured interviews were performed with students who experienced m-learning in their course. ANCOVA was conducted and revealed the impact of m-learning on the attitudes and their components. An inductive and deductive content analysis was conducted. Eleven subthemes stemmed out of three main themes. These subthemes included: personalized learning, visualization of learning motivation, less learning frustration, enhancing participation, learning on familiar devices, and social interaction, which emerged from the data. The researchers recommended that higher education institutions adhere to a set of guiding principles when creating m-learning policies. Additionally, they should customize the m-learning environment with higher levels of interactivity to meet students’ needs and learning styles to improve their attitudes towards m-learning. Full article

Data physicalisations, or physical visualisations, represent data physically, using variable properties of physical media. As an emerging area, Data physicalisation research needs conceptual foundations to support thinking about and designing new physical representations of data and evaluating them. Yet, it remains unclear at the moment (i) what encoding variables are at the designer’s disposal during the creation of physicalisations, (ii) what evaluation criteria could be useful, and (iii) what methods can be used to evaluate physicalisations. This article addresses these three questions through a narrative review and a systematic review. The narrative review draws on the literature from Information Visualisation, HCI and Cartography to provide a holistic view of encoding variables for data. The systematic review looks closely into the evaluation criteria and methods that can be used to evaluate data physicalisations. Both reviews offer a conceptual framework for researchers and designers interested in designing and evaluating data physicalisations. The framework can be used as a common vocabulary to describe physicalisations and to identify design opportunities. We also proposed a seven-stage model for designing and evaluating physical data representations. The model can be used to guide the design of physicalisations and ideate along the stages identified. The evaluation criteria and methods extracted during the work can inform the assessment of existing and future data physicalisation artefacts. Full article

The paper presents an existing scenario related to the advanced integration of digital technologies in the field of house museums, based on the critical literature and applied experimentation. House museums are a particular type of heritage site, in which is highlighted the tension between the evocative capacity of the spaces and the requirements for preservation. In this dimension, the use of a seamless approach amplifies the atmospheric component of the space, superimposing, through hybrid digital technologies, an interactive, context-driven layer in an open dialogue between digital and physical. The methodology moves on the one hand from the literature review, framing the macro themes of research, and on the other from the overview of case studies, selected on the basis of the experiential value of the space. The analysis of the selected cases followed as criteria: the formal dimension of the technology; the narrative plot, as storytelling of socio-cultural atmosphere or identification within the intimate story; and the involvement of visitors as individual immersion or collective rituality. The paper aimed at outlining a developmental panorama in which the integration of hybrid technologies points to a new seamless awareness within application scenarios as continuous and work-in-progress challenges. Full article

Recent technological advances have enabled the development of sophisticated prosthetic hands, which can help their users to compensate lost motor functions. While research and development has mostly addressed the functional requirements and needs of users of these prostheses, their broader societal perception (e.g., by external observers not affected by limb loss themselves) has not yet been thoroughly explored. To fill this gap, we investigated how the physical design of artificial hands influences the perception by external observers. First, we conducted an online study (n = 42) to explore the emotional response of observers toward three different types of artificial hands. Then, we conducted a lab study (n = 14) to examine the influence of design factors and depth of interaction on perceived trust and usability. Our findings indicate that some design factors directly impact the trust individuals place in the system’s capabilities. Furthermore, engaging in deeper physical interactions leads to a more profound understanding of the underlying technology. Thus, our study shows the crucial role of the design features and interaction in shaping the emotions around, trust in, and perceived usability of artificial hands. These factors ultimately impact the overall perception of prosthetic systems and, hence, the acceptance of these technologies in society. Full article

To create a truly accessible and inclusive society, we need to take the more than 2.2 billion people with vision impairments worldwide into account when we design our cities, buildings, and everyday objects. This requires sympathy and empathy, as well as a certain level of understanding of the impact of vision impairments on perception. In this study, we explore the potential of an extended version of our vision-impairment simulation system XREye to increase sympathy and empathy and evaluate its educational value in an expert study with 56 educators and education students. We include data from a previous study in related work on sympathy and empathy as a baseline for comparison with our data. Our results show increased sympathy and empathy after experiencing XREye and positive feedback regarding its educational value. Hence, we believe that vision-impairment simulations, such as XREye, have merit to be used for educational purposes in order to increase awareness for the challenges people with vision impairments face in their everyday lives. Full article

Measuring learning curves allows for the inspection of the rate of learning and competency threshold for each individual, training lesson, or training method. In this work, we investigated learning curves in acupuncture needle manipulation training with continuous performance measurement using a vision-based needle training system. We tracked the needle insertion depth of 10 students to investigate their learning curves. The results show that the group-level learning curve was fitted with the Thurstone curve, indicating that students were able to improve their needle insertion skills after repeated practice. Additionally, the analysis of individual learning curves revealed valuable insights into the learning experiences of each participant, highlighting the importance of considering individual differences in learning styles and abilities when designing training programs. Full article

There is a large interest in the annotation of speech addressed to infants. Infant-directed speech (IDS) has acoustic properties that might pose a challenge to automatic speech recognition (ASR) tools developed for adult-directed speech (ADS). While ASR tools could potentially speed up the annotation process, their effectiveness on this speech register is currently unknown. In this study, we assessed to what extent open-source ASR tools can successfully transcribe IDS. We used speech data from 21 Dutch mothers reading picture books containing target words to their 18- and 24-month-old children (IDS) and the experimenter (ADS). In Experiment 1, we examined how the ASR tool Kaldi-NL performs at annotating target words in IDS vs. ADS. We found that Kaldi-NL only found 55.8% of target words in IDS, while it annotated 66.8% correctly in ADS. In Experiment 2, we aimed to assess the difficulties in annotating IDS more broadly by transcribing all IDS utterances manually and comparing the word error rates (WERs) of two different ASR systems: Kaldi-NL and WhisperX. We found that WhisperX performs significantly better than Kaldi-NL. While there is much room for improvement, the results show that automatic transcriptions provide a promising starting point for researchers who have to transcribe a large amount of speech directed at infants. Full article

Accurate and timely diagnosis is a pillar of effective healthcare. However, the challenge lies in gathering extensive training data while maintaining patient privacy. This study introduces a novel approach using federated learning (FL) and a cross-device multimodal model for clinical event classification based on vital signs data. Our architecture employs FL to train several machine learning models including random forest, AdaBoost, and SGD ensemble models on vital signs data. The data were sourced from a diverse clientele at a Boston hospital (MIMIC-IV dataset). The FL structure trains directly on each client’s device, ensuring no transfer of sensitive data and preserving patient privacy. The study demonstrates that FL offers a powerful tool for privacy-preserving clinical event classification, with our approach achieving an impressive accuracy of 98.9%. These findings highlight the significant potential of FL and cross-device ensemble technology in healthcare applications, especially in the context of handling large volumes of sensitive patient data. Full article

This paper takes a dynamic interactive stance to music listening. It revolves around the focal concept of entrainment as an operational tool for the description of fine-grained dynamics between the music as an entraining stimulus and the listener as an entrained subject. Listeners, in this view, can be “entrained” by the sounds at several levels of processing, dependent on the degree of attunement and alignment of their attention. The concept of entrainment, however, is somewhat ill-defined, with distinct conceptual labels, such as external vs. mutual, symmetrical vs. asymmetrical, metrical vs. non-metrical, within-persons and between-person, and physical vs. cognitive entrainment. The boundaries between entrainment, resonance, and synchronization are also not always very clear. There is, as such, a need for a broadened approach to entrainment, taking as a starting point the concept of oscillators that interact with each other in a continuous and ongoing way, and relying on the theoretical framework of interaction dynamics and the concept of adaptation. Entrainment, in this broadened view, is seen as an adaptive process that accommodates to the music under the influence of both the attentional direction of the listener and the configurations of the sounding stimuli. Full article

While Industrial Augmented Reality (IAR) has many applications across the whole product lifecycle, most IAR applications today are custom-built for specific use-cases in practice. This contribution builds upon a scoping literature review of IAR data representations to present a modern, modular IAR architecture. The individual modules of the presented approach are either responsible for user interface and user interaction or for data processing. They are use-case neutral and independent of each other, while communicating through a strictly separated application layer. To demonstrate the architecture, this contribution presents an assembly process that is supported once with a pick-to-light system and once using in situ projections. Both are implemented on top of the novel architecture, allowing most of the work on the individual models to be reused. This IAR architecture, based on clearly separated modules with defined interfaces, particularly allows small companies with limited personnel resources to adapt IAR for their specific use-cases more easily than developing single-use applications from scratch. Full article

Augmented reality has emerged as a transformative technology, with the potential to revolutionize the tourism industry. Nonetheless, there is a scarcity of studies tracing the progression of AR and its application in tourism, from early exploration to recent advancements. This study aims to provide a comprehensive overview of the evolution, contexts, and design elements of AR in tourism over the period (2002–2022), offering insights for further progress in this domain. Employing a dual-method approach, a bibliometric analysis was conducted on 861 articles collected from the Scopus and Web of Science databases, to investigate the evolution of AR research over time and across countries, and to identify the main contexts of the utilization of AR in tourism. In the second part of our study, a systematic content analysis was conducted, focusing on a subset of 57 selected studies that specifically employed AR systems in various tourism situations. Through this analysis, the most commonly utilized AR design components, such as tracking systems, AR devices, tourism settings, and virtual content were summarized. Furthermore, we explored how these components were integrated to enhance the overall tourism experience. The findings reveal a growing trend in research production, led by Europe and Asia. Key contexts of AR applications in tourism encompass cultural heritage, mobile AR, and smart tourism, with emerging topics such as artificial intelligence (AI), big data, and COVID-19. Frequently used AR design components comprise mobile devices, marker-less tracking systems, outdoor environments, and visual overlays. Future research could involve optimizing AR experiences for users with disabilities, supporting multicultural experiences, integrating AI with big data, fostering sustainability, and remote virtual tourism. This study contributes to the ongoing discourse on the role of AR in shaping the future of tourism in the post COVID-19 era, by providing valuable insights for researchers, practitioners, and policymakers in the tourism industry. Full article

Projected walk-through fogscreens have been created, but there is little research on the evaluation of the interaction performance with fogscreens. The present study investigated mid-air hand gestures for interaction with a large fogscreen. Participants (N = 20) selected objects from a fogscreen using tapping and dwell-based gestural techniques, with and without vibrotactile/haptic feedback. In terms of Fitts’ law, the throughput was about 1.4 bps to 2.6 bps, suggesting that gestural interaction with a large fogscreen is a suitable and effective input method. Our results also suggest that tapping without haptic feedback has good performance and potential for interaction with a fogscreen, and that tactile feedback is not necessary for effective mid-air interaction. These findings have implications for the design of gestural interfaces suitable for interaction with fogscreens. Full article