Multimodal Technologies and Interaction

26 pages, 2793 KiB

Open AccessReview

Design Considerations for Immersive Virtual Reality Applications for Older Adults: A Scoping Review

by Kiran Ijaz, Tram Thi Minh Tran, Ahmet Baki Kocaballi, Rafael A. Calvo, Shlomo Berkovsky and Naseem Ahmadpour

Multimodal Technol. Interact. 2022, 6(7), 60; https://doi.org/10.3390/mti6070060 - 20 Jul 2022

Cited by 26 | Viewed by 6041

Abstract

Immersive virtual reality (iVR) has gained considerable attention recently with increasing affordability and accessibility of the hardware. iVR applications for older adults present tremendous potential for diverse interventions and innovations. The iVR literature, however, provides a limited understanding of guiding design considerations and [...] Read more.

Immersive virtual reality (iVR) has gained considerable attention recently with increasing affordability and accessibility of the hardware. iVR applications for older adults present tremendous potential for diverse interventions and innovations. The iVR literature, however, provides a limited understanding of guiding design considerations and evaluations pertaining to user experience (UX). To address this gap, we present a state-of-the-art scoping review of literature on iVR applications developed for older adults over 65 years. We performed a search in ACM Digital Library, IEEE Xplore, Scopus, and PubMed (1 January 2010–15 December 2019) and found 36 out of 3874 papers met the inclusion criteria. We identified 10 distinct sets of design considerations that guided target users and physical configuration, hardware use, and software design. Most studies carried episodic UX where only 2 captured anticipated UX and 7 measured longitudinal experiences. We discuss the interplay between our findings and future directions to design effective, safe, and engaging iVR applications for older adults. Full article

(This article belongs to the Special Issue Virtual Reality and Augmented Reality)

► Show Figures

Figure 1

16 pages, 12323 KiB

Open AccessArticle

An Interdisciplinary Design of an Interactive Cultural Heritage Visit for In-Situ, Mixed Reality and Affective Experiences

by Xabier Olaz, Ricardo Garcia, Amalia Ortiz, Sebastián Marichal, Jesús Villadangos, Oscar Ardaiz and Asier Marzo

Multimodal Technol. Interact. 2022, 6(7), 59; https://doi.org/10.3390/mti6070059 - 18 Jul 2022

Cited by 5 | Viewed by 3238

Abstract

Interactive technologies, such as mixed-reality and natural interactions with avatars, can enhance cultural heritage and the experience of visiting a museum. In this paper, we present the design rationale of an interactive experience for a cultural heritage place in the church of Roncesvalles [...] Read more.

Interactive technologies, such as mixed-reality and natural interactions with avatars, can enhance cultural heritage and the experience of visiting a museum. In this paper, we present the design rationale of an interactive experience for a cultural heritage place in the church of Roncesvalles at the beginning of Camino de Santiago. We followed a participatory design with a multidisciplinary team which resulted in the design of a spatial augmented reality system that employs 3D projection mapping and a conversational agent acting as the storyteller. Multiple features were identified as desirable for an interactive experience: interdisciplinary design team; in-situ; mixed reality; interactive digital storytelling; avatar; tangible objects; gestures; emotions and groups. The findings from a workshop are presented for guiding other interactive cultural heritage experiences. Full article

(This article belongs to the Special Issue Digital Cultural Heritage (Volume II))

► Show Figures

Figure 1

16 pages, 791 KiB

Open AccessArticle

The Appraisal Principle in Multimedia Learning: Impact of Appraisal Processes, Modality, and Codality

by Joerg Zumbach, Ines Zeitlhofer, Bettina Mann, Sandra Hoermann and Birgit Reisenhofer

Multimodal Technol. Interact. 2022, 6(7), 58; https://doi.org/10.3390/mti6070058 - 18 Jul 2022

Cited by 2 | Viewed by 2004

Abstract

This paper presents two experiments examining the influences of media-specific appraisal and attribution on multimedia learning. The first experiment compares four different versions of learning material (text, text with images, animation with text, and animation with audio). Results reveal that the attributed type [...] Read more.

This paper presents two experiments examining the influences of media-specific appraisal and attribution on multimedia learning. The first experiment compares four different versions of learning material (text, text with images, animation with text, and animation with audio). Results reveal that the attributed type of appraisal, (i.e., the subjective impression of whether a medium is easy or difficult to learn with) impacts invested mental effort and learning outcomes. Though there was no evidence for the modality effect in the first experiment, we were able to identify it in a second study. We were also able to replicate appraisal and attribution findings from study 1 in study 2: if media appraisal leads to the result that learning with a specific medium is difficult, more mental effort will be invested in information processing. Consequently, learning outcomes are better, and learners are more likely to attribute knowledge acquisition to their own abilities. Outcomes also indicate that the modality effect can be explained by avoidance of split-attention rather than modality-specific information processing in working memory. Full article

► Show Figures

Figure 1

22 pages, 813 KiB

Open AccessArticle

Harvesting Context and Mining Emotions Related to Olfactory Cultural Heritage

by M. Besher Massri, Inna Novalija, Dunja Mladenić, Janez Brank, Sara Graça da Silva, Natasza Marrouch, Carla Murteira, Ali Hürriyetoğlu and Beno Šircelj

Multimodal Technol. Interact. 2022, 6(7), 57; https://doi.org/10.3390/mti6070057 - 18 Jul 2022

Cited by 7 | Viewed by 3633

Abstract

This paper presents an Artificial Intelligence approach to mining context and emotions related to olfactory cultural heritage narratives, particularly to fairy tales. We provide an overview of the role of smell and emotions in literature, as well as highlight the importance of olfactory [...] Read more.

This paper presents an Artificial Intelligence approach to mining context and emotions related to olfactory cultural heritage narratives, particularly to fairy tales. We provide an overview of the role of smell and emotions in literature, as well as highlight the importance of olfactory experience and emotions from psychology and linguistic perspectives. We introduce a methodology for extracting smells and emotions from text, as well as demonstrate the context-based visualizations related to smells and emotions implemented in a novel smell tracker tool. The evaluation is performed using a collection of fairy tales from Grimm and Andersen. We find out that fairy tales often connect smell with the emotional charge of situations. The experimental results show that we can detect smells and emotions in fairy tales with an F1 score of 91.62 and 79.2, respectively. Full article

(This article belongs to the Special Issue Digital Cultural Heritage (Volume II))

► Show Figures

Figure 1

16 pages, 2616 KiB

Open AccessArticle

Detecting Emotions from Illustrator Gestures—The Italian Case

by Daniele Fundarò, Vito Gentile, Fabrizio Milazzo and Salvatore Sorce

Multimodal Technol. Interact. 2022, 6(7), 56; https://doi.org/10.3390/mti6070056 - 17 Jul 2022

Viewed by 2606

Abstract

The evolution of computers in recent years has given a strong boost to research techniques aimed at improving human–machine interaction. These techniques tend to simulate the dynamics of the human–human interaction process, which is based on our innate ability to understand the emotions [...] Read more.

The evolution of computers in recent years has given a strong boost to research techniques aimed at improving human–machine interaction. These techniques tend to simulate the dynamics of the human–human interaction process, which is based on our innate ability to understand the emotions of other humans. In this work, we present the design of a classifier to recognize the emotions expressed by human beings, and we discuss the results of its testing in a culture-specific case study. The classifier relies exclusively on the gestures people perform, without the need to access additional information, such as facial expressions, the tone of a voice, or the words spoken. The specific purpose is to test whether a computer can correctly recognize emotions starting only from gestures. More generally, it is intended to allow interactive systems to be able to automatically change their behaviour based on the recognized mood, such as adapting the information contents proposed or the flow of interaction, in analogy to what normally happens in the interaction between humans. The document first introduces the operating context, giving an overview of the recognition of emotions and the approach used. Subsequently, the relevant bibliography is described and analysed, highlighting the strengths of the proposed solution. The document continues with a description of the design and implementation of the classifier and of the study we carried out to validate it. The paper ends with a discussion of the results and a short overview of possible implications. Full article

(This article belongs to the Special Issue Digital Cultural Heritage (Volume II))

► Show Figures

Figure 1

37 pages, 9873 KiB

Open AccessArticle

Customizing and Evaluating Accessible Multisensory Music Experiences with Pre-Verbal Children—A Case Study on the Perception of Musical Haptics Using Participatory Design with Proxies

by Emma Frid, Claudio Panariello and Claudia Núñez-Pacheco

Multimodal Technol. Interact. 2022, 6(7), 55; https://doi.org/10.3390/mti6070055 - 17 Jul 2022

Cited by 10 | Viewed by 4336

Abstract

Research on Accessible Digital Musical Instruments (ADMIs) has highlighted the need for participatory design methods, i.e., to actively include users as co-designers and informants in the design process. However, very little work has explored how pre-verbal children with Profound and Multiple Disabilities (PMLD) [...] Read more.

Research on Accessible Digital Musical Instruments (ADMIs) has highlighted the need for participatory design methods, i.e., to actively include users as co-designers and informants in the design process. However, very little work has explored how pre-verbal children with Profound and Multiple Disabilities (PMLD) can be involved in such processes. In this paper, we apply in-depth qualitative and mixed methodologies in a case study with four students with PMLD. Using Participatory Design with Proxies (PDwP), we assess how these students can be involved in the customization and evaluation of the design of a multisensory music experience intended for a large-scale ADMI. Results from an experiment focused on communication of musical haptics highlighted the diversity in employed interaction strategies used by the children, accessibility limitations of the current multisensory experience design, and the importance of using a multifaceted variety of qualitative and quantitative methods to arrive at more informed conclusions when applying a design with proxies methodology. Full article

(This article belongs to the Special Issue Musical Interactions (Volume II))

► Show Figures

Figure 1

21 pages, 17802 KiB

Open AccessFeature PaperArticle

A Study on Attention Attracting Elements of 360-Degree Videos Based on VR Eye-Tracking System

by Haram Choi and Sanghun Nam

Multimodal Technol. Interact. 2022, 6(7), 54; https://doi.org/10.3390/mti6070054 - 14 Jul 2022

Cited by 3 | Viewed by 2999

Abstract

In 360-degree virtual reality (VR) videos, users possess increased freedom in terms of gaze movement. As a result, the users’ attention may not move according to the narrative intended by the director and miss out on important parts of the narrative of the [...] Read more.

In 360-degree virtual reality (VR) videos, users possess increased freedom in terms of gaze movement. As a result, the users’ attention may not move according to the narrative intended by the director and miss out on important parts of the narrative of the 360-degree video. Therefore, it is necessary to study a directing technique that can attract user attention in 360-degree VR videos. In this study, we analyzed the directing elements that can attract users’ attention in a 360-degree VR video and developed a 360 VR eye-tracking system to investigate the effect of the attention-attracting elements on the user. Elements that can attract user attention were classified into five categories: object movement, hand gesture, GUI insertion, camera movement, and gaze angle variation. We developed a 360 VR eye-tracking system to analyze whether five attention-attracting elements influence the user’s attention. Based on the eye tracking system, an experiment was conducted to analyze whether the user’s attention moves according to the five attention-attracting elements. Based on the experimental results, it can be seen that ‘hand gesture’ attracted the second most attention shift of the subjects, and ‘GUI insertion’ induced the smallest shift of attention of the subjects. Full article

► Show Figures

Figure 1

20 pages, 1954 KiB

Open AccessArticle

A Comparative Study of Methods for the Visualization of Probability Distributions of Geographical Data

by Sanjana Srabanti, Carolina Veiga, Edcley Silva, Marcos Lage, Nivan Ferreira and Fabio Miranda

Multimodal Technol. Interact. 2022, 6(7), 53; https://doi.org/10.3390/mti6070053 - 13 Jul 2022

Cited by 2 | Viewed by 2671

Abstract

Probability distributions are omnipresent in data analysis. They are often used to model the natural uncertainty present in real phenomena or to describe the properties of a data set. Designing efficient visual metaphors to convey probability distributions is, however, a difficult problem. This [...] Read more.

Probability distributions are omnipresent in data analysis. They are often used to model the natural uncertainty present in real phenomena or to describe the properties of a data set. Designing efficient visual metaphors to convey probability distributions is, however, a difficult problem. This fact is especially true for geographical data, where conveying the spatial context constrains the design space. While many different alternatives have been proposed to solve this problem, they focus on representing data variability. However, they are not designed to support spatial analytical tasks involving probability quantification. The present work aims to adapt recent non-spatial approaches to the geographical context, in order to support probability quantification tasks. We also present a user study that compares the efficiency of these approaches in terms of both accuracy and usability. Full article

► Show Figures

Figure 1

20 pages, 2196 KiB

Open AccessArticle

A Framework for Stakeholders’ Involvement in Digital Productions for Cultural Heritage Tourism

by Licia Calvi, Jessika Weber-Sabil, Daniel Asmar and Xavi Socías Perez

Multimodal Technol. Interact. 2022, 6(7), 52; https://doi.org/10.3390/mti6070052 - 27 Jun 2022

Cited by 1 | Viewed by 3053

Abstract

This paper proposes a new framework for the production and development of immersive and playful technologies in cultural heritage in which different stakeholders such as users and local communities are involved early on in the product development chain. We believe that an early [...] Read more.

This paper proposes a new framework for the production and development of immersive and playful technologies in cultural heritage in which different stakeholders such as users and local communities are involved early on in the product development chain. We believe that an early stage of co-creation in the design process produces a clear understanding of what users struggle with, facilitates the creation of community ownership and helps in better defining the design challenge at hand. We show that adopting such a framework has several direct and indirect benefits, including a deeper sense of site and product ownership as direct benefits to the individual, and the creation and growth of tangential economies to the community. Full article

(This article belongs to the Special Issue Co-Design Within and Between Communities in Cultural Heritage)

► Show Figures

Figure 1

17 pages, 2606 KiB

Open AccessArticle

Is Natural Necessary? Human Voice versus Synthetic Voice for Intelligent Virtual Agents

by Amal Abdulrahman and Deborah Richards

Multimodal Technol. Interact. 2022, 6(7), 51; https://doi.org/10.3390/mti6070051 - 27 Jun 2022

Cited by 11 | Viewed by 4959

Abstract

The use of intelligent virtual agents (IVA) to support humans in social contexts will depend on their social acceptability. Acceptance will be related to the human’s perception of the IVAs as well as the IVAs’ ability to respond and adapt their conversation appropriately [...] Read more.

The use of intelligent virtual agents (IVA) to support humans in social contexts will depend on their social acceptability. Acceptance will be related to the human’s perception of the IVAs as well as the IVAs’ ability to respond and adapt their conversation appropriately to the human. Adaptation implies computer-generated speech (synthetic speech), such as text-to-speech (TTS). In this paper, we present the results of a study to investigate the effect of voice type (human voice vs. synthetic voice) on two aspects: (1) the IVA’s likeability and voice impression in the light of co-presence, and (2) the interaction outcome, including human–agent trust and behavior change intention. The experiment included 118 participants who interacted with either the virtual advisor with TTS or the virtual advisor with human voice to gain tips for reducing their study stress. Participants in this study found the voice of the virtual advisor with TTS to be more eerie, but they rated both agents, with recorded voice and with TTS, similarly in terms of likeability. They further showed a similar attitude towards both agents in terms of co-presence and building trust. These results challenge previous studies that favor human voice over TTS, and suggest that even if human voice is preferred, TTS can deliver equivalent benefits. Full article

► Show Figures

Figure 1

21 pages, 1612 KiB

Open AccessArticle

Inter- and Transcultural Learning in Social Virtual Reality: A Proposal for an Inter- and Transcultural Virtual Object Database to be Used in the Implementation, Reflection, and Evaluation of Virtual Encounters

by Rebecca M. Hein, Marc Erich Latoschik and Carolin Wienrich

Multimodal Technol. Interact. 2022, 6(7), 50; https://doi.org/10.3390/mti6070050 - 25 Jun 2022

Cited by 3 | Viewed by 3249

Abstract

Visual stimuli are frequently used to improve memory, language learning or perception, and understanding of metacognitive processes. However, in virtual reality (VR), there are few systematically and empirically derived databases. This paper proposes the first collection of virtual objects based on empirical evaluation [...] Read more.

Visual stimuli are frequently used to improve memory, language learning or perception, and understanding of metacognitive processes. However, in virtual reality (VR), there are few systematically and empirically derived databases. This paper proposes the first collection of virtual objects based on empirical evaluation for inter-and transcultural encounters between English- and German-speaking learners. We used explicit and implicit measurement methods to identify cultural associations and the degree of stereotypical perception for each virtual stimuli (n = 293) through two online studies, including native German and English-speaking participants. The analysis resulted in a final well-describable database of 128 objects (called InteractionSuitcase). In future applications, the objects can be used as a great interaction or conversation asset and behavioral measurement tool in social VR applications, especially in the field of foreign language education. For example, encounters can use the objects to describe their culture, or teachers can intuitively assess stereotyped attitudes of the encounters. Full article

(This article belongs to the Special Issue Virtual Reality and Augmented Reality)

► Show Figures

Figure 1

16 pages, 2923 KiB

Open AccessArticle

Vocational Training in Virtual Reality: A Case Study Using the 4C/ID Model

by Miriam Mulders

Multimodal Technol. Interact. 2022, 6(7), 49; https://doi.org/10.3390/mti6070049 - 24 Jun 2022

Cited by 9 | Viewed by 4145

Abstract

Virtual reality (VR) is an emerging technology with a variety of potential benefits for vocational training. Therefore, this paper presents a VR training based on the highly validated 4C/ID model to train vocational competencies in the field of vehicle painting. The following 4C/ID [...] Read more.

Virtual reality (VR) is an emerging technology with a variety of potential benefits for vocational training. Therefore, this paper presents a VR training based on the highly validated 4C/ID model to train vocational competencies in the field of vehicle painting. The following 4C/ID components were designed using the associated 10 step approach: learning tasks, supportive information, procedural information, and part-task practice. The paper describes the instructional design process including an elaborated blueprint for a VR training application for aspiring vehicle painters. We explain the model’s principles and features and their suitability for designing a VR vocational training that fosters integrated competence acquisition. Following the methodology of design-based research, several research methods (e.g., a target group analysis) and the ongoing development of prototypes enabled agile process structures. Results indicate that the 4C/ID model and the 10 step approach promote the instructional design process using VR in vocational training. Implementation and methodological issues that arose during the design process (e.g., limited time within VR) are adequately discussed in the article. Full article

(This article belongs to the Special Issue Virtual Reality and Augmented Reality)

► Show Figures

Figure 1

25 pages, 1963 KiB

Open AccessArticle

AI Technologies for Machine Supervision and Help in a Rehabilitation Scenario

by Gábor Baranyi, Bruno Carlos Dos Santos Melício, Zsófia Gaál, Levente Hajder, András Simonyi, Dániel Sindely, Joul Skaf, Ondřej Dušek, Tomáš Nekvinda and András Lőrincz

Multimodal Technol. Interact. 2022, 6(7), 48; https://doi.org/10.3390/mti6070048 - 22 Jun 2022

Cited by 8 | Viewed by 4395

Abstract

We consider, evaluate, and develop methods for home rehabilitation scenarios. We show the required modules for this scenario. Due to the large number of modules, the framework falls into the category of Composite AI. Our work is based on collected videos with high-quality [...] Read more.

We consider, evaluate, and develop methods for home rehabilitation scenarios. We show the required modules for this scenario. Due to the large number of modules, the framework falls into the category of Composite AI. Our work is based on collected videos with high-quality execution and samples of typical errors. They are augmented by sample dialogues about the exercise to be executed and the assumed errors. We study and discuss body pose estimation technology, dialogue systems of different kinds and the emerging constraints of verbal communication. We demonstrate that the optimization of the camera and the body pose allows high-precision recording and requires the following components: (1) optimization needs a 3D representation of the environment, (2) a navigation dialogue to guide the patient to the optimal pose, (3) semantic and instance maps are necessary for verbal instructions about the navigation. We put forth different communication methods, from video-based presentation to chit-chat-like dialogues through rule-based methods. We discuss the methods for different aspects of the challenges that can improve the performance of the individual components. Due to the emerging solutions, we claim that the range of applications will drastically grow in the very near future. Full article

(This article belongs to the Topic Interactive Artificial Intelligence and Man-Machine Communication)

► Show Figures

Figure 1

Journal Menu

Journal Browser

Multimodal Technol. Interact., Volume 6, Issue 7 (July 2022) – 13 articles

Further Information

Guidelines

MDPI Initiatives

Follow MDPI