Multimodal Technol. Interact., Volume 5, Issue 7 (July 2021) – 10 articles

The objective of the study was to learn about the relationships between BookTubers and their subscribers by focusing on the comments left by viewers of the audio-visual literary reviews. We also examined whether viewer-BookTuber relationships resulted in the promotion of reading. A mixed qualitative-quantitative methodology was followed, including a descriptive analysis of contents and a case study. The main tools used were MAXQDA to process the qualitative data and Excel to obtain the quantitative data. The sample was a non-random selection of four BookTubers channels, taking into account both their impact and gender equality (two female and two male BookTubers). The categorization was conducted based on Cultural Studies and Reception Aesthetics. A total of eight videos (four reviews and four Book Hauls) were selected and 100 comments on each were analyzed, giving rise to four categories. The results indicated that in terms of content decoding, close relationships were established among community members, between both consumers and producers. In addition, message acceptance took place and a certain relationship was found between the BookTuber’s work and the promotion of reading. BookTubers were therefore identified as multimodal influencers. Full article

Numerous technological solutions have been proposed to promote piano learning and teaching, but very few with market success. We are convinced that users’ needs should be the starting point for an effective and transdisciplinary development process of piano-related Tactile Internet with Human-in-the-Loop (TaHIL) applications. Thus, we propose to include end users in the initial stage of technology development. We gathered insights from adult piano teachers and students through an online survey and digital interviews. Three potential literature-based solutions have been visualized as scenarios to inspire participants throughout the interviews. Our main findings indicate that potential end users consider posture and body movements, teacher–student communication, and self-practice as crucial aspects of piano education. Further insights resulted in so-called acceptance requirements for each scenario, such as enabling meaningful communication in distance teaching, providing advanced data on a performer’s body movement for increased well-being, and improving students’ motivation for self-practice, all while allowing or even promoting artistic freedom of expression and having an assisting instead of judging character. By putting the users in the center of the fuzzy front end of technology development, we have gone a step further toward concretizing TaHIL applications that may contribute to the routines of piano teaching and learning. Full article

Augmented reality (AR) enables an interactive experience with the real world where real-world objects are enhanced with computer-generated perceptual information. Twenty-five years have passed since the first AR application designed exclusively to be used in educational settings. Since then, this technology has been successfully implemented to enrich educational contexts providing learning gains, motivation, enjoyment, autonomy, among other benefits. This study provides an overview of AR technology in education from its origins to the present. Consequently, based on the analysis of its evolution, the study defines three generations of AR in education. Moreover, the study identifies some major challenges from previous AR applications and, finally, it poses some insights to address these challenges to enhance the benefits of AR for education. Full article

Virtual Reality (VR) has become a consumer-grade technology, especially with the advent of standalone headsets working independently from a powerful computer. Domestic VR mainly uses the visual and auditory senses since VR headsets make this accessible. Haptic feedback, however, has the potential to increase immersion substantially. So far, it is mostly used in laboratory settings with specialized haptic devices. Especially for domestic VR, there is underexplored potential in exploiting physical elements of the often confined space in which it is used. In a literature review (n = 20), we analyzed VR interaction using haptic feedback with or without physical limitations. From this, we derive a design space for VR haptics across three spatial scales (seated, standing, and walking). In our narrow selection of papers, we found inspirations for future work and will discuss two example scenarios. Our work gives a current overview of haptic VR solutions and highlights strategies for adapting laboratory solutions to an everyday context. Full article

Background: There are 4.9 million English Language Learners (ELLs) in the United States. Only 2% of educators are trained to support these vulnerable students. Social robots show promise for language acquisition and may provide valuable support for students, especially as we return to needing smaller classes due to COVID-19. While cultural responsiveness increases gains for ELLs, little is known about the design of culturally responsive child–robot interactions. Method: Therefore, using a participatory design approach, we conducted an exploratory study with 24 Spanish-speaking ELLs at a Pacific Northwest elementary school. As cultural informants, students participated in a 15-min, robot-led, small group story discussion followed by a post-interaction feedback session. We then conducted reflexive critiques with six ELL teachers who reviewed the group interactions to provide further interpretation on design feature possibilities and potential interactions with the robot. Results: Students found the social robot engaging, but many were hesitant to converse with the robot. During post-interaction dialogue students articulated the specific ways in which the social robot appearance and behavior could be modified to help them feel more comfortable. Teachers postulated that the social robot could be designed to engage students in peer-to-peer conversations. Teachers also recognized the ELLs verbosity when discussing their experiences with the robot and suggested such interactions could stimulate responsiveness from students. Conclusion: Cultural responsiveness is a key component to successful education in ELLs. However, integrating appropriate, cultural responsiveness into robot interactions may require participants as cultural informants to ensure the robot behaviors and interactions are situated in that educational community. Utilizing a participatory approach to engage ELLs in design decisions for social robots is a promising way to gather culturally responsive requirements to inform successful child–robot interactions. Full article

Hateful and abusive speech presents a major challenge for all online social media platforms. Recent advances in Natural Language Processing and Natural Language Understanding allow for more accurate detection of hate speech in textual streams. This study presents a new multimodal approach to hate speech detection by combining Computer Vision and Natural Language processing models for abusive context detection. Our study focuses on Twitter messages and, more specifically, on hateful, xenophobic, and racist speech in Greek aimed at refugees and migrants. In our approach, we combine transfer learning and fine-tuning of Bidirectional Encoder Representations from Transformers (BERT) and Residual Neural Networks (Resnet). Our contribution includes the development of a new dataset for hate speech classification, consisting of tweet IDs, along with the code to obtain their visual appearance, as they would have been rendered in a web browser. We have also released a pre-trained Language Model trained on Greek tweets, which has been used in our experiments. We report a consistently high level of accuracy (accuracy score = 0.970, f1-score = 0.947 in our best model) in racist and xenophobic speech detection. Full article

Robots have been proposed as intelligent technology that can support the independent living and health of older adults. While significant advances are being made regarding hardware and intelligent software to support autonomous actions of robots, less emphasis has been put on designing robot behavior that is comprehensible and pleasant for older adults. However, good usability and user experience are crucial factors for acceptance and long-term use. One way to actively engage older adults in behavioral design for social robots is participatory design. The Modality Card Deck is proposed, a tool that helps to engage older adults in human-robot interaction design process and participate in design decision for robot behavior. The cards guide the users towards creating ideas for design solutions which are detailed enough to be implemented by interaction designers and software developers. This paper provides a detailed description of the Modality Card Deck and presents an evaluation of the tool in the scope of a case study. In the case study, the card deck was used in participatory design workshops with older adults to develop multi-modal robot behaviors for the Pepper robot and a quiz game application. After describing the procedure of the case study, the workshop results and learnings about working with the Modality Card Deck and older adults are presented. Full article

Mobile handheld devices, such as smartphones and tablets, have become some of the most prominent ubiquitous terminals within the information and communication technology landscape. Their transformative power within the digital music domain changed the music ecosystem from production to distribution and consumption. Of interest here is the ever-expanding number of mobile music applications. Despite their growing popularity, their design in terms of interaction perception and control is highly arbitrary. It remains poorly addressed in related literature and lacks a clear, systematized approach. In this context, our paper aims to provide the first steps towards defining guidelines for optimal sonic interaction design practices in mobile music applications. Our design approach is informed by user data in appropriating mobile handheld devices. We conducted an experiment to learn links between control gestures and musical parameters, such as pitch, duration, and amplitude. A twofold action—reflection protocol and tool-set for evaluating the aforementioned links—are also proposed. The results collected from the experiment show statistically significant trends in pitch and duration control gesture mappings. On the other hand, amplitude appears to elicit a more diverse mapping approach, showing no definitive trend in this experiment. Full article

Many virtual reality (VR) applications use teleport for locomotion. The non-continuous locomotion of teleport is suited for VR controllers and can minimize simulator sickness, but it can also reduce spatial awareness compared to continuous locomotion. Our aim was to create continuous, controller-based locomotion techniques that would support spatial awareness. We compared the new techniques, slider and grab, with teleport in a task where participants counted small visual targets in a VR environment. Task performance was assessed by asking participants to report how many visual targets they found. The results showed that slider and grab were significantly faster to use than teleport, and they did not cause significantly more simulator sickness than teleport. Moreover, the continuous techniques provided better spatial awareness than teleport. Full article

The potential of Augmented Reality (AR) for educational and training purposes is well known. While large-scale deployments of head-mounted AR headsets remain challenging due to technical limitations and cost factors, advances in mobile devices and tracking solutions introduce handheld AR devices as a powerful, broadly available alternative, yet with some restrictions. One of the current limitations of AR training applications on handheld AR devices is that most offer rather static experiences, only providing descriptive knowledge with little interactivity. Holistic concepts for the coverage of procedural knowledge are largely missing. The contribution of this paper is twofold. We propose a scalabe interaction concept for handheld AR devices with an accompanied didactic framework for procedural training tasks called TrainAR. Then, we implement TrainAR for a training scenario in academics for the context of midwifery and explain the educational theories behind our framework and how to apply it for procedural training tasks. We evaluate and subsequently improve the concept based on three formative usability studies (n = 24), where explicitness, redundant feedback mechanisms and onboarding were identified as major success factors. Finally, we conclude by discussing derived implications for improvements and ongoing and future work. Full article