Multimodal Technol. Interact., Volume 6, Issue 2 (February 2022) – 10 articles

Recent years have introduced major shifts in scientific reporting and publishing. The scientific community, publishers, funding agencies, and the public expect research to adhere to principles of openness, reproducibility, replicability, and repeatability. However, studies have shown that scientists often have neither the right tools nor suitable support at their disposal to meet these modern science challenges. In fact, even the concrete expectations connected to these terms may be unclear and subject to field-specific, organizational, and personal interpretations. Based on a narrative literature review of work that defines characteristics of open science, reproducibility, replicability, and repeatability, as well as a review of recent work on researcher-centered requirements, we find that the bottom-up practices and needs of researchers contrast top-down expectations encoded in terms related to reproducibility and open science. We identify and define reproducibility as a central term that concerns the ease of access to scientific resources, as well as their completeness, to the degree required for efficiently and effectively interacting with scientific work. We hope that this characterization helps to create a mutual understanding across science stakeholders, in turn paving the way for suitable and stimulating environments, fit to address the challenges of modern science reporting and publishing. Full article

The development of new approaches for creating more “life-like” artificial intelligence (AI) capable of natural social interaction is of interest to a number of scientific fields, from virtual reality to human–robot interaction to natural language speech systems. Yet how such “Social AI” agents might be manifested remains an open question. Previous research has shown that both behavioral factors related to the artificial agent itself as well as contextual factors beyond the agent (i.e., interaction context) play a critical role in how people perceive interactions with interactive technology. As such, there is a need for customizable agents and customizable environments that allow us to explore both sides in a simultaneous manner. To that end, we describe here the development of a cooperative game environment and Social AI using a data-driven approach, which allows us to simultaneously manipulate different components of the social interaction (both behavioral and contextual). We conducted multiple human–human and human–AI interaction experiments to better understand the components necessary for creation of a Social AI virtual avatar capable of autonomously speaking and interacting with humans in multiple languages during cooperative gameplay (in this case, a social survival video game) in context-relevant ways. Full article

In the context of a European project, we identified over 150 requirements for the development of an augmented reality (AR) head-mounted display (HMD) specifically tailored to support highly challenging manual surgical procedures. The requirements were established by surgeons from different specialties and by industrial players working in the surgical field who had strong commitments to the exploitation of this technology. Some of these requirements were specific to the project, while others can be seen as key requirements for the implementation of an efficient and reliable AR headset to be used to support manual activities in the peripersonal space. The aim of this work is to describe these ergonomic requirements that impact the mechanical design of the HMDs, the possible innovative solutions to these requirements, and how these solutions have been used to implement the AR headset in surgical navigation. We also report the results of a preliminary qualitative evaluation of the AR headset by three surgeons. Full article

Whether or not a product matches the user’s mental model and therefore his understanding of how it works influences the perceived usability. Therefore, it is beneficial if an interface is based on the user’s initial mental model, hence it works just as expected. If it contradicts preexisting models, operating errors and frustrated users are to be expected. This work proposes a method to increase the probability of correspondence between a developed product and the user’s mental model by addressing a common source of error in the product development process: Product designers assuming their own mental model matches the user’s. The process was demonstrated using the example of an adaptive user interface for commercial vehicles. A questionnaire was used to identify the underlying dimensions of the user group’s mental model of adaptive user interfaces. By conducting two expert workshops and a user survey with 75 truck drivers, a questionnaire consisting of 37 items and four dimensions was constructed. Thereby, the initial mental model of truck drivers regarding an adaptive user interface for commercial vehicles was determined. Full article

There is a debate about the way to introduce computational thinking (CT) in schools. Different proposals are on the table; these include the creation of new computational areas for developing CT, the introduction of CT in STEM areas, and the cross-curricular integration of CT in schools. There is also concern that no student should be left behind, independently of their economic situation. To this effect, an unplugged approach is the most cost-effective solution. In addition, this topic is interesting in the context of a pandemic situation that has prevented the sharing of materials between students. This study analyzes an unplugged cross-curricular introduction of CT in the Social Sciences area among sixth grade students. A group of 14 students was selected to carry out an unplugged intervention design—where they were required to program an imaginary robot on paper—in the Social Sciences area. Their CT development and academic results were compared to those of 31 students from the control group who continued attending regular classes. Results showed that an unplugged teaching style of CT in Social Sciences lessons significantly increased CT (p < 0.001) and with a large effect size (d = 1.305) without differences in students’ academic achievement. The findings show that children can potentially develop their CT in non-STEM lessons, learning the same curricular contents, and maintaining their academic results. Full article

As emotions play a central role in human communication, automatic emotion recognition has attracted increasing attention in the last two decades. While multimodal systems enjoy high performances on lab-controlled data, they are still far from providing ecological validity on non-lab-controlled, namely “in-the-wild” data. This work investigates audiovisual deep learning approaches to emotion recognition in in-the-wild problem. Inspired by the outstanding performance of end-to-end and transfer learning techniques, we explored the effectiveness of architectures in which a modality-specific Convolutional Neural Network (CNN) is followed by a Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) using the AffWild2 dataset under the Affective Behavior Analysis in-the-Wild (ABAW) challenge protocol. We deployed unimodal end-to-end and transfer learning approaches within a multimodal fusion system, which generated final predictions using a weighted score fusion scheme. Exploiting the proposed deep-learning-based multimodal system, we reached a test set challenge performance measure of 48.1% on the ABAW 2020 Facial Expressions challenge, which advances the first-runner-up performance. Full article

The potential of games in empowering underrepresented groups is a central theme in the field of media studies. However, to ensure that everyone can benefit from them, it is necessary to ensure that they are inclusive and accessible. In the present work, we have implemented a Participatory Action Research (PAR) approach to target this problem, right at the stage of training new game designers and game developers. Thus, through a game-design-based or experimental game design pedagogical approach, we intended to promote inclusive and accessibility-driven game design and development skills in students, while decreasing their negative attitudes towards people with Intellectual Disability (pwID). A protocol with a Non-Governmental Organization in the field was established and, during two semesters, students were challenged to develop 10 accessible games and physical interfaces for pwID and motor disabilities, through participatory processes. Pre and post assessment was conducted, through a mixed-method approach. After participating in this process, students reported satisfaction and increased knowledge of cognitive and motor accessibility, and inclusive game design in general. Moreover, the attitudinal assessment showed significantly lower levels of prejudiced beliefs towards pwID (p < 0.05). Full article

The relationship between sign systems and the meaning potentials and affordances of multimodal technologies has received increasing attention in research on digital technology use in education. Students constantly adhere to and engage with semiotic shifts in sign systems when they work with digital technologies for learning purposes. This study explores students’ use of digital technologies in Swedish schools. We trace the way semiotic activity systems and cognitive processes are transformed and realized when students engage with shifts in sign systems into various meaning-making strategies. Methodologically, the study is based on a data set of video recordings, interviews, and observations of classroom practice in three primary schools. An analysis that draws on quantitative ethnography was applied to process and analyse the data. The main findings revealed that sign systems prompted by the technologies and the social space compete to some extent for students’ attention, and that technology design is monotonously rendering lower levels of mediation. These findings show that various sign system prompts need to be balanced and streamlined to support students in their meaning-making. This article conveys the importance of understanding sign systems, as they are the most common resources for technology-assisted learning, and change the prerequisites for meaning-making. Full article

Game-based learning is becoming increasingly popular in education. The playful experience especially promises a high degree of students’ motivation. In this research, we examine the influence of sequential scaffolding within a digital educational escape room game. Escape rooms are usually games where players have to escape from a room within a given time limit by completing different tasks and quests. Therefore, we developed an educational virtual escape room for biology classes, focusing on the topic of sex education. In an experiment, we modified this learning environment and developed two different conditions: in one escape room, scaffolding was implemented using sequential learning aids; in the other escape room, which was assigned to the control group, no additional learner support was provided. The main objective of this quantitative research is to measure the escape room’s impact on learning and cognitive load. In addition, motivation, flow experience and experience of immersion are analyzed. A comparison between the two escape rooms shows that additional scaffolding does not significantly increase cognitive load or have any effect on learning. Results show that motivation and knowledge acquisition can be successfully supported by using game-based learning with escape rooms. Full article