Multimodal Technol. Interact., Volume 3, Issue 3 (September 2019) – 20 articles

A shape-changing user interface is a type of interface that interacts with users by changing its physical form. Although researchers have been extensively studying shape-changing user interfaces, relevant research on its various design aspects—including tools and methods—remains limited. Prototyping shape-changing interfaces often requires sophisticated equipment and knowledge, which makes this sphere of design unwelcoming for designers with limited resources and technical knowledge (e.g., design students). In this study, we propose ClothSurface—a simple and low-cost prototyping tool to design for shape displays—and explore its use through a series of design sessions. The results reveal that ClothSurface can allow inexperienced designers to illustrate their ideas and to explore the design space of shape displays. Full article

With the prevalence of digital technologies and internet connectivity, combined with the reduction in footfall on high streets, banks have taken steps to move most of their customer base online. This has left many older adults behind, trying to keep up with the changes and having to learn to use sometimes complex online banking interfaces. In this work we investigate whether skeuomorphic design can create a more usable online banking system for older adults, compared to the more commonplace flat design. This work took a user-centered approach, beginning with interviews with older adults that were conducted to gather data to be used in the production of prototype user interfaces. Two prototypes were then created: a flat user interface and a skeuomorphic one. We evaluated these interfaces with 15 older adults, gathering a combination of data, including data from the System Usability Scale, observations, and interviews. Results of the experiments showed that our older users preferred the flat prototype to the skeuomorphic one, but raised some potentially useful guidelines for the design of future skeuomorphic user interfaces for older adults. A validation experiment with 17 younger adults (aged 20–25) also showed that the skeuomorphic interface was more usable for older adults than younger ones. Full article

The number of cyberattacks on organizations is growing. To increase cyber resilience, organizations need to obtain foresight to anticipate cybersecurity vulnerabilities, developments, and potential threats. This paper describes a tool that combines state of the art text mining and information retrieval techniques to explore the opportunities of using these techniques in the cybersecurity domain. Our tool, the Horizon Scanner, can scrape and store data from websites, blogs and PDF articles, and search a database based on a user query, show textual entities in a graph, and provide and visualize potential trends. The aim of the Horizon Scanner is to help experts explore relevant data sources for potential threats and trends and to speed up the process of foresight. In a requirements session and user evaluation of the tool with cyber experts from the Dutch Defense Cyber Command, we explored whether the Horizon Scanner tool has the potential to fulfill its aim in the cybersecurity domain. Although the overall evaluation of the tool was not as good as expected, some aspects of the tool were found to have added value, providing us with valuable insights into how to design decision support for forecasting analysts. Full article

This paper investigates the effects of gaze-based player guidance on the perceived game experience, performance, and challenge in a first-person exploration game. In contrast to existing research, the proposed approach takes the game context into account by providing players not only with guidance but also granting them an engaging game experience with a focus on exploration. This is achieved by incorporating gaze-sensitive areas that indicate the location of relevant game objects. A comparative study was carried out to validate our concept and to examine if a game supported with a gaze guidance feature triggers a more immersive game experience in comparison to a crosshair guidance version and a solution without any guidance support. In general, our study findings reveal a more positive impact of the gaze-based guidance approach on the experience and performance in comparison to the other two conditions. However, subjects had a similar impression concerning the game challenge in all conditions. Full article

In the medical domain, user-generated social media text is increasingly used as a valuable
complementary knowledge source to scientific medical literature. The extraction of this knowledge is
complicated by colloquial language use and misspellings. However, lexical normalization of such
data has not been addressed effectively. This paper presents a data-driven lexical normalization
pipeline with a novel spelling correction module for medical social media. Our method significantly
outperforms state-of-the-art spelling correction methods and can detect mistakes with an F₁ of 0.63
despite extreme imbalance in the data. We also present the first corpus for spelling mistake detection
and correction in a medical patient forum. Full article

Digital storytelling, including text, images, audio, music, and video, has been researched as a means of enhancing learners’ motivation, autonomy, and engagement and as a way to improve oral and speaking skills in foreign language learning. This study examined the relation between adults’ engagement in digital storytelling (scaffolded by an interactive learning environment) and their speaking skills and motivation when learning a foreign language. The study used a pre-test, post-test control group design with two groups of 20 Russians who were beginners in learning Greek as a foreign language (n = 40). The 12-h intervention was technology-supported only for the experimental group. Even though the comparison of participants’ recorded speech pre- and post-intervention revealed a statistically significant decrease of mistakes made during speech from pre- to post-intervention for both groups, an independent samples t-test to compare the groups’ post-intervention speaking performance revealed a statistically significant difference in favor of the experimental group (t(38) = 4.05, p < 0.05). The analysis of results from a motivation questionnaire administered pre- and post-intervention showed a statistically significant increase in the motivation of the experimental group only. Findings provide an indication that digital storytelling, scaffolded by an interactive learning environment, supports the development of adults’ speaking skills in a foreign language and increases their motivation. Full article

Keyphrase extraction is an important part of natural language processing (NLP) research, although little research is done in the domain of web pages. The World Wide Web contains billions of pages that are potentially interesting for various NLP tasks, yet it remains largely untouched in scientific research. Current research is often only applied to clean corpora such as abstracts and articles from academic journals or sets of scraped texts from a single domain. However, textual data from web pages differ from normal text documents, as it is structured using HTML elements and often consists of many small fragments. These elements are furthermore used in a highly inconsistent manner and are likely to contain noise. We evaluated the keyphrases extracted by several state-of-the-art extraction methods and found that they did not transfer well to web pages. We therefore propose WebEmbedRank, an adaptation of a recently proposed extraction method that can make use of structural information in web pages in a robust manner. We compared this novel method to other baselines and state-of-the-art methods using a manually annotated dataset and found that WebEmbedRank achieved significant improvements over existing extraction methods on web pages. Full article

Current advancements in music technology enable the creation of customized Digital Musical Instruments (DMIs). This paper presents a systematic review of Accessible Digital Musical Instruments (ADMIs) in inclusive music practice. History of research concerned with facilitating inclusion in music-making is outlined, and current state of developments and trends in the field are discussed. Although the use of music technology in music therapy contexts has attracted more attention in recent years, the topic has been relatively unexplored in Computer Music literature. This review investigates a total of 113 publications focusing on ADMIs. Based on the 83 instruments in this dataset, ten control interface types were identified: tangible controllers, touchless controllers, Brain–Computer Music Interfaces (BCMIs), adapted instruments, wearable controllers or prosthetic devices, mouth-operated controllers, audio controllers, gaze controllers, touchscreen controllers and mouse-controlled interfaces. The majority of the AMDIs were tangible or physical controllers. Although the haptic modality could potentially play an important role in musical interaction for many user groups, relatively few of the ADMIs (14.5%) incorporated vibrotactile feedback. Aspects judged to be important for successful ADMI design were instrument adaptability and customization, user participation, iterative prototyping, and interdisciplinary development teams. Full article

With an increasing worldwide need for energy and the ever-decreasing availability of energy resources, a wide range of interactive visualizations are being developed to allow people to use energy more efficiently by monitoring their consumption patterns and changing their energy usage behavior. For these visualizations to achieve their aim, they must not only target people’s energy saving objectives but also support the necessary factors that help people to change their underlying energy consumption behavior. In this paper, we survey several categories of existing interactive energy visualizations and through a number of selected examples in each case, identify possible potentials for supporting any user behavior changes. For this survey, we have used the behavior change model originally proposed by B. J. Fogg, which defines three factors of motivation, trigger and ability. Our survey has shown that most existing interactive visualizations target the motivation factor, with some supporting trigger or ability and only a few dealing with all the three factors of the behavior change model. Full article

The design of educational serious games to be used as motivational learning environments is very rewarding but also very challenging. The integration of learning with playing activities seems to be one of the major challenges. Previous work has not examined the integration of science content based on the school curriculum in the gameplay mechanics of a computer role playing game (CRPG) and in a storyline which ends with a dramatic conflict. In this study, an educational CRPG was designed for learning concepts in the physical sciences according to the curriculum of the correspondent book. We integrated the content into the gameplay which included several tasks to be performed and puzzles to be solved, advancing players to successfully complete the game, according to the game’s storyline. Finally, we performed a usability test for ease-of-use and enjoyment issues. Most testers considered the educational CRPG entertaining. Computer role playing game’s gameplay mechanics provide affordances for the integration of science and technology courses in a playful learning environment. Full article

Given a meeting participant’s turn-taking dynamics during one segment of a meeting, and their contribution to the group discussion up to that point, our aim is to automatically predict their activity level at a later point of the meeting. The predictive models use verbal and nonverbal features derived from social network representations of each small group interaction. The best automatic prediction models consistently outperform two baseline models at multiple time-lags. We analyze which interaction features are most predictive of later meeting activity levels, and investigate the efficacy of the verbal vs. nonverbal feature classes for this prediction task. At long time-lags, linguistic features become more crucial, but performance degrades compared with prediction at short time-lags. Full article

This paper presents a dialogue act taxonomy designed for the development of a conversational agent for elderly. The main goal of this conversational agent is to improve life quality of the user by means of coaching sessions in different topics. In contrast to other approaches such as task-oriented dialogue systems and chit-chat implementations, the agent should display a pro-active attitude, driving the conversation to reach a number of diverse coaching goals. Therefore, the main characteristic of the introduced dialogue act taxonomy is its capacity for supporting a communication based on the GROW model for coaching. In addition, the taxonomy has a hierarchical structure between the tags and it is multimodal. We use the taxonomy to annotate a Spanish dialogue corpus collected from a group of elder people. We also present a preliminary examination of the annotated corpus and discuss on the multiple possibilities it presents for further research. Full article

With the growth in popularity of video games in our society many teachers have worked to incorporate gaming into their classroom. It is generally agreed that by adding something fun to the learning process students become more engaged and, consequently, retain more knowledge. However, although the characteristics of video games facilitate the dynamics of the educational process it is necessary to plan a pedagogical project that includes delimitation of learning goals and profile of the addressees, the conditions of application of the educational project, and the methodologies of evaluation of the learning progress. This is how we can make a real difference between gamification and video game based learning. The paper addresses the design of an educational resource for special education needs (SEN) students that aims to help teach communicative skills related to prosody. The technological choices made to support the pedagogic issues that underlie the educational product, the strategies to convert learning content into playful material, and the methodology to obtain measures of its playability and effectiveness are described. The results of the motivation test certified that the video game is useful in encouraging the users to exercise their voice and the indicators of the degree of achievement of the learning goals serve to identify the most affected prosodic skills. Full article

Meeting minutes are useful, but creating meeting summaries are a time consuming task. Aiming at supporting such task, this paper proposes prediction models for important utterances that should be included in the meeting summary by using multimodal and multiparty features. We will tackle this issue from two approaches: Handcrafted feature models and deep neural network models. The best handcrafted feature model achieved 0.707 in F-measure, and the best deep-learning based verbal and nonverbal model (V-NV model) achieved 0.827 in F-measure. Based on the V-NV model, we implemented a meeting browser, and conducted a user study. The results showed that the proposed meeting browser better contributes to the understanding of the content of the discussion and the participant roles in the discussion than the conventional text-based browser. Full article

The integration of clickers in Higher Education settings has proved to be particularly useful for enhancing motivation, engagement and performance; for developing cooperative or collaborative tasks; for checking understanding during the lesson; or even for assessment purposes. This paper explores and exemplifies three uses of Socrative, a mobile application specifically designed as a clicker for the classroom. Socrative was used during three sessions with the same group of first-year University students at a Faculty of Education. One of these sessions—a review lesson—was gamified, whereas the other two—a collaborative reading activity seminar, and a lecture—were not. Ad-hoc questionnaires were distributed after each of them. Results suggest that students welcome the use of clickers and that combining them with gamification strategies may increase students’ perceived satisfaction. The experiences described in this paper show how Socrative is an effective means of providing formative feedback and may actually save time during lessons. Full article

The Norwegian Museum of Science and Technology has developed a learning concept for school classes in science centres named “learning trails”. In this concept, groups of students perform a series of thematically related experiments with installations in the science centre. The learning trails are designed to support the generic learning outcomes for science centre visits. We argue that the previously developed Engagement Profile can be used to translate exhibit properties into both media forms and generic learning outcomes for such learning concepts. Further, we implemented the learning trails in two modes: one mode used paper-based content to guide the students, while the other mode supported the use of tablet PCs where engaging content is triggered when the students approach the location of an experiment in the learning trail. We studied the engagement factors of the learning trails and observed how school classes use these. In a study with 113 students from lower secondary school (age 16), they answered short questionnaires that were integrated into the implementation of the learning trails. While the concept of the learning trails was evaluated positively, we could not find significant differences in how engaging the two implemented modes were. Full article

Simulators have been traditionally used for centuries during medical gestures training. Nowadays, mechatronic technologies have opened the way to more evolved solutions enabling objective assessment and dedicated pedagogic scenarios. Trainees can now practice in virtual environments representing various kind of patient and body parts including physio-pathologies issues. Gestures, to be mastered, vary according to each medical specialty (e.g., ultrasound probe orientations, or forceps installation during assisted delivery). Hence, medical students need kinesthetic feedback in order to significantly improve their learning capabilities. Gesture simulators require haptic devices with variable stiffness actuators. Existing solutions do not always fit the requirements because of their significant size. Contrary to electric actuators, pneumatic technology is low-cost, available off-the-shelf and offers a better mass–power ratio. However, it presents two main drawbacks: nonlinear dynamics and need for a compressed air supply. During the last decade, we have developed several haptic solutions based on pneumatic actuation (e.g., birth simulator, epidural needle insertion simulator) and, recently, in a joint venture with Prisme laboratory, a pneumatic probe master device for remote ultrasonography. This paper recalls literature scientific approaches on pneumatic actuation developed in the medical context and illustrated with the aforementioned applications to highlight the benefits. Full article

Based on analyzing verbal and nonverbal features of small group conversations in a task-based scenario, this work focuses on automatic detection of group member perceptions about how well they are making use of available information, and whether they are experiencing information overload. Both the verbal and nonverbal features are derived from graph-based social network representations of the group interaction. For the task of predicting the information use ratings, a predictive model using random forests with verbal and nonverbal features significantly outperforms baselines in which the mean or median values of the training data are predicted, as well as significantly outperforming a linear regression baseline. For the task of predicting information overload ratings, the multimodal random forests model again outperforms all other models, including significant improvement over linear regression and gradient boosting models. However, on that task the best model is not significantly better than the mean and median baselines. For both tasks, we analyze performance using the full multimodal feature set versus using only linguistic features or only turn-taking features. While utilizing the full feature set yields the best performance in terms of mean squared error (MSE), there are no statistically significant differences, and using only linguistic features gives comparable performance. We provide a detailed analysis of the individual features that are most useful for each task. Beyond the immediate prediction tasks, our more general goal is to represent conversational interaction in such a way that yields a small number of features capturing the group interaction in an easily interpretable manner. The proposed approach is relevant to many other group prediction tasks as well, and is distinct from both classical natural language processing (NLP) as well as more current deep learning/artificial neural network approaches. Full article

In this study, we define and test measures that capture aspects of collaboration in interaction within groups of three participants performing a task. The measures are constructed upon turn-taking and lexical features from a corpus of triadic task-based interactions, as well as upon demographic features, and personality, dominance, and satisfaction assessments related to the corpus participants. Those quantities were tested for significant effects and correlations they have with each other. The findings indicate that determinants of collaboration are located in measures quantifying the differences among dialogue participants in conversational mechanisms employed, such as number and frequency of words contributed, lexical repetitions, conversational dominance, and in psychological and sentiment variables, i.e., the participants’ personality traits and expression of satisfaction. Full article