Multimodal Technol. Interact., Volume 2, Issue 2 (June 2018) – 23 articles

The use of augmented reality (AR) in cultural heritage (CH) applications opens a whole set of possibilities, including the virtual transformation of CH elements. This paper presents TinajAR, a mirror-based AR application designed to serve both as an edutainment application in the field of CH and also as an artistic expression. As an edutainment application, TinajAR features a multi-marker video-based AR application designed to show virtual ceramic pieces and explain the pottery process through virtual avatars. As an artistic expression, TinajAR seeks to reinterpret an ancient type of cellar called calado, which was used in the past for storing wine in northern Spain. The reinterpretation consists in giving a different but meaningful use to the space. TinajAR was used by around 1800 people during a ceramics exhibition in La Rioja, Spain and was assessed at the satisfaction level with 56 users by means of a system usability scale, giving very satisfactory results. Full article

A multitude of challenges comes into play when attempting to design (and evaluate) an interactive digital storytelling experience for use by visitors in a museum. This paper reports on the evaluation of the prototype mobile-based storytelling “guides” designed, developed and deployed as part of a research project at the Acropolis Museum in Athens, Greece. Experiences designed for different visitor profiles were evaluated several times throughout the iterative design process, in a number of on-site studies, and with more than 180 museum visitors of all ages (with this paper reporting on two studies conducted with a total of 53 users visiting individually or in pairs). The evaluation methods included ethnography (i.e., observation of visitors in the Museum’s galleries), pre- and post-experience in-depth interviews and questionnaires to measure the Users’ Experience (UX), as well as data logging. The analysis of the data focused on themes representing components of the experiences, such as interactive story plot and narration, staging and way-finding in the physical space, personalization and social interaction. Our findings confirmed that understanding UX and what makes it effective or not in the rich context of a cultural setting is a complex endeavor. The paper discusses our findings and proposes relevant recommendations for the design of digital experiences for cultural, educational, and recreational purposes. Full article

A key factor influencing the effectiveness of a user interface is the usability resulting from its design, and the overall experience generated while using it, through any kind of device. The two main design trends that prevail in the field of user interface design is skeuomorphism and flat design. Skeuomorphism was used in UI design long before flat design and it is built upon the notion of metaphors and affordances. Flat design is the main design trend used in most UIs today and, unlike skeuomorphic design, it is considered as a way to explore the digital medium without trying to reproduce the appearance of the physical world. This paper investigates how users perceive the two design approaches at the level of icon design (in terms of icon recognizability, recall and effectiveness) based on series of experiments and on data collected via a Tobii eye tracker. Moreover, the paper poses the question whether users perceive an overall flat design as more aesthetically attractive or more usable than a skeuomorphic equivalent. All tested hypotheses regarding potential effect of design approach on icon recognizability, task completion time, or number of errors were rejected but users perceived flat design as more usable. The last issue considered was how users respond to functionally equivalent flat and skeuomorphic variations of websites when given specific tasks to execute. Most tested hypotheses that website design affects task completion durations, user expected and experienced difficulty, or SUS (System Usability Scale) and meCUE questionnaires scores were rejected but there was a correlation between skeuomorphic design and increased experienced difficulty, as well as design type and SUS scores but not in both websites examined. Full article

As technologies diversify and become embedded in everyday lives, the technologies we expose to animals, and the new technologies being developed for animals within the field of Animal Computer Interaction (ACI) are increasing. As we approach seven years since the ACI manifesto, which grounded the field within Human Computer Interaction and Computer Science, this thematic literature review looks at the technologies developed for (non-human) animals. Technologies that are analysed include tangible and physical, haptic and wearable, olfactory, screen technology and tracking systems. The conversation explores what exactly ACI is whilst questioning what it means to be animal by considering the impact and loop between machine and animal interactivity. The findings of this review are expected to form the first grounding foundation of ACI technologies informing future research in animal computing as well as suggesting future areas for exploration. Full article

Food-tracking apps constitute a major category of the thousands of food-related apps now available. They are promoted as helping users monitor and measure their food consumption to improve their health or to lose weight. In this article, I present six vignettes drawn from interviews with Australian women about their use and non-use of food-tracking apps. The vignettes provide detailed insights into the experiences of these women and their broader sociocultural and biographical contexts. The analysis is based on feminist materialism theoretical perspectives, seeking to identify the relational connections, affective forces, and agential capacities generated in and through the human-app assemblage. The vignettes reveal that affective forces related to the desire to control and manage the body and conform to norms and ideals about good health and body weight inspire people to try food-tracking apps. However, the agential capacities promised by app developers may not be generated even when people have committed hope and effort in using the app. Frustration, disappointment, the fear of becoming too controlled, and annoyance or guilt evoked by the demands of the app can be barriers to continued and successful use. Sociocultural and biographical contexts and relational connections are also central to the capacities of human-app assemblages. Women’s ambivalences about using apps as part of efforts to control their body weight are sited within their struggles to conform to accepted ideals of physical appearance but also their awareness that these struggles may be too limiting of their agency. This analysis, therefore, draws attention to what a body can and cannot do as it comes together with food tracking apps. Full article

Silk was a major factor for progress in Europe, mostly along the Western Silk Road’s network of production and market centers. The silk trade also allowed for the exchange of ideas and innovations, having impacts at economic, technical, functional, cultural and symbolic levels. However, silk has today become a seriously endangered heritage. Although many European specialized museums are devoted to its preservation, they usually lack the size and resources to take advantage of state-of-the-art digital technologies. The aim of this paper is twofold; firstly, we introduce SILKNOW, an interdisciplinary project that has been recently funded by the H2020 Programme of the European Union in order to preserve and promote the heritage of silk textiles; secondly, we introduce a set of interactive tools related to the project. Full article

Cow lameness is a common manifestation in dairy cattle that causes severe health and life quality issues to cows, including pain and a reduction in their life expectancy. In our previous work, we introduced an algorithmic approach to automatically detect anomalies in the walking pattern of cows using a wearable motion sensor. In this article, we provide further insights into a system for automatic lameness detection, including the decisions we made when designing the system, the requirements that drove these decisions and provide further insight into the algorithmic approach. Results from a controlled experiment we conducted indicate that our approach can detect deviations in cows’ gait with an accuracy of 91.1%. The information provided by our system can be useful to spot lameness-related diseases automatically and alarm veterinarians. Full article

Historic buildings are representations of cultural systems throughout time. Thus, it is important to shed light on any given historic building’s chronological information by means of restitution, that is, the descriptions of different (re)compositions of a building for different historical states. However, the representation of information beyond the geometric, spatial, and physical characteristics of a historic building carries a crucial importance in the holistic understanding of historical changes. This requirement consists of representation, archival, assessment, and management aspects. Nevertheless, covering all these aspects is still quite challenging. Thus, three-dimensional (3D) visualization in digital platforms is considered, which indeed seems to be the most desirable way today. This article proposes a methodology through adopting an approach based on the Geographical Information System (GIS) and further develops a digital “container” as a 3D digital model for the archiving of a building’s chronological information. This does not only cover the geometric and spatial aspects, but also the historical resources and their reliability. The paper concludes that the developed prototype may lessen the future investigation effort of heritage specialists by making the chronological information of a historical building more integrated and coherent, through increased readability, accessibility, and visibility. This, in turn, supports the planning of restoration projects, as well as provides a static and comprehensive archive. Full article

The range of modes through which the new conditions of the postdigital society is leading to a redefinition of the processes of assigning value, and the values themselves, which have hitherto prevailed in the comprehension of cultural heritage, are diverse and broad. Within this framework of critical inquiry, this paper discusses the mechanisms of canon-formation in the context of the web as the new laboratory of cultural production. It is argued that the main dynamics observed can be elucidated under the form of a triad: hypercanonization, socialdecanonization, and transcanonization. These three processes operate simultaneously interlaced and unfold in dialectical tension between the rise of the new (practices, actors, values, ideas), and the maintenance of the old (those structures that already exist). This paper delves into the paths through which such interlace dynamics and tension might reshape the principles by which canonicity develops, as well as poses open questions about the challenges facing us, which should be discussed in further studies and approaches to the problem. Full article

Deafblind persons’ perception and experiences are based on their residual auditive and visual senses, and touch. Their haptic exploration, through movements and orientation towards objects give blind persons direct, independent experience. Few studies explore the aesthetic experiences and appreciation of artefacts of deafblind people using cochlear implant (CI) technology, and how they interpret and express their perceived aesthetic experience through another sensory modality. While speech recognition is studied extensively in this area, the aspect of auditive descriptions made by CI users are a less-studied domain. This present research intervention describes and analyses five different deafblind people sharing their interpretation of five statues vocally, using sounds and written descriptions based on their haptic explorations. The participants found new and multimodal ways of expressing their experiences, as well as re-experiencing them through technological aids. We also found that the CI users modify technology to better suit their personal needs. We conclude that CI technology in combination with self-made sound descriptions enhance memorization of haptic art experiences that can be re-called by the recording of the sound descriptions. This research expands the idea of auditive descriptions, and encourages user-produced descriptions as artistic supports to traditional linguistic, audio descriptions. These can be used to create personal auditive–haptic memory collections similar to how sighted create photo albums. Full article

This paper studies the use of eye tracking in a First-Person Shooter (FPS) game as a mechanism to: (1) control the attention of the player’s avatar according to the attention deployed by the player; and (2) guide the gameplay and game’s procedural content generation, accordingly. This results in a more natural use of eye tracking in comparison to a use in which the eye tracker directly substitutes control input devices, such as gamepads. The study was conducted on a custom endless runner FPS, Zombie Runner, using an affordable eye tracker. Evaluation sessions showed that the proposed use of eye tracking provides a more challenging and immersive experience to the player, when compared to its absence. However, a strong correlation between eye tracker calibration problems and player’s overall experience was found. This means that eye tracking technology still needs to evolve but also means that once technology gets mature enough players are expected to benefit greatly from the inclusion of eye tracking in their gaming experience. Full article

Embodied Interaction faces designers with the challenge of thinking about users and interaction from different viewpoints with respect to traditional technologies. This task is even more complex when designing non-task oriented systems. We propose a framework to guide researchers in thinking and designing non-task-oriented Embodied Interaction Environments or, in other words, embodied experiences that users can enjoy for its own sake and not as means for accomplishing a task or achieving an extrinsic goal. The framework is grounded on experience-centered design approaches and will present four qualities ((1) Spatial, Corporeal and Material Consistency, (2) Contingent Enhancement, (3) Mindful Embodied Engagement and (4) Situated Reflexivity) aimed at providing critical lenses, strategies and techniques to guide the design and research processes. Finally, we will discuss how designers can implement the proposed framework in different stages of the design process and paths for future research. Full article

Many projects concerning the protection, conservation, restoration, and dissemination of cultural heritage are being carried out around the world due to its growing interest as a driving force of socio-economic development. The existence of reliable, digital three-dimensional (3D) models that allow for the planning and management of these projects in a remote and decentralized way is currently a growing necessity. There are many software tools to perform the modeling and complete three-dimensional documentation of the intervened monuments. However, the Architecture, Engineering and Construction (AEC) sector has adopted the Building Information Modeling (BIM) standard over the last few decades due to the progress that has been made in its qualities and capabilities. The complex modeling of cultural heritage through commercial BIM software leads to the consideration of the concept of Heritage BIM (H-BIM), which pursues the modeling of architectural elements, according to artistic, historical, and constructive typologies. In addition, H-BIM is considered to be an emerging technology that enables us to understand, document, advertize, and virtually reconstruct the built heritage. This article is a review of the existing literature on H-BIM and its effective implementation in the cultural heritage sector, exploring the effectiveness and the usefulness of the different methodologies that were developed to model families of elements of interest. Full article

In recent years, immersive VR had a great boost in terms of adoption and research perspectives, especially those regarding the serious gaming universe. Within the cultural heritage field, virtual re-contextualization of items is a crucial task to be accomplished by individuals to understand a 3D reconstructed environment as a whole and to assign a meaning and a value to a specific cultural object. Immersive VR and consumer HMDs still present several issues related to motion sickness and locomotion: the interest in real-walking techniques outperforming other locomotion methods is growing year by year, although limited by physical constraints, higher costs, or current technology. In this work, we propose a novel game model (μVR) that combines real-walking techniques and an adaptive, game-driven, multi-scale progression to craft immersive re-contextualization applications. The presented model aims to minimize motion sickness while fully exploiting the physical tracked area and augmenting the understanding of what the user is experiencing at different world scales. We define and formalize the μVR model and its components mathematically for the sake of reproducibility, then we present results from a pilot test planned to validate the model on real users. Results assure the usability and effectiveness of VR model even if further implementation needs to be done. Full article

Both Internet and offline video gaming have become a normal aspect of child development, with estimates of children playing video games ranging from 90% to 97%. Research on problematic video gaming (PVG) has grown substantially in the last decade. Much of that research has focused on community samples, while research on clinically referred children and youth is lacking. The present study includes 5820 clinically referred children and youth across 44 mental health agencies, assessed using the interRAI Child and Youth Mental Health Assessment. Logistic regression analyses revealed that older age, male sex, extreme shyness, internalizing symptoms, externalizing symptoms, and poor relational strengths are all significant predictors of problematic video gaming (PVG). Further analyses suggested that, out of the internalizing symptoms, anhedonia was predictive of PVG in both males and females, but depressive symptoms and anxiety were not predictive of PVG when controlling for other variables in the model. Moreover, proactive aggression and extreme shyness were predictive of PVG in males, but not in females. The implications of these findings are discussed. Full article

The Internet of Things (IoT) is inspired by network interconnectedness of humans, objects, and cloud services to facilitate new use cases and new business models across multiple enterprise domains including healthcare. This creates the need for continuous data streaming in IoT architectures which are mainly designed following the broadcast model. The model facilitates IoT devices to sense and deliver information to other nodes (e.g., cloud, physical objects, etc.) that are interested in the information. However, this is a recipe for privacy breaches since sensitive data, such as personal vitals from wearables, can be delivered to undesired sniffing nodes. In order to protect users’ privacy and manufacturers’ IP, as well as detecting and blocking malicious activity, this research paper proposes privacy-oriented IoT architecture following the provenance technique. This ensures that the IoT data will only be delivered to the nodes that subscribe to receive the information. Using the provenance technique to ensure high transparency, the work is able to provide trace routes for digital audit trail. Several empirical evaluations are conducted in a real-world wearable IoT ecosystem to prove the superiority of the proposed work. Full article

Over the last few years, Augmented Reality (AR) technology has quickly evolved and today there is commercial software and hardware that allow the creation of AR experiences. However, it might still be cumbersome to create rich AR experiences without a deep knowledge of the technology. Artists have collaborated with IT experts during the last few years in order to enhance artistic pieces embedding AR technology, leading to the emergence of interesting collaborations between artists and engineers, computer scientists, architects, etc. The resulting works can be referred as AR Art. However, the interdisciplinary work behind these collaborations is usually not addressed. The aim of this paper is to review some of the reported AR Art work since the AR term was first coined in 1990, focusing on collaborations between different disciplines, especially during the early years, where art and technology became one. Additionally, the author reports her own experience in contributing to AR Art from an interdisciplinary perspective. Full article

In this paper, we present a two-part study. The first part was a cultural appreciation study. Through this study, we explored the specific approach of cross-cultural aesthetic appreciation and mapped out the potential insights for a prototype design. In the second part, we carried out a design-led study. We designed a tablet application and conducted focus group studies to explore the interactive technology that assists in the support of cross-cultural audiences’ aesthetic appreciation and engagement of traditional Chinese painting. Based on these findings, we went on to further explore an approach of interactive engagement which is specific to supporting cross-cultural appreciation, while also reflecting upon the interactive design suggestions for the development of aesthetic appreciation to offer various transferable insights to the Human–Computer Interaction (HCI) community. Full article

‘Embodiment’ need not focus on isolated individuals or group interactions. This article articulates the potential for designs that prompt participants to bring relationships with other people to mind. These can be fleeting relationships between participants and unknown others, or remembered relationships with romantic partners, family members, or close friends who are not physically co-present or digitally represented. In either case, it is possible to generate affective responses that profoundly shape participants’ emotional and physical reactions to, and co-creation of, the designed interaction. This article presents existing practices of immersive theatre to frame our exploration of this phenomenon. It introduces three theories—mise-en-sensibilité, narcissistic spectatorship and affect—through which we illuminate both the internally felt and the externally designed experience, whether or not it is explicitly framed as theatrical performance. Through analysis of two immersive performances (one-on-one interactions that could easily be understood in terms of experience design) and two designs of our own, we argue that the affect generated by personal relationships in immersive experiences can both shape and drive participation, and we offer a three-point guideline by which one can design for the affective consequences of bringing relationships to mind. Full article

The use of virtual elements for developing new service prototyping environments and more realistic simulations has been suggested as a way to optimise the service prototyping process. This work examines the application of virtual reality (VR) in prototyping service journeys and it hypothesises that VR can recreate service journeys in a highly immersive, agile, and inexpensive manner, thus allowing users to have a representative service experience and enabling service designers to extract high-quality user feedback. To that end, a new service prototyping method, called VR service walkthrough, is presented and evaluated through an empirical comparative study. A VR service walkthrough is a virtual simulation of a service journey, representing how the service unfolds over space and time. A comparative study between the VR service walkthrough method and an adapted service walkthrough method evaluates the application of both methods using a location-based audio tour guide service as a case study. Two user groups (each with 21 users) were used to evaluate both methods based on two factors: the user experience they offered and the subjective meaningfulness and quality of feedback they produced. Results show that the VR service walkthrough method gave a performance similar to that of the service walkthrough method. It was also able to communicate the service concept in an immersive way and foster constructive feedback. Full article

With the emerging trend in Human-Computer Interaction (HCI) shifting focus from usability to facilitating meaningful experiences, the notion of embodied cognition provides designers and researchers with valuable insight into how the body–mind interplay can influence meaning-making during embodied experiences. This paper presents an approach to designing embodied interactions with ambient media. Building on theories of embodied cognition and cognitive semantics, we developed our approach by conducting a series of studies, including an interpretive case analysis, empirical research into audience experience and design ideations, as well as designerly reflections on design. Our findings showed that an embodied approach is applicable for designing meaningful interactions, by coupling bodily engagement with metaphorical meanings. Design implications and future work are also presented. Full article

The number of webpages in the Internet has increased tremendously over the last two decades however only a part of it is indexed by various search engines. This small portion is the indexable web of the Internet and can be usually reachable from a Search Engine. Search engines play a big role in making the World Wide Web accessible to the end user, and how much of the World Wide Web is accessible on the size of the search engine’s index. Researchers have proposed several ways to estimate this size of the indexable web using search engines with and without privileged access to the search engine’s database. Our report provides a summary of methods used in the last two decades to estimate the size of the World Wide Web, as well as describe how this knowledge can be used in other aspects/tasks concerning the World Wide Web. Full article

In this paper we present the first results of a pilot experiment in the interpretation of multimodal observations of human experts engaged in solving challenging chess problems. Our goal is to investigate the extent to which observations of eye-gaze, posture, emotion and other physiological signals can be used to model the cognitive state of subjects, and to explore the integration of multiple sensor modalities to improve the reliability of detection of human displays of awareness and emotion. Domains of application for such cognitive model based systems are, for instance, healthy autonomous ageing or automated training systems. Abilities to observe cognitive abilities and emotional reactions can allow artificial systems to provide appropriate assistance in such contexts. We observed chess players engaged in problems of increasing difficulty while recording their behavior. Such recordings can be used to estimate a participant’s awareness of the current situation and to predict ability to respond effectively to challenging situations. Feature selection has been performed to construct a multimodal classifier relying on the most relevant features from each modality. Initial results indicate that eye-gaze, body posture and emotion are good features to capture such awareness. This experiment also validates the use of our equipment as a general and reproducible tool for the study of participants engaged in screen-based interaction and/or problem solving. Full article