*Article* **A Semantic Mixed Reality Framework for Shared Cultural Experiences Ecosystems**

**Costas Vassilakis 1,\*, Konstantinos Kotis 2, Dimitris Spiliotopoulos 1, Dionisis Margaris 3, Vlasios Kasapakis 2, Christos-Nikolaos Anagnostopoulos 2, Georgios Santipantakis 4, George A. Vouros 4, Theodore Kotsilieris 5, Volha Petukhova 6, Andrei Malchanau 6, Ioanna Lykourentzou 7, Kaj Michael Helin 8, Artem Revenko 9, Nenad Gligoric <sup>10</sup> and Boris Pokric <sup>11</sup>**


Received: 19 March 2020; Accepted: 14 April 2020; Published: 20 April 2020

**Abstract:** This paper presents SemMR, a semantic framework for modelling interactions between human and non-human entities and managing reusable and optimized cultural experiences, towards a shared cultural experience ecosystem that might seamlessly accommodate mixed reality experiences. The SemMR framework synthesizes and integrates interaction data into semantically rich reusable structures and facilitates the interaction between different types of entities in a symbiotic way, within a large, virtual, and fully experiential open world, promoting experience sharing at the user level, as well as data/application interoperability and low-effort implementation at the software engineering level. The proposed semantic framework introduces methods for low-effort implementation and the deployment of open and reusable cultural content, applications, and tools, around the concept of cultural experience as a semantic trajectory or simply, experience as a trajectory (eX-trajectory). The methods facilitate the collection and analysis of data regarding the behaviour of users and their interaction with other users and the environment, towards optimizing eX-trajectories via reconfiguration. The SemMR framework supports the synthesis, enhancement, and recommendation of highly complex reconfigurable eX-trajectories, while using semantically integrated disparate and heterogeneous related data. Overall, this work aims to semantically manage interactions and experiences through the eX-trajectory concept, towards delivering enriched cultural experiences.

**Keywords:** intelligent interaction; semantics; usability; mixed reality; cultural experience

#### **1. Introduction**

Cultural applications are increasingly used for the development and delivery of cultural experiences to users. To this end, research and development activities have targeted the design and implementation of applications and related systems to support all of the phases of the creation and operation of cultural applications, including content creation and organization, application development [1], application operation within venues [2,3], and in broad Internet of Things (IoT) environments [4–6].

Insofar, however, each cultural application is designed, implemented, and deployed separately, increasing the associated development costs (content development, code creation and testing, infrastructure deployment, and maintenance), while, at the same time, limiting the opportunities for sharing and reusing cultural experiences to the level of recommending isolated points of interests (PoIs) or coarse-grained routes [7–9]. The impact of these challenges is more pronounced in augmented, virtual, and mixed reality (AR/VR/MR) systems, for which content development, code implementation and deployment infrastructure are more complex and demanding. In addition, the range of the required hardware and software systems [10] poses further data integration and reuse issues.

The SemMR semantic framework proposes an integrated multi-technology and multi-entity approach towards addressing these challenges and supports current, as well as future, interactive technologies that are of low effort and cost, being accessible to all businesses. In particular, SemMR is inclusive towards technologies that are based on MR (including AR/VR). The main ingredients of the SemMR approach are the use of semantic technology for the utilization/integration of data and information discovered on Web sources, the cloud of Linked Open Data (LOD), and the IoT. Through this approach, the SemMR framework promotes experience sharing at user level, as well as data/application interoperability and low-effort implementation at the software engineering level. The framework is based on the notion of a shared cultural experience ecosystem (SCEE) in order to enable and support the management of enhanced user experience in the cultural domain. IoT is the key factor of future interaction [11] and the prominent source of the immense amounts of semantically linked data/information. In order to do so, the user behaviour must drive and be driven by the semantically integrated data/information/knowledge, thus creating a new world of seamless and immersive MR interaction between the real-world entities and the virtual entities. When specialized in the domain of MR, the SCEE ecosystem is denoted in this paper as MR-SCEE.

SemMR is based on two main concepts; the concept of the shared cultural experience ecosystem (SCEE) and the concept of the cultural experience as a semantic trajectory (eX-trajectory):


interaction timeline, those experiences may dynamically intersect and interchange, resulting to unseen, but relevant, eX-trajectories.

SemMR is a semantic framework for modelling interactions between human and non-human entities and managing reusable and optimized eX-trajectories. It comprises of methods for: (1) creating and managing open, reusable, and optimized eX-trajectory content, applications and tools, (2) tracking, monitoring, and analysing user behaviour during interactions with the environment and with other entities, (3) optimizing (via reconfiguration) eX-trajectories at runtime or at development time, and (4) synthesizing eX-trajectories into new, but still reconfigurable, eX-trajectories, which are augmented by exploiting semantically integrated related data/information that is sourced from diverse and heterogeneous resources.

The contribution of this paper is outlined in the following three points:


The structure of the paper is outlined, as follows: Section 2 presents the related work. Section 3 details the proposed eX-trajectory concept. Section 4 introduces the SCEE, while Section 5 presents the system architecture for SemMR. Section 6 presents the instantiated implementation of SemMR for cultural experiences and Section 7 presents the user behaviour modelling. Section 8 presents the evaluation on the scalability of the proposed framework for a sizable cultural site, an archaeological museum, and, finally, Section 9 discusses the proposed framework and outlines future work.

#### **2. Related Work**

As cultural application development and use proliferates, researchers have developed a number of approaches that underpin and facilitate different parts of the cultural application lifecycle. Amato et al. [12] present SNOPS, a system that consolidates participatory sensing, IoT platforms, and recommendation systems under an instantiation of the Service-Oriented Architecture, targeting the collection of information from data sources, which are then exploited for the formulation of context-aware recommendations for users. The context of the recommendations is represented as an upper-level ontology, which encompasses classes for modelling users, objects/places of interest, time intervals, activities (either explicitly modelled or deduced), environmental conditions, and devices (both user access devices and sensors). Chianese et al. [2] describe the design and implementation of a system that is able to leverage cultural spaces into smart cultural environments following the concept of Single Smart Spaces (S3), which result in enhanced user experience. The system that is proposed in [2] retrieves (a) data from sensors that perceive the real world, (b) information from structured and unstructured data sources, and (c) knowledge from users moving into the smart cultural environment, and processes the input to deliver knowledge to users to facilitate a number of tasks, including navigation and information finding. The concept of S<sup>3</sup> is also adopted in [12,13], where a context-aware framework for cultural heritage applications is presented. The framework presented in [12,13] captures contextual information under a Context Dimension Tree, which represents six dimensions of the contextual information: users interacting with the system; items within the smart space; activities performed on items; situations within which activities are carried out; locations of activities; and, times when activities were performed. From this information, the system continuously learns usage patterns and propagates the resulting knowledge to users.

The exhiSTORY approach [14,15] integrates IoT and semantic technologies, together with clustering and personalization techniques to leverage exhibits within cultural venues to smart, self-organizing exhibits that cooperate with each other and provide visitors with comprehensible, rich, diverse, personalized, and highly stimulating experiences. In more detail, within the exhiSTORY approach, each exhibit maintains an amount of self-descriptive data and semantic information, and communicates with both (a) neighbouring exhibits and (b) the smart space, to create multiple meaningful collections of items. Each collection tells a story about a specific subject. Subsequently, personalization technologies are employed to select the most prominent stories to be told to visitors, after consulting their profiles.

In the following paragraphs, we elaborate on the research work and technologies related to the main axes of the SemMR framework, namely (i) semantic data management, (ii) virtual entities and IoT, (iii) user profiling, and (iv) mixed reality.

#### *2.1. Semantic Data Management: Link Discovery and Data Integration*

Link discovery (LD) is the process of identifying relations (links) between data/information objects of different provenance (i.e., that that have been retrieved from different sources). The identified relations are then used to support several tasks, including data/information integration and deduplication. In the case of spatial datasets, the goal of LD is to identify pairs of spatial objects for which a given set of relations is satisfied. Existing works in this area have mainly targeted the discovery of topological relations (intersects, contains, crosses, meets, etc.) between spatial objects, while the recent work on maskLink [16] has been employed for discovering proximity relations, as well as in trajectory reconstruction and semantic enriching of trajectory segments.

RADON [17] is a recent topological relation discovery approach for relations between data sources of areas and it can discover efficiently multiple relations while using space tiling. Smeros and Koubarakis [18] studied link discovery on spatiotemporal RDF data by examining several topological relations that are defined on polygons. Finally, the maskLink approach [16,19] tackles both topological and proximity relations. It has been implemented in a flexible framework, which includes features, such as:


Going beyond the state-of-the-art methods in LD and data integration, SemMR develops LD algorithms for discovering spatiotemporal relations (as well as other well-defined semantic relations) between the eX-trajectories, supporting the meaningful exploitation of similar and related trajectories.

#### *2.2. Virtual Entities: IoT Management and Trustful Interactions*

The interaction with objects in MR worlds requires sensing from physical space, or even sensing of user parameters to be able to provide high value user experience. To achieve this, a full IoT infrastructure for collecting important data for VR/AR space reconstruction, as well as device virtualization, device management, and trustful interaction must be provided. In device virtualization, there are several commercial IoT-related products that aim to aggregate all of the data that IoT can generate in cloud storage and then expose them to developers through RESTful interfaces and libraries for enhanced service creation. However, these initiatives remain bound to an information-centric view, where the main value of the things is more on the information they can generate and less on the possibility to include augmented AR/VR interaction with an object and between users, offering services and actuation on it.

In IoT management paradigm, trust-related issues need also be addressed, for instance, ways to manage trust between entities without the existence of a central authority. These issues may be addressed while using clear and simple semantics. As trust management mechanisms have been widely studied in various research fields [21], it is now commonly accepted that the seamless integration of trust management mechanisms in IoT is needed [22,23]. The recommendation and standardization of

a well-defined trust negotiation language supporting the semantic interoperability of IoT context is a challenging and open IoT-trust modelling and management topic [24,25].

SemMR delivers an integrated framework for: (a) capturing and virtualizing human and non-human (mobile and smart) entities (users, smart rooms, smart phones, smart bands, smart tags, etc.) and their interconnections, supporting their automated identification and recognition, and their open (re)use by cultural experience authoring environments, and (b) modelling and computing trust for the interaction of the virtualized entities, based on principles, such as friendship, ownership, collaboration, as well as on contextual information, such as environmental conditions that are sensed by the smart tag.

#### *2.3. User Profiling*

SemMR uses profiling methods to adapt user interfaces and interactions to the specific characteristics of users, particularly their age, gender and cultural background, their physical and cognitive abilities, their level of engagement, and their preferences. Consequently, it is necessary to model the user profiles at different levels: their intrinsic characteristics (physical characteristics, identity, age, disabilities, behaviour, emotional state, skills, etc.), their physical environment (location), their social environment (job position, tasks), and their needs and preferences. User profiles can be explicitly built by inquiring the users for direct information, or implicitly by deducing their profile from their interaction with the system. Implicit and explicit profiles are complementary aspects. It is important to keep in mind that user profiles change over time and that, in that context, a dynamic user profile is fundamental for successful personalisation. Other works utilise the user personality traits to deduce work leadership profiles and construct harmonious and effective teams [26].

User profile creation and maintenance can be supported through a multitude of mainstream methods. Fine-grained tracking of facial expressions and body movement on the visible spectrum can be achieved using hardware, such as Intel®RealSense, 3D Kinect and eye-trackers (e.g., Tobii Glasses). Moreover, biometric signals can be recorded and tracked while using sensors, such as NeXus EXG and Blood Volume Pulse. These allow for multimodal interaction, a very natural social form of interaction that has been shown to improve human learning and the treatment of medical conditions. Learning and user experience and acceptance may be enhanced by immersive interactive environments [27]. Learning might be reinforced by multi-sensory approaches that may be used for the personalisation of the assessment and reflection phases for improved user experience.

User profiles may be associated with, or abstracted to, user behaviour models. There are several paradigms for user behaviour modelling and action planning for domains of varying complexity with most prominent concerning Finite State Machines (FSM), Agent-Based Modelling approaches, Social Force models and Activity-Based Models [28]. In FSM, each user action leads to a new state. Simple algebraic structures relate internal states to input and output sequences offering a general model of user behaviour. FSMs were successfully used to model human-robot interactions and dialogue behaviour [29,30]. Agent-based systems are developed for simulating (virtual) human behaviour in a variety of disciplines, from knowledge building in collaborative online communities, like wikis [31,32] to task assignment in crowd work environments [33–35] to the way people select which exhibits to see in the physical space of a museum [36]. Users are represented as intentional rational agents. An agent model includes perception, beliefs, desires, planning/reasoning, commitment, intentions, and acting, and represents a comprehensive model of user behaviour simulation.

Social or behavioural forces specify the degree of behavioural change (e.g., changes in acceleration or in direction), as reaction to external forces that are exerted by the environment or other agents. These forces have a stimulating or repelling effect on the motivation of humans to perform certain activities [37].

SemMR models adaptive user and system behaviour in dynamic non-sequential interactions. For this, cognitive models that produce detailed simulations of human (multi-)task performance are designed in order to implement simulated artificial agents to play a role in a multi-agent (multi-entity) setting.

#### *2.4. Mixed Reality*

MR refers to environments where real world and virtual world objects are presented together in a single display. The two most common methods for creating such MR environments are AR and Augmented Virtuality (AV). AV blends elements from the real world to the Virtual Environment (VE), while AR works by superimposing computer-generated objects upon the Real Environment (RE). Virtual Reality (VR) is an alternative approach that constructs and displays entirely synthetic worlds that may simulate the physical properties of the real world, where users can be totally immersed in [38,39]. In most AR applications, the RE is streamed through the camera feed of a device, such as a smartphone or a camera-equipped Head Mounted Display (HMD), with the virtual objects being superimposed on the RE by either using computer vision with fiducial markers, or sensors, to properly adjust their position and rotation. However, recently, marker-less AR received significant attention and it is now widely used in popular AR applications development platforms.

Nowadays, most of the popular VR and AV application development platforms utilize sophisticated sensors to support room-scale applications, allowing for hand presence in the virtual world and full body motion support [40], along with wireless support. Eye tracking is also exploited in some high-end HMD platforms to provide better experience in AV.

Wireless HMDs feature high quality inside-out tracking, allowing for developers to seamlessly blend real and virtual environment in AV and AR applications. Finally, state-of-the art technology enables brain activity and eye movement detection, which allows for user behaviour tracking for real time personalization and enhancement of user experience in MR applications.

SemMR enhances MR development systems/platforms by (a) integrating a graphical drag-n-drop code-free authoring environment for synthesizing open and reusable MR experiences, (b) recommendation methods for automatically suggesting related external data/info to be attached to MR content for enhancing it, and for automatically suggesting new eX-trajectories to support the reconfiguration of existing (towards optimization), (c) focusing on IoT to allow for seamless and 'live' interaction of interconnected trustworthy deployed entities (human and non-human ones), (d) properly understanding human behaviour and cognition while experiencing MR worlds, (e) semantically integrating external heterogeneous and disparate information in order to enrich the content of the MR experiences, improving their quality and, thus, the quality of the user experience, and (f) appropriate models and methods for the reuse of connections between virtual and real objects, in more than one MR world, enlarging, this way, the MR environment

#### **3. Experience as a Trajectory (eX-trajectory)**

A movement track represents the ability to capture the movement of an object or entity moving in a geographical space over some period of time. This temporal sequence of the spatiotemporal positions is represented as pairs of 'instant' and 'point'. Additional data (depending on the capabilities of the movement recording device) may also be recorded, e.g., the instant speed or stillness, acceleration, direction, and rotation. Such captured data are the raw data. In some applications, there is no interest in keeping and analysing continuous non-stop records of raw movement data. Instead, segments of interest may be selected, i.e., a specific movement track within a 'start' and 'stop' (Begin and End) point. Trajectories are the segments of an object movement track that are of interest for a given application. For instance, when considering an application that is required to track and analyse tourist movement and cultural activities in the city of Athens. For this (big) data recording example, the application identifies a trajectory for the whole track that is left by an individual tourist in Athens (e.g., 'inside Athens' trajectory), but also another trajectory tracking a specific daily cultural experience track of this individual (e.g., 'a tourist in Athens on a Sunday tour' or 'a tour in the Museum of Acropolis on Friday morning').

In some cases, the application processes require using contextual data or information to complement and augment raw data. For instance, to be able to interpret the trajectories of people in a city, additional information regarding the city (e.g., the map or the PoIs of the city) is required. Spatiotemporal data (coordinates) can be reverse geocoded and transformed into names/identities of PoIs (e.g., monuments, parks, or shop names) or names of streets and squares. For example, information about events of any nature, from football games to concerts and protest marches, enables traffic monitoring applications to distinguish among normal and exceptional traffic conditions, leveraging the interpretation of spatiotemporal data (positions). Adding information to raw trajectories is called semantic enrichment of trajectories. Such enrichment requires the process of complementing existing data with additional data/information, i.e., annotations. Additional data/information is connected either to parts of (segments, points) of a trajectory, or to the trajectory as a whole. In this context, a semantic trajectory is a trajectory that has been augmented with annotations that add context. In a tourist example, recording the movement pattern of a museum visitor (e.g., ant, grasshopper) [41] is a trajectory-level semantic annotation. On the other hand, marking the person's presence at a location, as a visit to a temporary modern art exhibition, is a semantic annotation at the location level.

## *3.1. eX-Trajectory Modelling*

Trajectories are widely defined as the segments of the object's movement track that are of interest for a given application. Semantic trajectories are trajectories that have been enhanced with semantic annotations and/or one or more complementary segmentations. One might superimpose a structure of homogeneous segments that are meaningful for the particular application to enhance the knowledge captured by trajectories. These homogeneous segments are called episodes. Episodes are defined as a maximal subsequence of a trajectory, such that all of its spatiotemporal positions comply with a given predicate [42].

Existing semantic trajectory modelling and representation approaches impose limitations on the structure or the elements of trajectories. More specifically, they either:


For example, in some works, semantic trajectories are sequences of sub-trajectories [43], while, in others, they are sequences of episodes [46]. For the datAcron ontology [48], the representation of trajectories at multiple and interlinked levels of analysis is supported. In related works, a rich set of constructs for the representation of semantic trajectories is presented as sequences of episodes, each being associated with raw trajectory data and (optionally) with a spatiotemporal model of movement, although without a fine association between raw movement data and abstract models of movements [46]. In Bogorny et al. [43], semantic trajectories are defined as lists of sub-trajectories, and each sub-trajectory as a list of points. Events and episodes are connected to specific resources at specific levels of analysis: events that are mostly related to the environment are only connected to points [43], while episodes concern things occurring at trajectory-level and they can be linked to specific models of movement [46].

#### *3.2. eX-Trajectory Management and Analytics*

A trajectory behaviour is defined as a set of specific characteristics that can be used to identify a particular connection or link to a moving object or to a set of moving objects. The behaviour is

defined by a predicate that expresses whether a given trajectory (or a given set of trajectories) shows the corresponding behaviour [42]. For instance,


A trajectory can be characterized by several behaviours. For instance, a trajectory can show both a "Speeding" and a "Tourist" behaviour and simultaneously be part of a group of trajectories showing the "Meet" behaviour. For each behaviour, the predicate relies on different characteristics. The trajectories of tourists visiting a city may be analysed for (a) creating tourist profiles and recommending personalized itineraries and services, (b) the flow regulation of visitors and tourist either within cultural venues or at city level, etc. Processes that analyse trajectories to identify similarities and dissimilarities among them (including a feature-based comparison between trajectories), classify the trajectories into types based on their similarity, and extract the salient features that differentiate one trajectory group from another underpin all of this. A set of distinguishing features (called patterns or behaviours) forms a summary description of the group of trajectories.

Several systems for trajectory data management and analysis exist. SpatialHadoop [49] and Simba [50] enable distributed spatial analytics based on the MapReduce paradigm. Nevertheless, they do not exploit the characteristics of trajectory data for efficient data management and analytics. A cloud-based system by Bao et al. [51] and Elite [52] provide distributed solutions for big trajectory data. They utilize specific partitioning strategies in distributed environments in order to support data retrieval. Other systems that offer distributed storage and computing also exist. SnappyData [53] integrates Apache Spark and Apache Geode to support efficient streaming, transactions, and interactive analytics. Although these systems provide solutions that enhance Spark and eliminate inefficiencies of heterogeneous systems, they do not provide flexible operations and optimizations for trajectory data analytics. A recent related work on UlTraMan [54] adopts a flexible framework that supports customizable data formats, partitioning strategies, index structures, processing methods, and analysis techniques, which offer better support to realize optimizations and complex analytics. UlTraMan also adopts a unified engine that supports efficient trajectory data management and analytics.
