*5.1. eX-Trajectory Management and Analytics*

This component is responsible for managing and analysing eX-trajectories while using methods for representing, storing, querying, linking, synthesizing, and enriching eX-trajectories. It identifies similarities between eX-trajectories, which are the types of relations between one or more eX-trajectories. It supports a multitude of analytic tasks for eX-trajectory behaviour, compilation of statistics, and advanced methods for mining co-movement pattern from eX-trajectories. It provides a number of services, by exploiting data from other architectural components and, more specifically: (a) it generates recommendations for the reconfiguration of eX-trajectories, utilizing user behaviour analytics data that are sourced from the user behaviour analysis component; (b) it supports the integration of VEs into eX-trajectories through the use of data that were obtained from the IoT/VE management component; and, (c) it enriches the eX-trajectory data and information by utilizing data that were sourced from the data/information management component.

#### *5.2. IoT*/*VEs Management*

This component is responsible for (a) promoting connectivity between human and non-human entities, (b) enabling abstract representation (virtualisation) of those entities using the semantics of the SemMR ontology, (c) managing VEs by offering services for the efficient storage and querying of data/information created by VEs in the context of their involvement in eX-trajectories, and (d) supporting trustful interactions in the context of cultural spaces, through the realization of a relevant trust model and the computation of trust between VEs.

#### *5.3. Semantic Data*/*Information Management*

This component facilitates: (a) search and discovery of disparate and heterogeneous domain-specific data and information related to eX-trajectories, (b) transformation of data to a common syntax and model (RDF) and data integration under a common view, based on the SemMR ontology as well as suitable domain-specific ontological models, and (c) the enhancement of eX-trajectories, through the computation of offering recommendations containing unified and integrated data and information; to this end, the semantic data/information management employs semantic matchmaking methods.

#### *5.4. User Behaviour Monitoring and Analysis*

This component is designed for facilitating (a) the tracking and monitoring of human entities during interaction, (b) analysing user behaviour to identify confusion, boredom, uncertainty, frustration, etc., (c) examining the user affective mental state and preferences to reach decisions regarding intervening actions that must be taken, including offering of comments, the generation of recommendations for path changing, (d) identification and presentation of additional content, (e) augmenting the initial user profile, already present in SemMR by contributing physiological aspects that are derived from the monitor and analysis of user movement, and (f) realizing an integrated interactive interface, through which feedback might be provided.

#### *5.5. The Cultural Authoring Environment*

This component is responsible for authoring eX-trajectories and developing cultural apps with low effort. It provides the following: (a) a graphical drag-n-drop code-free authoring interface of eX-trajectories, (b) methods for synthesising open and reusable eX-trajectories, (c) methods for integrating VEs into eX-trajectories, and (d) methods for enhancing eX-trajectories through semantic enrichment. This environment allows for the creation of domain-specific cultural applications that are delivered to the intended audience for immediate use. The cultural authoring environment interfaces with the MR devices and, in general, the infrastructure that is required for the MR application delivery and makes these interfaces available to developers as objects and APIs for a higher level of abstraction. In this fashion, the MR application development is disassociated from the idiosyncrasies and peculiarities of the hardware and, therefore, can be more efficiently developed and with better portability.

In the heart of SemMR, there is a triple store for storing and querying integrated data in Resource Description Framework (RDF) data model [55] and the SemMR ontology encoded in the OWL W3C ontology language [56]. The SemMR ontology is designed to encompass representations for eX-trajectories, VEs, and all additional domain-specific data and information that is required, i.e.,


The design of concepts and properties in the SemMR ontology regarding semantic trajectories is based on the definitions provided by the datAcron ontology for the representation of semantic trajectories of moving objects [48] and the semantic trajectories design patterns provided by Zhang et al. [57].

The SemMR ontology development process will follow a collaborative workflow, such as the one specified in the HCOME methodology [58], while using the collaborative ontology engineering tool WebProtégé and discussion threads via e-mail and Google docs/groups. In the architecture diagram that is illustrated in Figure 2, three stores are depicted as a conceptual approach to the organization of the SemMR data/information management; however, this is not necessarily the case for implementing three different physical stores.

Figure 3 depicts the SemMR key offerings and supported technologies.

**Figure 3.** SemMR key technologies.

#### **6. SemMR Example Ontological Specifications**

In the following subsections, we present the cultural heritage domain considerations and the ontology specifications that implement the semantic trajectory for the domain semantics.

#### *6.1. Cultural Heritage Domain*

The CrossCult ontology (IRI: http://kb.crosscult.eu/) organizes cultural object semantics in multiple layers, aiming to underpin the representation of connections between cultural heritage data. The CrossCult ontology extends the basic CRM modelling of CH data semantics, by accommodating the modelling of users (of applications) and visitors (of venues), as well as venues (sites, buildings, rooms, etc.). Furthermore, it defines semantics such as interest and review (subclasses of *E73 Information Object* in CRM). Moreover, the CrossCult ontology reuses elements from the namespaces of FOAF (e.g., for modelling persons, their activities and relationships between persons), Dublin-Core (to describe periods of time and specialised datatypes), and SKOS (to model relationships between concepts), as well as DBpedia (primarily for enriching instances of the Upper-level ontology with links to DBpedia concepts) to offer a more comprehensive coverage of concepts of interest.

A modelling choice of CrossCult ontology, adopted in the SemMR ontology modelling, is the reuse of the specialised CIDOC CRM [59] classes, such as *E22.Physical Man Made Object* and *E24.Physical Man Made Thing*, which provide a unified view to a wide range of concepts. Artefacts, paintings, museum exhibits, monuments, are modelled as instances of those CRM classes. This design choice also enables the use of standard semantics for modelling spatial, temporal, geometrical, and other semantic relationships.

Figure 4 depicts a description of an exhibit (PoI), while Figure 5 illustrates the exhibit (PoI) modelling within SemMR. The CrossCult definition of a *Visit* (and related classes i.e., *Visitor* and *VisitingStyle*) in its *User-Model* namespace has a central role in the representation of the knowledge in SemMR.

In the example that is depicted in Figure 6, we illustrate how individual routes followed by users are linked with a particular visiting style (e.g., ant, grasshopper, butterfly, etc. [36]). Note that a relationship between users and visiting styles is also accommodated at the user profile level, indicating which visiting style a user typically follows. However, it is possible to represent differentiations from the typical user behaviour, occurring at the individual route level, as shown in the example. The visiting style of each individual visit is determined by analysing the user traces against the locations of the exhibits.

**Figure 4.** Textual description of an exhibit (PoI).

**Figure 5.** Informational model for an exhibit (point of interest) in the SemMR ontology.

**Figure 6.** Linking of visitors, visits, and visiting styles.

#### *6.2. Semantic Trajectories*

Motivated by real-life needs in critical domains, such as aviation and maritime, the datAcron ontology provides a coherent and generic representation of semantic trajectories for moving objects [48]. It reuses the ontologies: DUL, SimpleFeature, NASA Sweet, and SSN. A semantic trajectory consists of a sequence of temporally non-overlapping trajectory parts, as shown in the ontology definitions in OWL syntax depicted in Figure 7; each of these parts may be either (a) a semantic node, (b) a raw position obtained from some sensing device, or (c) a trajectory segment. Each trajectory part can be associated with spatiotemporal information regarding its occurrence; in this context, the point or region of the trajectory part occurrence can be expressed via a specific geometry, while the temporal dimension of the trajectory part occurrence can be expressed through a time instant or interval.

Similar to the above definitions, but in a more simplified and generic way, the semantic trajectory design pattern [60] presented in the OWL snapshots that are depicted in Figure 8, defines a semantic trajectory as a number of segments and fixes (synonym to points in other vocabularies). According to OWL, it defines a number of interfaces to incorporate additional geographic information, domain knowledge, and device data.

#### **Figure 7.** Semantic trajectory definition according to datAcron ontology.

**Figure 8.** Definition of a semantic trajectory as a number of segments and fixes.

In SemMR, we link/connect any of the two semantic trajectories representation to the CrossCult ontology, in order to be able to formally represent the eX-trajectory, as defined in this work. To do so, we propose a number of simple design patterns in the SemMR namespace, as described below. In the description of the design patterns, the ontologies listed in Table 1 are used; Table 1 also lists the mapping between the prefixes used in the examples and the full ontology IRIs:

1. Define new concept: eXtrajectory as subClassOftraj:SemanticTrajectory

$$\exists e \text{X} \newline \text{Tra} \newline \text{x} \newline \text{y} \subseteq \text{Semanatic} \newline \text{Tra} \newline \text{x} \newline \text{y} \newline \tag{1}$$

2. Link/connect *traj:SemanticTrajectory* to *cros:Visit* (which represents a specific visit by a particular user) via an object property *(:visitMade*)

$$\exists \ x \text{X} \\ \text{trajector} \\ y \sqsubseteq \exists \text{wis} \\ \text{t} \\ \text{Made.} \text{Visit} \tag{2}$$

3. Link/connect *traj:SemanticTrajectory* to *crm:E28\_Conceptual\_Object*, to allow the explicit representation of the semantics that are associated with a trajectory. Similarly, *traj:Segment* and *traj:Fix* are linked/connected to *crm:E28\_Conceptual\_Object*, to allow for the representation

of semantics that are associated with trajectory segments and individual points. Under this arrangement, all representational levels of eX-trajectories may bear relationships to semantics. However, it should be noted that the semantics appearing at the highest level, used in any context, are the ones conveying the actual meaning of the trajectory more accurately, overriding the lower-level ones. For instance, an eX-trajectory *exTr0001* may be comprised of the segments "visit to a museum" (associated with cultural semantics), "launch at a restaurant" (associated with food service and local cuisine semantics), and "shopping at a flea market" (associated with street market and local products semantics). However, the eX-trajectory *exTr0001* might be associated with tourist behaviour semantics, modelling the overall behaviour, and not the lower level details of the constituent parts.

4. Link/connect *cros: Visit* to *crm:E28\_Conceptual\_Object*, to allow for the explicit representation of the semantics targeted by a particular user visit. This is required, since different users may be following a specific semantic eX-trajectory (which is linked to the *cros:Visit object*), focusing on the diverse semantics of the eX-trajectory. For instance, a visitor may follow the "Acropolis of Athens" eX-trajectory focusing on the ritual aspects of monuments, while another visitor may follow the same eX-trajectory, focusing on the architectural aspects of monuments. By capturing the point of view of each individual user, more elaborate analytics can be produced, and more accurate matching can be performed, leading to better recommendations.


#### **Table 1.** Ontologies used for the framework evaluation.
