**8. Evaluation**

We ran a set of simulation experiments with artificial users in order to evaluate the feasibility of the approach, the appropriateness of the recommended eX-trajectories, and the scalability of the proposed framework. In this set of experiments, eX-trajectory recommendations were formulated and served to (artificial) users with diverse profile characteristics. Subsequently, the suggestions were evaluated for suitability against the relevant user profiles. Furthermore, the simulation process entailed concurrent formulation of recommendations to measure the execution time that is required for different levels of concurrency and quantify the response time of the SemMR instantiation.

For these experiments we used two machines. The first one was equipped with 2x Intel Xeon 8-Core CPUs, 64GB of RAM, and 480GB SSD with a transfer rate of 550MBps; the total estimated cost of this machine is less than 1.5K Euros, refurbished. This machine hosted the proposed framework, as well as the SemMR instantiation (database and MR eX-trajectory services). It also managed the database items, exhibit locations and semantic descriptions (already pre-processed as VEs), user profiles and trajectories, new content from IoT links (already retrieved but not pre-processed), and recorded and analysed the user paths. The second machine created the pool of concurrent users-visitors. For each user/visitor, the respective trajectory was built, when compared to existing trajectories and synthesized new, suggested trajectories, randomly, between 2–5 trajectories and 4–7 suggestions for each user tour. The machines were connected through an 1Gbps local area network.

For the cultural experience validation, we simulated scenarios for the visitors of the Acropolis Museum in Athens, which contains nearly 4250 works of art and welcomes an average of 4.5K visitors per day (nearly 1.5M per year) and while assuming that less than half of them visit it at the same time.

The user profiles that were utilized in the validation experiment were synthetically generated and included the following aspects: age, music choices, game choices, art preferences, museum theme preferences, mood, visiting style, gender, place of origin, and a returning visitor indicator, following the results of the study presented in Alexandersson et al. [63]. Each user profile could be matched to one of the six personas (predetermined user stereotypes) identified for the Acropolis museum in [64], however the degree of similarity for the best match varied from an absolute match to 5/8 attributes (the "mood" and the "returning visitor indicator" were not part of the stereotype modelling, only appearing in the user profile). Similarly, the eX-trajectory database was synthetically populated. Descriptions of landmark exhibits were crafted according to the permanent collection descriptions offered in the museum's website and, subsequently, areas of the museum were also modelled while using information from the Acropolis Museum application in the Google Arts and Culture website complemented with information from in-situ visits. Both landmark exhibit and museum area descriptions were semantically tagged. The semantic tags included the thematic area of the objects (e.g., religion, everyday life, wars, death, mythology, etc.), the artefact era, and the type of the artefacts (statues, household objects, buildings, grave goods). Subsequently, eX-trajectories with varying duration and spatiotemporal patterns were created and inserted into the eX-trajectory database, observing the profiles of the users. For instance, visitors with ant-type visiting style move sequentially along the areas of the museum, increasing their speed when the theme of the museum area does not intersect with their theme preferences, while moving more slowly in areas having content they are interested in. On the other hand, grasshopper visitors only approach certain exhibits falling within their interests spending a significant amount of time in front of them, crossing empty spaces and moving at a fast pace in other cases [36]. Physical fatigue was also considered in the generation of spatiotemporal sequences, with the effect being more significant in spatiotemporal sequences that are

associated with older persons, as asserted in [63]. Overall, 100 user profiles and 620 eX-trajectories were generated and inserted into the virtual entities and eX-trajectory data store.

The eX-trajectories that were generated by the system were parsed and evaluated for appropriateness to each user profile and visitor path. They were found to be in alignment with the user's visiting style, thematic preferences, and associated stereotype. For approximately 70% of the user/path combinations, 1–3 of the suggestions contained exhibits beyond the user profile thematic preferences, thus fostering novelty and serendipity [65]. As far as performance is concerned, Figure 9 indicates the overhead of formulating the eX-trajectory recommendations under varying degrees of concurrency.

**Figure 9.** Trajectory proposals formulation time for varying degrees of concurrency.

In Figure 9, we can clearly see that the overhead per execution is small (approximately from 0.18 to 0.3 s, depending on the level of concurrency) and scales linearly with the number of concurrent executions (users). However, for concurrency levels that are higher than 200, the machine was saturated, and performance dropped rapidly, indicating that a second machine (of the same specifications, as the one used in our experiment) must be added.

Overall, the cost required for the machines hosting the SemMR framework is estimated to be less than 17K Euros, which is deemed to be reasonable and affordable, in order to fully support a museum of the size of the Acropolis Museum.

#### **9. Discussion and Conclusions**

This paper presented SemMR, a framework for creating a new ecosystem of shared and optimised cultural experiences, offering the potential to accommodate MR interaction. The SemMR approach employs semantic technology for the utilization/integration of data/information discovered on the current Web sources, the Linked Open Data (LOD) cloud, and the Internet of Things (IoT), promoting experience sharing at user level, as well as data/application interoperability and low-effort implementation at the software engineering level. Sustainability is demonstrated by the validation of the framework in real-world use cases involving both end-users, as well as paired technology integrators from industry that not only aim to deploy SemMR, but also to create the experiences per situation and lead the formative user studies for usability evaluation, focusing on the integration and long-term impact of the integrated technologies to the specifics of the application environment.

This paper presents the SemMR framework, along with the proposed architectural design for its implementation and the scalability metrics based on a use case instantiation. The presented work introduces methods and tools for multi-entity interactions between entities of different types, to create reusable and optimized cultural experiences with low effort, towards a shared cultural experience ecosystem. Specifically, the SemMR framework introduces methods and interfaces for code-free implementation and the deployment of shared and reusable MR content, applications, and tools, emphasizing the notion of the eX-trajectory. The SemMR instantiations (implementation) of the framework deliver high quality cultural experiences, facilitating the interaction between a variety of entity types that interact in a virtual and fully experiential world. In addition, SemMR proposes methods for tracking, monitoring, and analysing user behaviour and the user interaction with the environment and other users, towards optimizing MR experiences by recommending their reconfiguration in two modes, which is at run-time (dynamically) or at development time (statically).

In the current work presented in this paper, the following SemMR components have been implemented: the ontology and the data stores, the MR eX-trajectory services for the management and analytics of trajectories, and, partially, the methods for user monitoring and user behaviour analysis for the evaluation of the framework with the experimental use case.

For future work, the SemMR system implementation that includes all components presented in Figure 2 that correspond to all key technologies that are described in this paper will enable the system deployment to real users for formative evaluation. Such an endeavour is expected to create new content that might be evaluated by both users and experts, such as museum curators. For the latter, an open research challenge is the automatic selection of the most relevant content from external sources and its integration to the authored content, for a seamless, yet enhanced, cultural experience.

**Author Contributions:** Conceptualization, C.V. K.K. and D.S.; methodology, C.V., K.K., D.S., D.M., V.K., C.-N.A., G.S., G.A.V., T.K., V.P., A.M., I.L., K.M.H., A.R., N.G. and B.P.; software, K.K., D.M.; validation, D.M. and K.K; formal analysis, C.V., K.K., D.S., D.M., V.K., C.-N.A., G.S., G.A.V., T.K., V.P., A.M., I.L., K.M.H., A.R., N.G. and B.P.; investigation, X.X.; resources, X.X.; data curation, X.X.; writing—original draft preparation, C.V., K.K., D.S., D.M., V.K., C.-N.A., G.S., G.A.V., T.K., V.P., A.M., I.L., K.M.H., A.R., N.G. and B.P.; writing—review and editing, C.V., K.K., D.S. and D.M.; visualization, C.V., K.K., D.S., D.M., V.K., C.-N.A., G.S., G.A.V., T.K., V.P., A.M., I.L., K.M.H., A.R., N.G. and B.P.; supervision, C.V., K.K., D.S. and D.M. All authors have read and agree to the published version of the manuscript.

**Funding:** This research received no external funding.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
