User eXperience (UX) Evaluation for MR Cultural Applications: The CEMEC Holographic Showcases in European Museums

Abstract

Within the EU CEMEC project framework, a novel approach for using holographic showcases in museums has been conceived and experimented upon in different venues in the context of an itinerant exhibition dealing with Early Medieval European collections. The purpose of this holographic showcase, the so-called “box of stories”, is to improve the link and interaction between real and virtual contents in the museum’s context, making the exhibited object “alive” in the visitors’ perception. An Avar sword and a Byzantine treasure have been used as the main case studies, and they have been experienced in the museums of several European regions by audiences with different cultural backgrounds. This has been a great opportunity to carry out user experience (UX) evaluations in order to collect feedback (from about 600 museum visitors) regarding the attractiveness of such a mixed reality (MR) system, its usability, the comprehension of the contents, the efficacy of the logistics and environmental conditions, as well as the educational impact. The results of such inquiries helped the CNR ISPC team to identify the most meaningful User eXperience Analytics (UXA) able to support the work of UX evaluators and UX designers to assess the efficacy of digital cultural products. Indeed, this manuscript presents UXA and tries to draft a concrete and effective evaluation model for future digital projects for museum contexts.

1. Introduction

1.1. General Objectives of the Research

This manuscript lays the foundation for a coherent and feasible methodology to conduct user studies about digital products. Specifically, the research team of the Italian National Research Council’s (CNR) Institute for Heritage Science (ISPC) (https://www.ispc.cnr.it, accessed on 5 November 2021) presents an innovative evaluation procedure applicable to different digital and virtual applications, tested for the occasion of the CEMEC European project (https://www.cemec-eu.net/, accessed on 5 November 2021). The latter saw the CNR ISPC team working on the design and development of a new digital output for museums and exhibitions: the holographic showcase, which is conceived to contextualize museum’s objects and tell their stories, making clear their functions to the public [1,2,3,4]. It consists of a physical structure where the real museum object is included. Thanks to the virtual projections on it and around it, such an object is brought back to life by means of dramatized and narrative techniques based on Pepper’s ghost effects [5]. A holographic showcase is therefore a mixed reality (MR) solution.

Studies on interface design, usability, user’s experience and appreciation, as well as learning benefits have been conducted for 4 years, namely whenever the holographic showcase was displayed in one of the CEMEC museums’ networks. The outcomes of such studies brought the CNR ISPC team to identify six User eXperience Analytics (UXA) useful for further investigating and drafting the efficacy of MR solutions displayed in museums.

It is true indeed that UX evaluations need to always become more systemic in any creative process for the cultural heritage sector. Digital applications and their audiences have always been studied as separated worlds. Recently, the importance of cross-methodological studies has been accepted by the academic domain for their usefulness in the process of assessing the impact of such applications.

1.2. Research Aims

The research presented in this manuscript tries to answer two needs and two academic domains.

The first is providing UX evaluators and experts in user studies with a tool (and a referencing conceptual framework) able to assist them during the various phases of analysis of digital cultural products both in the laboratory and when on public display. The second is providing content designers and communication scientists with valuable answers to questions related to the concept, design, development, and overall experience of innovative museum solutions such as the holographic showcase, which are as follows:

-

Do visitors appreciate how the holographic showcase is conceived and realized?

-

Do visitors appreciate the subjects, how the story is told, and its duration? What do they understand and remember?

-

Do visitors appreciate the appearance of the showcase in terms of the holographic effect, visual grammar, and adopted 3D technique?

-

How comfortable is it for visitors to access the contents?

-

Do the museum’s environmental conditions favor the enjoyment of the holographic showcase?

-

What does the evaluator notice about the users’ behavior?

In trying to answer such questions, we structured our evaluation model while taking into account differences in audiences, different technological adaptations and set-ups, the diversity in contents, and the role of the evaluator. Moreover, the results of previously conducted UX evaluations were also used in order to obtain a more general overview on lessons learned about the holographic experience in museums.

1.3. The Case Study under Application: The CEMEC Project in a Nutshell

Experimentation with holographic showcases was carried on in the framework of the European Connecting Early Medieval European Collections (CEMEC) project in 2015–2019 (https://www.cemec-eu.net/, accessed on 5 November 2021), where universities and research institutions, museums, and technical partners cooperated to advance the knowledge and promotion of the Early Medieval arts and cultures of different European regions, particularly in the museology sector. A traveling exhibition was created (2017–2019) which visited different European countries and where the holographic showcase was presented and evaluated in relation to its impact on the public and to different museum objects. Efforts have been oriented toward the creation of a so-called “box of stories” and the definition of a coherent communicative format, where the (1) real and digital contents, (2) structure, (3) materials, (4) interior set-up, (5) illumination, (6) perception, (7) audio-visual grammar, (8) narrative approach and dramatization, and (9) scalability are designed as a coherent “communication eco-system” [1].

The “box of stories” has been used to tell the story of some objects of the seventh century A.D., which was taken into account for the CEMEC project (https://vimeo.com/285977554; https://vimeo.com/236305120, accessed on 5 November 2021), and they were the following:

-

The Kunágota sword: an Avar sword exhibited at the National Hungarian Museum of Budapest.

-

The Mytilene treasure: a set of 70 Byzantine objects from the Byzantine and Christian Museum of Athens. Three representative objects of the treasure—a golden bracelet, a candlestick, and a trulla (a tool for water spilling)—were chosen for the holographic showcase.

-

The Brooch from Bonn: a golden disk-shaped brooch exhibited at the LVR LandesMuseum of Bonn.

-

The Brooch from Kolked: a golden brooch exhibited at the National Hungarian Museum of Budapest.

-

The Buckle of Kolked: a golden buckle exhibited at the National Hungarian Museum of Budapest.

The holographic showcase saw different set-ups according to the available spaces in each venue of the “Crosswords” exhibition (e.g., big holographic showcase, small holographic showcase, showcase surrounded by a projection wall, or video on a TV screen):

-

Budapest: Hungarian National Museum (HNM);

-

Amsterdam: Allard Pierson Museum (APM);

-

Athens: Byzantine and Christiane Museum (BCM);

-

Bonn: LVR LandesMuseum (LVR).

We created a precise storytelling according to the selected museum objects (although starting from a common approach) and we tried to solve engineering issues, which we developed and adapted them to from time to time. In addition, the position of the holographic showcase was decided each time upon several evaluations made by local curators, architects, technicians, and directors.

The results of the UX evaluations made in each venue helped the CNR ISPC team to improve the overall structure as well as the storytelling duration and usability.

For the purpose of this manuscript, the discussion is focused on the evaluation conducted on only two objects: the Kunágota sword and some selected artifacts of the Mytilene Treasure (Figure 1). Not all venues hosted the same configurations of these objects, but the evaluation involved the same set of questions and evaluative issues and requirements.

1.4. Manuscript’s Structure

This manuscript illustrates the case study used to test the UXA, and it drafts the evaluation model.

Section 2 refers to the key communicative criteria followed for the design of the MR solution as starting points of our experimentation in order to answer the research aims presented in Section 1.2.

Section 3 presents the evaluation method usable for any digital cultural product, showing the state of the art in such types of investigation, the selected techniques, the pipeline of work, and the different contexts of analysis.

Section 4 presents the results of the CEMEC evaluation, reporting them as analytics (UXA) in order to immediately provide a thematic grid of reference for future works.

Section 5 clarifies the lessons learned on the use of holographic showcases in museums emerging from the evaluative process.

Section 6 closes the manuscript by trying to provide readers with a concrete and effective model for conducting and analyzing UX evaluations while providing the research field with possible hints for the next generation’s design of MR applications.

2. Background Knowledge and State of the Art in User Studies

The conceptual framework on which the CNR ISPC team structured the holographic showcase relies on the idea of putting the user at the center of a coherent and stimulating experience (so called user-centered design) from different points of view (e.g., emotional, cognitive, sensorial, physical, and social) by using digital technology, or in this case, by using holographic systems.

Five key communicative criteria address the efforts toward the definition of a reliable and effective solution for cultural fruition and understanding, and they are as follows:

Use of mixed reality (MR) techniques: overlapping the virtual contents and the real objects is crucial to bring visitors’ attention to a museum’s collections, and MR brings the objects to life again (Section 2.1).

Importance of museum objects’ legibility and re-contextualization: the object is told and shown in meaningful moments of its construction, life, and use, providing users with its attribution of meanings (Section 2.2).

Focus on emotional narration: emotional narration stimulates motivation, self-identification, and memorization in visitors (Section 2.3).

Design of simple user experiences: the digital and virtual experience needs to be easy-to-use and surprising in terms of users’ expectations. Moreover, sociality and the possibility of exchanging opinions on certain contents are important for visitors’ engagement and understanding. Generally, interaction is not always a need or a desideratum for visitors (Section 2.4).

Learning as the final mandatory goal: the final goal of any communicative and technological effort is to let visitors learn from the visit and objects on display. Museums have to favor and enhance the cultural transmission between contents and visitors (Section 2.5).

2.1. Use of Mixed Reality Techniques

In recent years, the spread of MR technology [6] applied in cultural contexts appears to be elective and able to create innovative forms of narration and sensory stimulation, as they are particularly useful for favoring the readability of cultural content [7].

With respect to the field of cultural heritage and holographic technologies, the visualization calculated by the system (which runs beyond the holographic effects) is the element in which all these factors converge, applying to the representation of both the real and the virtual. In practice, users will have to use all their perceptive and cognitive faculties to read reality as they usually do in daily life, influenced by various contextual agents and adding and superimposing onto it a virtual level, which is denser in information units. The view, therefore, will have to undertake a path of convergence between the two levels of representation. It must then be located in a given space so as to allow the collimation of the two levels and the contingency of digital elements juxtaposed to the real ones.

2.2. Objects’ Legibility and Re-Contextualization

On a cognitive level, the experience of a cultural object (or site) and its contents is aimed at facilitating those steps of the learning process that are usually complex to activate when observing something that takes one back to the past. In detail, we talk about abstraction (imagination), or that moment in which the users face a fragmentary, decontextualized historical archeological find and they must reconstruct in their mind what this form refers to, what its original aspect was, and for what it was needed, among other aspects.

Thanks to holography (and virtual reconstructions in general), this cognitive step is facilitated and supported.

2.3. Focus on Emotional Narration

Technology is a tool which helps us shape the sense of involvement in the digitally augmented reality. The public is usually attracted by technological solutions. Nevertheless, several surveys on people’s behaviors which we carried out inside museums or exhibitions showed how the attention toward tools and devices is not long-lasting; it rapidly decreases if the cultural contents are not able to keep it alive and, similarly, if the interaction is difficult and not natural. For such a reason, it is necessary to rethink technologies at a deeper level of usage, but how? The answer is by working on the cognitive and emotional domains.

The emotional component is fundamental in a digital learning experience because it generates motivation, the first aspect that pushes people to face technological solutions and lets the learning process take place [8,9]. If we think about holographic applications, the process of stimulating emotions and human senses is central. Combined with emotions, storytelling does indeed help users recreate the sphere of life of ancient characters, objects, and sites.

For the CEMEC holographic showcase, each object was presented by alternating between (1) a short presentation with a more descriptive style to communicate the basic information (so-called “neutral vision”) and (2) a dramaturgy, or scenarios where each object was contextualized in its original environment and shown during its daily usage. Hence, the evocation and dramatization were brought inside the showcase as a holographic projection and told by a mixture of several voices and whisperings, environmental effects recalling events, and episodes and characters belonging to the object’s past life. This is a first-person drama. Contexts are historically verified or reliable, evocative and symbolic, and they combine 3D graphics and real actors (in the case of the Mytilene Treasure). The trulla and the bracelet were printed in 3D to be used by actors in their fictional historical settings [10].

2.4. Design of Simple User Experiences

The experience of any MR application must take into account a set of variables that can vary from time to time: the type and structure of the content; the time of use of the content; the context of use (i.e., single experience or collective experience); the place of use; and the conditions of use. All these aspects influence the users’ behaviors and the usability of the MR system when the users interact with it [11,12,13].

In general, it is clear that the design of the user experience is strategic not only for the purpose of (1) the aesthetic enjoyment of the users in front of the real cultural asset reconstructed in 3D and the virtual visit path designed for them but also (2) the stimulation of curiosity toward the story that is being told so as to be able to proceed with the storyline and arrive at the final epilogue and, finally, (3) the comprehensibility of the information, values, and meanings referable to each element of the past, be they landscapes, historical characters, or artifacts.

It is therefore essential to use a human- or user-centered approach when designing with holography, guided by the analysis and articulated knowledge of the needs and characteristics of the end users and the context of use [14,15,16].

2.5. Learning as the Final Mandatory Goal

The emergence of new immersive, dialogic, and immaterial learning contexts is required when dealing with MR solutions. Places of culture—when turned into digital representations—have enormous potential from a cognitive point of view. Indeed, digital cultural applications “transform” such places into multisensory and relational environments, favoring the emergence of emotions, motivations, and interests in users as well as stimulating multiple intelligences: social, introspective, spatial, bodily kinesthetic, and musical.

In this regard, the American pedagogist Edgar Dale (1900, Minnesota; 1985, Ohio) found that our memory is more influenced by our multisensory experiences [17,18,19]. The more these experiences are particular and full of emotions, the more users will remember them easily even after some time. From his studies, the famous “learning cone” was born, from which it is possible to deduce which stimuli and which channels are functional to solicit attention, memorization, and understanding in the individual. The answer is to involve multiple senses in the learning process [20].

3. The CEMEC UX Evaluation

3.1. Evaluation Method

As a researcher in experimental psychology, Jeff Johnson pointed out “you cannot create a user experience (user experience of use), but you can design something so that a user experience takes place. In particular, it is not possible to design a satisfying experience, but only to ensure that the ergonomic characteristics of a product can evoke this sensation in order to also guarantee a positive educational response in the user” [21]. It is therefore obvious how MR products like the holographic showcase serve various disciplinary sectors, from social sciences to information design and from cognitivism to information technology. These digital solutions need to integrate multiple investigative models and techniques.

That is why the CNR ISPC team adopted a holistic approach to user experience evaluation. This concept was first introduced in the education domain [22], and we adopted it as it perfectly fit the digital cultural heritage field. A holistic vision for the CEMEC project was essential to study how behaviors, learning paradigms, and performance objectives were interrelated. Moreover, the CNR ISPC team wanted to investigate the extent of learning benefits and human performance that could be measured against the general standards of user interaction with digital products as presented in the available literature.

Two evaluation models were implied to test the research aims of Section 1.2: (1) the objective measure of human–system interaction out of the user’s experience with the product, such as initial performance, long-term performance, learnability, memorability, and interaction with the graphical user interface (GUI) and its related functions, and (2) the subjective measure of human–system interaction, such as the impressions that users have of the system and their immediate satisfaction.

In order to translate the holistic approach into a usable framework for conducting UX evaluation, the CNR ISPC team took advantage of three research theories and tools:

-

Education: Kolb and de Freitas, experiential learning [23,24];

-

Interaction design: Nielsen and Norman, heuristics [25,26];

-

Sociology: Goleman, emotional intelligence [8,27].

The results of the convergence of such theories and tools produced User Experience Analytics (UXA).

3.2. User eXperience Analytics (UXA): Analytic Indexes of Evaluation

User Experience Analytics (UXA) takes its roots in the domains of data visualization and human–computer interaction [28]. Its purpose is to provide the CNR team and, in the future, professionals and experts in the field of museum studies with a grid of reference that can be considered when designing, developing, and assessing digital products such as the holographic showcase.

Being the CEMEC project representative of an entire category of multidisciplinary projects, touching on different skills, competencies, goals, and methodological procedures, UXA tries to incorporate three major domains: (1) education and pedagogy, (2) experience and interaction design, and (3) ICT and sociology.

UXA wants to study the holographic showcase starting from six main categories, which are also the results of years of analysis and evaluations conducted by the CNR ISPC team in the virtual museums sector. These categories are respectively described by UXA’s subcategories.

UXA and its sub-analytics are presented in

Table 1. UXA’s grid of reference.

	Analytics	Theoretical Question	Sub-Analytics	Rationale
1	Conception and Desiderata	Do visitors appreciate how the MR solution has been conceived and realized, along with the different exhibition set-up?	Overlapping of real and digital contents	Level of virtual coherence and perspective of real museum object and its digital replica
			Dramatization	Level of appreciation of the type and quality of narration
			Understanding, legibility, and object’s contextualization	Level of museum object’s visibility, readability, comprehension, and relation with its original place of discovery or usage in the past
			Others	Elements which may influence how the MR solution is conceived (e.g., competences of professionals involved, budget at disposal, museum space availability, or defined historical period in which the story is addressed)
2	Story and Storytelling	Do visitors appreciate the subjects of the MR solution? Do they appreciate how the story has been told? What about its duration? Do they consider the storytelling reliable? What do they understand and remember of the cultural information?	Historical fidelity	Level of reliability, scientific coherence, and historical pertinence of the museum object and its digital replica in the context of usage or discovery
			Content or subject	Interest toward the subject of the narration for the MR solution and appeal of the content for the selected target
			Style	Level of appreciation of the style of the narration
			Duration	Level of suitability of the duration of the story told about the museum object and its digital replica
			Soundscape	Level of appreciation of the sound chosen for the narration about the museum object
			Learnability	Level of understandability of the cultural information and narration chosen
			Real actors	Level of appreciation toward the use of real actors on a green screen to represent the story of the museum object
3	Visibility and Appearance	Do visitors appreciate the aesthetical features of the MR solution in terms of holographic effect, visual grammar, and the 3D technique applied to it? How about the object’s visibility in the MR solution? What about its illumination and its distance from the visitors?	3D Reconstructions and colors	Level of appreciation of 3D reconstructions of museum objects and general historical context, storytelling, and characters
			Object’s visibility and distance	Museum object’s location in the MR solution (inside, outside, close, or far), its visibility, and readability of its details
			Object’s illumination	Museum object’s appearance and visibility, level of efficacy of the lighting system, and level of coherence of illumination and storytelling (general atmosphere)
			Window accessibility, reflection, and glass	Level of efficacy and coherence of the protective glass which separates the real museum object from users and any issue related to the transparency or lights’ reflection which might disturb the vision
4	Logistics and Set-up	Is it easy for visitors to access cultural information? Is the interaction with the MR solution user-friendly and coherent? What about the visibility of the MR solution along the museum visit path? And what about comfort?	Position	Location of the MR solution along the exhibition pathway
			Chairs and facilities	Indication of the presence of benches, chairs, or other types of facilities close to the MR solution and along the exhibition pathway
			Sound system	Type, location, and volume of the sound system of the MR solution and of the exhibition in general and any potential audio conflict with other multimedia solutions
			Language selection	Chance to select the language of the MR solution and number of languages selectable
5	Environment and Plus Conditions	Do the museum environmental conditions favor the enjoyment and appreciation of the MR solution in terms of viability, crowding, noise, and general fruition?	Flow or viability	Level of viability and circulation around the MR solution and along the exhibition pathway
			Other multimedia	Presence of any other multimedia close to the MR solution and level of interference and usage
			Surrounding noise	Level of noise of the space where the MR solution is located and number of people allowed to enter and the presence of any surveillance
6	Audience	What does the evaluator notice about the visitors’ behavior when using the MR solution? Are they positive toward the experience? Are they curious? How do they behave when in front of the MR solution?	Behavior	Users’ way of behaving toward the MR solution, relevance of body language, type of movements or comments they perform, and type of experience they have
			Evaluation’s participation	Level of users’ involvement and free participation in the survey, level of consciousness of the usefulness of such evaluative activity, seriousness toward the compilation of the survey, and users’ self-confidence toward the survey in general
			Sociality	Level of users’ involvement with the MR solution and with other visitors and type of communication with the evaluator (if any)
			Researcher’s notes	Personal notes of evaluator about curators’ comments, technical features of the MR solution, enhancements to the MR solution, and general feeling about the MR solution of the users

in relation to the evaluation of the CEMEC holographic showcases so as to understand the process of analysis and the relation between the different thematic aspects. Anyway, UXA can be easily applied to any kind of MR application.

3.3. Evaluation Techniques

UXA was generated out of two evaluation techniques: qualitative and quantitative types of inquiry. The qualitative method serves to understand the reasons, motivations, opinions, and trends that hide behind the numerical data of quantitative research. It takes advantage of oral answers which are translated in a second phase into an accurate analysis of words, correlations, and meanings. The selected techniques for the CEMEC project are in-depth questionnaires (QT), which allow users to express their thoughts and doubts or respond to a specific request or opinion in a voluntary and unconditional manner (set of questions of the CEMEC questionnaire), and observations (OBS), which also partially fall into this category as they collect objective–subjective data of the users who are observed, including their actions, gestures, and paths, which are recorded according to a pre-established form. For the CEMEC project, this technique was also used. Quantitative research, on the other hand, is used to quantify using numerical data or, in any case, data that can then be easily transformed into statistics and to measure the behavior, opinions, and attitudes of a very large sample of respondents. The selected technique was again questionnaires as well as observing tips. The collected data are then transcribed into numbers, graphs, and statistics.

3.4. Scheduling

UXA came from data collected during the CEMEC exhibitions through manuscript-based materials: a user questionnaire and an observation template (for the operator only). The user questionnaire was autonomously filled in by users; only in some particular cases (old people) did the operator conduct a sort of interview. This was mainly composed of three sections:

-

Questions about the technological solution, design, and interaction;

-

Questions about the contents related to the museum objects exhibited in the holographic showcase in terms of learnability and memorization;

-

Questions about general appreciation and satisfaction.

The observation template instead reported the user roadmap when in front of the holographic showcase, with items related to user behavior, timing, actions, relations with others, and comments from the operator.

The user questionnaire and the observation template lasted an average of 15–20 min for the compilation.

The evaluation lasted approximately 1 week in each venue with the presence of one operator. The latter was the same person in all venues except in HNM, where a group of volunteers were trained to conduct the evaluation following the same methodological approach.

Evaluations were carried out on the days of general visitor flow from 9:00 a.m. to 7:00 p.m., according to the open hours of the museum. The operator made observations during the visitors’ presence in front of the holographic showcase, asking visitors to fill in the questionnaire just after the experience. This occurred just after they got out of the dark room where the installation was exhibited at NHM, once they got out of the last exhibition room at APM, and once they finished enjoying the story as it happened in BCM and also in LVR.

3.5. Pipeline of Work

For the Kunágota sword and the Mytilene Treasure, the pipeline of work set out for the UX evaluation consisted of the following:

1.

The analysis of each multimedia version (same contents presented as a virtual appearance in the big holographic showcase or as a video in a usual TV screen positioned near the original artifact) separately, collecting a specific set of questionnaires and observations (Figure 2);

2.

The creation of a summary prospectus only for the holographic showcases referring to the Kunágota sword and the Mytilene Treasure. This prospectus was divided into venues, and the summary was then compared with the one from the TV screens showing the same contents in order to understand how the perception and engagement of contents changed depending on the device;

3.

The identification of n.6 UXA and related sub-analytics as explained in Section 3.2 and Section 4;

4.

The study of collected answers and observations, where data were analyzed by comparing them thematically and dividing them by UXA category (also adding the operator’s feedback). This phase saw the division of the questions of the QT and OBS into closed questions (CQT) and open questions (OQT) in order to proceed accordingly with qualitative or quantitative analysis;

5.

The production of graphs and charts on Excel files, where statistics and formulas were applied singularly on each UXA, and a descriptive analysis was pursued. Specifically, the average, weighted average, median, and covariance were applied to a subset of cells or full columns;

6.

A median analysis was finally applied to generate a unique average value for each UXA related to the four venues.

3.6. Contexts of Analysis

UXA was applied to four venues of the CEMEC exhibition which differed in terms of dimensions, environment, lights, position along the museum visit path, and general atmosphere (Figure 3, Figure 4, Figure 5 and Figure 6). A summary is presented below.

Budapest, Hungary (HNM). The venue had the following features (Figure 3):

-

Quite dark space around the holographic showcase;

-

Holographic showcase placed in an isolated space (set inside a dedicated room of the museum);

-

Narration broadcasted through audio speakers (stereo sound);

-

Holographic showcase provided with a frontal walking space (maximum 10 persons) to dynamically position oneself at the correct distance from the projections;

-

Inner and outer projections alternating with each other and presenting the storytelling.

Amsterdam, Holland (APM). The venue had the following features (Figure 4):

-

No dark space around the holographic showcase, as it is illuminated by museum lights;

-

Holographic showcase placed not in an isolated space but along the museum visit path;

-

Narration broadcasted through a sound shower (mono sound) set on the top of the showcase;

-

Limited walking space (maximum four persons) around the holographic showcase;

-

Only the inner projection presents the storytelling.

Athens, Greece (BCM). The venue had the following features (Figure 5):

-

No dark space around the holographic showcase, which was illuminated by museum lights, and a ceiling cover to protect the showcase from environmental light;

-

Holographic showcase placed not in an isolated space but along the museum visit path;

-

Narration broadcasted through audio directional speakers (stereo sound);

-

Holographic showcase located in the middle of the museum pathway leading to another museum room;

-

Only the inner projection presenting the storytelling.

Bonn, Germany (LVR-LANDESMUSEUM). The venue had the following features (Figure 6):

-

No dark space;

-

Quite isolated and silent space (at the end of the museum corridor);

-

Headphones (two pairs different from those of the general audio guide);

-

Passed along the walking space (in the middle of a pathway to another museum room);

-

Inner projection only.

3.7. Data Collected

The data collected between 2017 and 2019 are shown in

Table 2. Data collected in each venue for the two types of content in the three configurations.

Object on Display	HNM	APM	BCM	LVR
Kunágota sword	170 OBS 142 QT big holographic showcase	170 OBS 72 QT holographic showcase	96 OBS 25 QT video configuration	81 OBS 70 QT holographic showcase
Mytilene Treasure	-	-	106 OBS 69 QT holographic showcase	41 OBS 20 QT video configuration

.

Not all venues presented the same technological configuration (Table 2)—even in the case of the holographic showcase—because at the time of each exhibition, the design and development of the system underwent several evaluation phases. These produced modifications, enhancements, and enrichment of some functions (such as the possibility to select the preferred language at any moment, even while the animation was running) as well as a few changes in the contents (such as their duration in the case of the Kunágota sword). Moreover, the number of collected QT and OBS differed according to visitors’ participation in the UX evaluation. Nevertheless, a respectful amount of data for each method was ensured, allowing the CNR ISPC team to work on average estimations.

Each location had its own public (according to the regular museum’s open hours) ticketing, audience provenance, and any collateral initiative ongoing during CEMEC exhibition for each venue. Thus, the public was different from time to time. None of the visitors, according to what the authors know, had been travelling between the four European locations.

It turned to be quite difficult to frame the audience’s engagement and participation in each CEMEC exhibition. Nevertheless, what was possible to draft was a general tendency of north European visitors (APM and LVR) to be familiar with technological solutions for museums, thus showing a more confident approach in using them during the exhibition, curiosity toward the content, and its way of being released. For visitors of the south European museums (HNM and BCM), instead, the behaviors were generally more conservative, quite confused about what was going on, and hesitant in approaching the digital experience.

It is true, however, that the older visitors in all venues were more intrigued by the holographic showcase, showing more directness in the usage of technology without caring about failing to experience the digital contents or any other issue that might occur, whereas younger visitors seemed to be timid and hesitant to use the holographic solution, especially when not alone but with school friends or colleagues.

4. User eXperience Analytics (UXA) Evaluation Results

4.1. First UXA: Concept and Desiderata

The first UXA refers to the idea which stands behind the design and development of the holographic showcase by the CNR ISPC team in terms of the MR solution, communication strategy, and overall learning goals (Figure 7). It tries to answer the following research question: “Do visitors appreciate how the MR product is conceived and realized?”.

According to the OQT, CQT, and OBS results (

Table 3. Concept and design UXA results from UX evaluation, with percentages of QT and OBS.

UXA Sub-Analytic	Positive Feedback	Negative Feedback
OVERLAPPING REAL AND DIGITAL CONTENT	94%	6%
DRAMATIZATION	87%	13%
UNDERSTANDING (Legibility, object’s contextualization)	93%	7%
OTHERS	89%	11%

), the users highly appreciated the effect of MR whenever the digital elements appeared on the real museum object inside the holographic showcase, with a score of 94% positive answers. The dramatization of the narrative and the graphics was again pleasant for 87% of the users; they affirmed them to be informative and new, and many of them liked to see how the object was used, while nobody complained about the lack of interaction. These data are surely supported by the high level of understanding of the contents by the users, especially those related to the object’s place of provenance, function, meaning, and original shape. Some concerns emerged for the digital product itself, especially at LVR. Here, the concept of an “augmented” showcase was not fully understood by the users, who wondered why there was not, for example, a simple video. Additionally, the dramatization was not reported as well-received for all users because it was associable to a childish and simplistic style. Someone complained again about insufficient warning of the digital products along the museum visit path, as in BCM and LVR.

4.2. Second UXA: Story and Storytelling

The second UXA refers to the content elaboration process, which brought the CNR ISPC team to transform the document-based materials and literature on the Kunágota sword and the Mytilene Treasure into engaging stories specifically suitable for the holographic solution (Figure 8).

This was enacted to try and answer the following research questions: “Do visitors appreciate the subjects, how the story is told, and its duration? Do they think the story is credible (and reliable)? What do they understand and remember?”

The second UXA turned out to be extremely positive (

Table 4. Story and storytelling UXA results from UX evaluation, with percentages of QT and OBS.

UXA Sub-Analytic	Positive Feedback	Negative Feedback
CONTENT AND SUBJECT	81%	19%
STYLE	79%	21%
DURATION	70%	30%
SOUNDSCAPE	80%	20%
LEARNING	83%	17%
REAL ACTORS	100%	-
HISTORICAL FIDELITY	70%	30%

), with more than 80% positive answers for all sub-analytics, apart from two of them: duration and historical fidelity (70%). Here, some comments came not only from the museum public but also from curators and experts. For the length of the storytelling, it was perceived to be a bit long by 30% of the users, especially for the Kunágota sword (almost 7 min. compared with 5 min. for the Mytilene Treasure). However, if one analyzes the answers in detail, it is clear how the length was strictly connected to the fruition condition: no presence of seats or chairs, which inevitably influenced the judgment of the users due to tiredness. Instead, for the reliability of the contents, 30% of the users contested the preciseness of the story, as it might have presented hypotheses as actual facts. This may have been provoked by the visual narrative and not by the descriptive style (linked to the results in Section 4.1).

4.3. Third UXA: Visibility and Appearance

The third UXA refers to the aesthetics of the holographic contents and structure, how it is conceived by the CNR ISPC team, the 3D models, and the overall appearance, especially in relation to the museum objects which are exhibited inside (Figure 9).

This tried to answer the following research questions: “Do visitors appreciate the appearance of the installation in terms of the holographic effect, visual grammar, and adopted 3D technique? How about the object’s visibility, its illumination, and its distance from the observers?”.

The results for the OQT, CQT, and OBS revealed (

Table 5. Visibility and appearance UXA results from UX evaluation, with percentages of QT and OBS.

UXA Sub-Analytic	Positive Feedback	Negative Feedback
3D RECONSTRUCTIONS AND COLORS	93%	7%
OBJECT’S VISIBILITY AND DISTANCE	77%	23%
OBJECT’S ILLUMINATION	92%	8%
WINDOW ACCESSIBILITY, REFLECTIONS, AND GLASS	75%	25%

) an almost total appreciation regarding the 3D reconstructions and models, the visual effects, and the animations, as well as for the object’s illumination inside the holographic showcase. The Kunágota sword was a unique object with lighting spots on it, while the Mytilene Treasure had three objects with the same lighting spots. It was interesting to experiment with lights that switched on and off according to the narration as dynamic elements of the composition. The users turned out to be positive toward this (92%). The issues referred to visibility in terms of reflections on the window glass (25% of negative feedback). In APM and BCM, due to external environmental lights or labels, which produced boring effects on the front view, the users were not able to clearly see the holographic showcase contents.

4.4. Fourth UXA: Logistics and Set-Up

The fourth UXA refers to all those issues related to the logistics and hardware set-up of the holographic showcase inside the museum context. This includes the position and its conditions of use (including chairs, sound systems, and interaction interfaces) (Figure 10).

This tried to answer the following research questions: “How comfortable is it for visitors to access the contents? What about visibility of the showcase along the museum visit path and users’ comfort?”.

This UXA clearly showed (

Table 6. Logistics and set-up UXA results from UX evaluation, with percentages of QT and OBS.

UXA Sub-Analytic	Positive Feedback	Negative Feedback
SOUND SYSTEM	75%	25%
LANGUAGE SELECTION	88%	12%
POSITION	42%	58%
CHAIRS AND FACILITIES	17%	83%

) how the logistics and the set-up of the museum installation influenced the user experience, particularly its position and location inside the exhibition. In APM and BCM, the holographic showcase was in the middle of the museum visit path, and for different reasons, the logistics did not favor a relaxed and absorbed experience in both cases. In BCM, it was in the center of the path, and in APM, it was along the path but in a hidden corner in a very narrow space near the way out. In this second case, the visibility of the entire structure was not always recognizable, and users ended up discovering the showcase by chance (since no indications or signals were available in the room or on the museum guide). Furthermore, the possibility to have seats, benches, or chairs was stressed as relevant to the users. In BCM, their presence was appreciated by users, while in APM and LVR, people complained of their absence. Instead, in HNM, chairs were not suitable for the type of user experience, since the holographic showcase was integrated in a big projection wall. Digital contents sometimes appeared inside and sometimes appeared outside the showcase, and people had to walk a few steps backward in order to enjoy the outer projection from a longer distance. Sound was another issue, even if the style was appreciated by 75% of the users. The problems pertained to the volume and the diffusion. In APM, we had the sound shower, which was very limited in the mono audio quality, even if it was easy to install. In BCM, we had speakers, but this conflicted with another multimedia application located directly at the back of the holographic showcase, producing overlap and noise in the room. In LVR, we had two sets of headphones, but this limited the experience of the showcase to only two persons at a time, even if silence was maintained in the museum room. Another problem in LVR was that the headphones for the holographic showcase were not the same ones used for traditional guided tours, so the same visitors did not immediately understand that they had to change devices. Finally, language selection was considered very good by users, with 88% positive feedback.

4.5. Fifth UXA: Environment and Plus Conditions

The fifth UXA refers to the conditions of the museum environment which may influence the experience of the holographic showcase in terms of users’ flow, crowdedness, noise, or silence and interference with other digital products (Figure 11).

This tries to answer the following research question: “Do the museum environmental conditions favor the enjoyment of the holographic showcase in terms of viability, crowding, and noise?”

This UXA proved what already emerged in Section 4.4 and Section 4.5 in terms of environmental conditions (

Table 7. Environment and plus conditions UXA results from UX evaluation, with percentages of QT and OBS.

UXA Sub-Analytic	Positive Feedback	Negative Feedback
FLOW AND VIABILITY	76%	24%
SURROUNDING NOISE	88%	12%
OTHER MULTIMEDIA	69%	31%

). This issue, in some cases, negatively affected the learnability of the content and the overall experience, especially in APM and BCM. The experience of the holographic showcase must be intense and somehow inclusive, and that is why a secluded and intimate space is preferred; otherwise, no emotional involvement is guaranteed. The selection of digital applications to be included into the museum spaces needs to be carefully planned in order to avoid an overabundance of technology in a close range or to better distribute multimedia along the museum visit path. Unfortunately, museum curators and traditional designers are not completely aware of these issues when dealing with multimedia.

4.6. Sixth UXA: Audience

The sixth and last UXA refers to the audience’s engagement toward the evaluation in each venue and its participation and feedback. Such data pertain to both the direct answers of users and the operator’s comments about what he or she observed during the evaluation sessions. This tries to answer the following research questions: “What does the evaluator notice about the users’ behavior? How do they behave when in front of the holographic showcase? How is their experience with the installation shaped?”.

The results for the OQT, CQT, and OBS revealed (

Table 8. Audience UXA results from UX evaluation, with percentages of QT and OBS.

UXA Sub-Analytic	Positive Feedback	Negative Feedback
BEHAVIOR	-	-
PARTICIPATION TO EVALUATION	95%	5%
SOCIALITY	50%	50%
RESEARCHERS’ NOTES	-	-

) a good predisposition and the openness of the users participating in the inquiries if these initiatives were presented to them well in terms of importance, prestige, usefulness, and scientificity. Audiences were different from venue to venue, and the operator’s behavior also needed to change to meet the users’ habits and culture. In general, what emerged was that 50% of the users visited the CEMEC exhibition alone, while another 50% went with friends or family. The average age was around 43 years old with great differences for HNM and LVR, which saw a younger audience, while BCM’s audience was middle-aged and APM had an older audience. The holographic solution in museums stimulated discussions among users, as the operator observed: the fascination of the effects, the revealed story of the museum objects, the funny facts and the secrets told by the narrator, the voices, the music, and the use of characters and actors in the storytelling. In general, the evaluations confirmed a great interest toward the new narrative approach and the holographic system designed for the Kunágota sword and the Mytilene Treasure. There was a marked sense of curiosity toward such multimedia installations. Visitors came close to the holographic showcase not really having in mind what to expect from it, but at the end, they seemed satisfied and fascinated by what they had just experienced.

5. Discussion: What to Care about When Designing, Developing, and Experiencing Holographic Showcases

According to the research questions (Section 1.2) and based upon the results of the UXA (Section 4) referable to the Kunágota sword and Mytilene Treasure’s holographic showcases, the CNR ISPC team was able to summarize the lessons learned out of the UXA identified during the several evaluations:

1.

For the UXA “Conception and Desiderata”, it is important to care about the following:

-

Mixed reality (interaction with virtual and real contents) was confirmed to be appreciated as an efficacious means of museum objects’ contextualization through OQT and OBS;

-

Dramatized storytelling pushed visitors within the story and brought the object back to life. This is because it harmonized well with the magic of the holograms, as it was conceived by the CNR team, and the CQT, OQT, and OBS confirmed such an issue in comparison with the actual used museum communication channels;

-

Visitors appreciated the novelty of the holographic showcase, but its efficacious integration in the context of the exhibition was an issue from a technical and logistical point of view. Certain elements did not always work, which happened in APM and BCM due to the MR solution being located in an inappropriate manner (too close to other multimedia).

2.

For the UXA “Story and Storytelling”, it is important to care about the following:

-

Even in the absence of user interaction with the system, the highest level of cognitive learning was reached through attention, memorization, and elaboration of historical contents, as the CQT and OQT revealed, especially for cultural content questions;

-

The style of the story divided users’ opinions. The Kunágota sword was considered pleasant by most visitors and confusing or childish by a smaller part of them whereas the Mytilene Treasure amassed positive appreciation for its essentiality and the use of real actors (which provided emotions and self-identification). This issue was merely personal and subjective, and it was influenced by the visitors’ provenance and their attitudes and familiarity toward digital applications in museums;

-

Concerning doubts about historical fidelity, for the Kunágota sword, some users referred uncertainty about the reliability of the contents, probably because of the pictorial style. Again, the issue here was subjective, and it differed according to the age of the public and the aesthetical layout they expected to see in museums;

-

A written description of the subject and declaration of the duration of the story turned to be necessary for users’ understanding and comfort when in front of the holographic showcase. In all venues, the authors missed some of the above-mentioned information, and visitors highlighted these needs in the OQT and OBS;

-

The duration of the story was an issue due to the time at the users’ disposal when in the museum, but it also depended on the environmental conditions and the type of museum experience (e.g., alone or in a group and standing or sitting). This aspect is controversial and needs to be well addressed before making the storytelling, already having in mind what the final set-up will be (e.g., cinema-like installation, serious game, or online application);

-

The users appreciated the voiceover and soundscape (better than written texts). Nevertheless, audio in museums needs to be managed to create a user roadmap where the acoustics and volume of digital applications do not interfere with the whole museum experience but still preserve their function and relevance. In APM and BCM, several problems occurred, since the MR solution did not have too much space at its disposal and due to the contiguity with other multimedia, which made the environment noisy. In HNM, the set-up was perfect in terms of audio diffusion, as was that in LVR, where earphones were used (but this opened up other problems, like the reduced number of people that contemporarily could listen to the storytelling).

3.

For the UXA “Visibility and Appearance”, it is important to care about the following:

-

Good illumination of the object inside the holographic showcase is a relevant issue for letting users fully experience the beauty of the real museum object. When such a condition was not guaranteed, the visitors did not appreciate the storytelling either, as the CQT and OQT revealed;

-

It is necessary to follow a thoughtful design for the height of the window of the holographic showcase to suit all visitors’ statures (e.g., children, elders, and families). In APM, BCM, and LVR, children were not able to see the digital contents, obliging their parents to pick them up;

-

Glass protection from reflections is needed for avoiding external interferences with the holographic effects happening inside the showcase;

-

A big holographic showcase seemed to work better than a small one in terms of narrative potential and involvement of groups of users, as the set-up in HNM and the OBS, CQT, and OQT revealed.

4.

For the UXA “Logistics and Set-up”, it is important to care about the following:

-

The position of the holographic structure needs to be visible, and in terms of space, comfort, and viability, good integration with the entire exhibition space (but not in the middle of the public passage) has to be provided. In APM and BCM, the situation was uncomfortable, while at LVR, the location was perfect in terms of viability and position along the exhibition pathway;

-

Leave a sufficient free space in front of the holographic showcase (at least 1 m of distance) for users to prevent them from seeing the trick of the Pepper’s ghost set-up;

-

Provide seats for users (especially children and elders). Users want to rest in front of multimedia, especially if it is not interactive. This influences the perception of the story’s duration and overall appreciation, as the CQT and OQT revealed;

-

Provide diffused audio whenever possible or several Bluetooth headphones to let more people enjoy the holographic show. The soundscape quality was greatly appreciated by users and also expected, as visitors reported in the OQT;

-

It is important to avoid waiting for language selection and provide a simple user experience in order to not frustrate users with multiple functions and an unclear system interface. This aspect was detected during the OBS and CQT;

5.

For the UXA “Environment and Plus Conditions”, it is important to remember the following:

-

Do not exhibit too much multimedia close to each other, because they can conflict in terms of audio and users’ tolerance (like what happened in BCM);

-

Take care to ensure an efficient visitor flow, because crowd or noise situations may interfere with the user experience of the holographic showcase. This happened in BCM and APM, and the CQT, OQT, and OBS confirmed this negative trend;

-

A dedicated space that is semi-dark and quiet for the holographic solution would be better, given the purpose of stimulating emotional intelligence to understand and virtually relive the past. Only in HNM was this guaranteed;

-

In addition, create an integrated communication strategy for multimedia. Mention their presence along the museum visit path in the catalogue or in the museum guided tour, as well as through signals and visual graphics. None of the CEMEC venues fulfilled this task, but it is highly recommended for the next generation of MR solutions in museums.

6.

For the UXA “Audience”, it is important to remember the following:

-

The entire museum experience highly influences the user’s interaction with technology due to the time at their disposal, the museum visit path, the crowd, the flow, and the type of experience (alone or in a group);

-

Storytelling was confirmed to be the most powerful way to involve the public when in front of the holographic showcase (and digital applications for communication in general);

-

It is fundamental to improve the dialogue between curators and creatives. It is important to take care of the results of the user experience to improve the technological solutions from time to time. By considering the users’ feedback, we were able to enhance the experience of the holographic showcase set up in the four venues.

6. Conclusions: A UX Model for Studying and Evaluating Virtual Museum Projects

What we learned from the CEMEC project was that a concrete and effective evaluation model can be essential when designing, developing, and experiencing digital solutions for museums and MR applications in general [1,2]. The meaningful research questions did not include “Does technology work in my museum with my audience?” Instead, they were the following. “Does my museum have good conditions to present digital contents to my audience?” “Is the content engaging and captivating for my audience?” “Is the design and interface of such technology clear for all users?”.

The focus should therefore not be on technology but on the conditions and contents. Experience design in museums is complex and requires specific competencies which involve scientific, engineering, artistic and graphic, pedagogical, psychological, technical, and educational skills [29]. This is meant to create something new and powerful for cultural transmission, integrating real and virtual contents for a greater sense of presence in the story. The future challenge in museum studies and digital technologies is to shift research from a perceptive sphere, both from a content point of view and from a technical point of view, for limiting the cognitive mistakes and putting users in a condition of comfort and enjoyment once they experience a digital application.

From a methodological point of view, the evaluative procedure applicable to digital and virtual applications also must rely on referencing models, and the UXA model can be proposed. UX evaluations need to always become more systemic in any creative and communicative process for the cultural heritage sector.

Digital applications and their audiences have always been studied as separate worlds. Recently, the importance of cross-methodological studies has been accepted by the academic domain for their usefulness in the process of assessing the impact of such applications. The CNR ISPC team is firmly convinced that adopting a holistic approach to UX evaluation is central for next-generation digital applications. Indeed, hedonic aspects such as emotions, senses, perception, and the environmental atmosphere as well as technical aspects like interface design, usability, and affordance have a precise and crucial role in the meaning making of any (cultural) experience.

Such an innovative approach in UX inquiries might allow experts and professionals to take advantage of a multidisciplinary framework to deepen the knowledge of user studies and audience engagement and development.