Multisensory Museums, Hybrid Realities, Narration, and Technological Innovation: A Discussion Around New Perspectives in Experience Design and Sense of Authenticity

Abstract

This paper examines multisensory museums, envisioned as extended or hybrid realities where the physical and digital coexist to offer cognitively and emotionally engaging experiences to the public, while enhancing the sense of authenticity. Multisensory communication can improve accessibility, as it conveys the content through multiple perceptual channels, and it can solicit more inclusive, participatory, and creative audience engagement through the stimulation of emotions. A methodological approach to digitisation, communication strategies, interaction, creative storytelling, immersive technologies, and accessibility issues are discussed in depth, in relation to emerging museological practices. Machine learning and generative AI are opening new scenarios in management practices and decision-making, in data analyses and in natural language processing, and in the creation of personalised content addressed to the audience. A short overview about how these algorithms evolve and work is presented, considering the challenges, threats, expectations, and ethical implications they pose. Another key topic is the sense of authenticity. How is it changing in today’s users? How do virtual and mixed realities, storytelling, and user experience design in museums influence it? This paper explores these issues, presenting some case studies which have cultural, social, philosophical, and ethical implications, at a time when museums are redesigning their role in a society undergoing profound transformation.

1. Introduction

1.1. The Museum Between Map and Territory

Territories and ‘meta-territories’ of the cultural objects are various: the place in which, or for which, they were created, or a new space, different from the original one, where they are displayed to the public, such as a museum. The latter is a non-place, an abstract, metaphysical space in which the logical connections among objects are divorced from their original reasons for being. On the other hand, the museum is also a new territory, a living space, crossed by people who, hopefully, want not only to look at objects but also to experience them, understand them, and become excited by entering their stories. Multiple contexts therefore intertwine. The past life of the objects and their present life establish a relationship with the experience of those approaching them [1].

Therefore, the process of contextualisation of the cultural object and attribution of meaning is diachronic, evolving through different cycles of interaction, through variations and redundancies, and differing from era to era, from person to person.

The virtual integrated into the real museum should foster a less hurried and consumerist, more profound comprehension of the cultural heritage, but also a creative approach, capable of soliciting the individual on multiple perceptual, cognitive, and emotional levels.

Thanks to multimedia and “phygital realities”, meaning the integration of real collections and digital content, museums can become active workshops of cultural experiences that build bridges between the lives and cultures of the past and our present time.

1.2. Topics and Structure of the Paper

This paper focuses on the topic of multisensory museums and the design of effective hybrid realities applied to cultural heritage, that is, cultural experiences where real and digital elements are combined and integrated, aiming at representing and transmitting cultural heritage to the public in an innovative and engaging way, considering both tangible and intangible values. The impact of multisensory communication in museums, enhanced by technological innovation, will also be considered in relation to the sense of authenticity of the cultural message perceived by the public engaged with such “extended” museum contents. Indeed, concepts such as the sense of “authenticity” and “embodiment” recur throughout various sections of this article, as they are fundamental pillars of multisensory museums, closely connected to digital communication, storytelling, appearance and enrichment of digitised contents, soundscape, sense of presence, and involvement in simulated virtual reality environments. What are the criteria for designing a deep, useful, and enriching cultural experience? How can we stimulate in the audience involvement, understanding, memorisation, imagination, and creative elaboration?

The considerations expressed in this article stem from the author’s academic background in the preservation and valorisation of cultural heritage, art history, and museology [2], as well as from 25 years of research activity and experience in the field of digital innovation in cultural venues and virtual museums. These insights are supported by the results of several surveys carried out on thousands of visitors approaching digital contents in Italian and European museums.

Section 1, the introduction, presents the main concepts and topics, the objectives, and the primary target of this contribution. It introduces the concepts of extended reality and “phygital” experience, along with the reasons for their application in museums and cultural venues. The state of the art of multisensory experiences in museums is discussed in relation to ethical issues, such as accessibility, content richness and reliability, sense of authenticity, connection with the territory, inclusivity, and the new frontiers disclosed by artificial intelligence (AI), NFTs, the metaverse, and by the resulting new value chains. The user-centred approach in communication is then introduced, addressing how perception, action, motivation, emotion, and cognition contribute to creating engaging and memorable experiences, based on principles other than fast cultural consumerism. Section 2, Materials and Methods, presents the basic principles of multisensory museums: narration, interaction, progressive immersion and understanding of contents, mixed reality, tactile interfaces, media combination and hybridisation, individual and social dimensions, sense of wonder, soundscape, embodiment. An important discussion develops around the sense of authenticity, which is connected to a multiplicity of rational, emotional, individual, and social factors. Essential preconditions are the quality of digitisation and the reliability of virtual reconstructions of lost or fragmented contexts, the criteria of which are shortly summarised. The user-centred approach is also closely connected to target audience profiling, where traditional methods are complemented and exponentially enhanced by artificial intelligence, which is revolutionising the field of cultural marketing. The use of AI in the cultural heritage and museums sector is discussed, considering a wide range of activities and applications, including a reflection on opportunities, threats, and ethical issues. Section 3, Results, presents some case studies in which the author was actively involved during their development. They have been selected to concretely exemplify the principles and methods discussed in the previous section. The purposes, target, processes, and technologies applied to each case are briefly presented, focusing on their most meaningful aspects. Section 4 discusses, in light of the previous sections, how the museum should be considered as a dynamic place of evolving narration, where the visitor can live perceptual experiences that are unique and not equally replicated outside the museum, beyond the opportunities provided by the smartphone, which people commonly use in their daily lives to reach every kind of content. Emphasis is placed on the design of the museum experience, which must be based on the effective integration of real collections, digital content, and exhibition layout: real and digital should be combined, to focus the visitor’s attention and emotion on the place in which he/she is currently immersed. Section 5, the Conclusions, summarises the main contents and the emerging values of this work, opening perspectives on future desirable developments and directions.

1.3. Museum, Virtual Heritage, and Extended Reality

In most museums today, we can approach the objects on display almost exclusively with the aid of sight: we can look at the artifacts but we cannot touch them, listen to the voices or sounds they hold back (think about textual or musical contents of manuscripts or epigraphs that remain totally unexpressed to the public, as in most cases, their language or writing is difficult to decode today), we cannot perceive their smells, or be aware of the artisanal processes that produced them. In most cases, our approach to cultural objects is still formal, taxonomic, and abstract.

Nowadays, thanks to social media, the museum promotes itself as a potential centre of open cultural production, engaging in dialogue with its audience. This phenomenon must be managed according to an open but authorial strategy, in which the museum, as a cultural and educational institution, welcomes contributions and ‘stories’ proposed by its public but also manages and directs them while maintaining a guiding role. The museum should envision and develop new perspectives, including but not merely following the demands of its users.

The digital life of the museum cannot be exclusively linked to social media. The layered paradigms of the virtual museum [3] can help in implementing a new digital curatorship based on high-quality audiovisual content, aesthetic enjoyment, the creation of a continuum between real and virtual experiences, structured storytelling, virtual and mixed reality, sensory immersion, and a sense of presence within stories and environments. Multisensory solicitation can evoke the sensory, perceptive, and symbolic dimension surrounding the objects or lost contexts of life, which would otherwise remain completely unexpressed. Through the virtual dimension, lost relations between objects, eras, stories, and contexts can be restored, represented, and experienced. What is most important, in the virtual dimension, is not the objective description or the digital replica of the real artifact, but rather the interactive processes that can be activated, processes that are open, personal, and diverse [4]. Thanks to digital “extensions”, objects and places can be usefully perceived and understood in terms of

• shape, potentially digitally restored;
• context, through virtual reconstructions or mixed reality;
• technical processes, symbolic values, and social attribution of meanings, through storytelling.

A virtual heritage network is therefore a multidisciplinary and multidimensional system connecting objects, places, authors, contents, users, real worlds, and virtual dimensions. Extended reality is a “continuum” where real and virtual coexist, complementing and validating each other reciprocally. Multisensory communication plays a crucial role in increasing the accessibility to museum content as it conveys contents through multiple perceptual channels and fosters inclusive, participatory, and creative audience engagement by stimulating ideas and emotions. Bringing space to life allows museums to break down the isolation between the objects and the public, enhancing the sense of wonder [5], astonishment, and empathy, and ultimately encouraging exchange and dialogue between people [6]. Audiovisual, tactile, even olfactory or gustatory narration, when guided by criteria of truthfulness and historical plausibility, allow the public to feel embodied in the experience and to be engaged with an interconnection of meanings, tangible materials, and symbolic values.

Storytelling is another key element in the processes of engagement and immersion in the cognitive experience. It should shape the museum space and the organisation of collections, fostering both intimate reflection and interactive, collaborative experiences through a variety of audiovisual languages, interaction levels, and technologies chosen to convey stories [2].

As a public place of education and culture, the museum naturally favours collective communication, social sharing, and exchange. The various forms of content representation, languages, and technologies are usually chosen to engage groups of people. However, museums should also encourage intimate reflection and self-awareness. It is increasingly common to find spaces within museums where visitors can enjoy an individual experience of immersive virtual reality (VR) using head-mounted displays. However, the quality and the integration of these experiences into museum pathways is still not fully convincing. If museums can effectively leverage the challenges offered by creative languages and innovative technologies, favoring quality over quantity, they can become privileged guides and interpreters of social demands, especially those of the younger generations, while avoiding an over-reliance on digital tools. Therefore, primary recipients of this contribution are museum and cultural venue curators involved in preserving, enhancing, and protecting cultural heritage, as well as university and school students, cultural and creative industries, and, more broadly, all those people interested in creating culturally meaningful, accessible, participatory experiences that foster a sense of well-being in the individual and the community.

1.4. State of the Art

The visit to the museum must offer a unique experience based on the richness and depth of its contents: collections, high-quality audiovisual narration, interaction, multisensory immersion, embodiment, and emotion are essential elements in creating an experience and are the paradigms of virtual museums. But where do we stand today?

In Prague, on 24 August 2022, the Extraordinary General Assembly of ICOM approved the new museum definition with 92.41% support: “A museum is a not-for-profit, permanent institution in the service of society that researches, collects, conserves, interprets and exhibits tangible and intangible heritage. Open to the public, accessible and inclusive, museums foster diversity and sustainability. They operate and communicate ethically, professionally and with the participation of communities, offering varied experiences for education, enjoyment, reflection and knowledge sharing.” [7]. This is a more up-to-date definition of “museum”—coherent with current times and the ever-changing dynamics of the world of culture, in comparison with the previous definition dating back to 2007: “A museum is a non-profit, permanent institution in the service of society and its development, open to the public, which acquires, conserves, researches, communicates and exhibits the tangible and intangible heritage of humanity and its environment for the purposes of education, study and enjoyment”.

In the new definition, ethical concepts such as accessibility, inclusiveness, diversity, sustainability, and community participation are underlined. The part in which the vision and the intellectual mission of museums are explained remains unchanged, understood as places of conservation, research, and exhibition of artifacts.

A resolution by the 31th General Assembly of ICOM on 3–9 July 2016 in Milan directed the responsibility of museums towards landscape [8]: “Museums are part of the landscape. They collect tangible and intangible testimonials linked to the environment. The collections forming part of their heritage cannot be explained without the landscape. Museums have a particular responsibility towards the landscape that surrounds them, urban or rural […] The concept of Cultural Landscape incorporates not only the physical size of a territory, but also a wide range of intangible factors–from language to lifestyle; from religious belief to the different forms of social life; from technology to ways of life and production, as well as to power relations and exchanges between generations. […] Such concept encompasses soundscapes, olfactory, sensory and mental landscapes, and also the landscapes of memory and of conflict, often incorporated in places, objects, documents and images, endlessly expanding opportunities for museums to take action on cultural landscapes”.

It is clear that museums need to adopt an interdisciplinary approach in their curatorial practices and multidimensional interconnection, where the link with the digital technologies is essential for creating both global and local dimensions, as well as new real and virtual communities.

Currently, museums and cultural venues are embracing the digital challenge with increasing confidence, but the real and virtual are mostly juxtaposed in the cultural offer, separately; they do not really interact with collections. In many cases, digital content offers an accessory experience, introducing the main topic of the exhibition at the beginning of the visit path, or offering additional contents like, for instance, interviews with experts or documentation of contemporary artists, in the form of non-interactive movies (most of the time with subtitles and without audio). Multimedia is rarely used to present simulations, tell stories, and attribute meanings to the exhibited artifacts along the path of visit. The connection between museum and territory and cultural landscape is still very weak. Similarly, digital tools are rarely used to encourage interaction, stimulate creativity and astonishment in visitors, or foster a deep contact with oneself, or with the artist’s concept (except in some contemporary art exhibitions). Dialogue and interaction among users typically take place on social media rather than within the museum space. Increasingly, museums are implementing supports and arrangements to make the cultural content accessible to people with physical and perceptual impairments, but these efforts are still partial and require further development.

Since 2005, the importance of the relation between cultural heritage, human rights, and democracy, as a condition for the social, cultural, and economic growth of the communities, has been emphasised by the Convention on Value of Cultural Heritage for Society (Faro Convention) [9]. In the following years, many initiatives, aligned with the Faro Convention, emerged at international, European, and national levels, with increasing attention on accessibility. In 2019, the International Council of Museums (ICOM), in accordance with the Italian Ministry of Culture, presented the guidelines on accessibility [10], inclusion, and usability in museums at the 25th General Conference in Kyoto [11,12]. This initiative encouraged the creation of an International Committee on Accessibility addressing motor, sensory, and cognitive disabilities, as well as social fragility. As a result, many governments have recently begun supporting this issue. In December 2019, a new version of the European guideline “Accessibility requirements for Information and Communication Technologies (ICT) products and services” (EN 301549) [13] was published, to specify functional accessibility requirements applicable to ICT products and services, to be used in public procurement in Europe. Based on these principles, in 2024, the Italian National Research Council, in collaboration with the Italian Ministry of Culture and the Italian National Radio-Television (RAI), published the “Design manual for accessibility and expanded enjoyment of cultural heritage. From the functioning of the person to the functioning of cultural places” [14]. More than thirty experts, from diverse cultural backgrounds, contributed to this work, which offers targeted guidance for the functional design of cultural venues, practical case studies, and a reasoned legal summary. The manual presents a cross-cutting design for all heritage places, including libraries, archives, museums, monuments, and archaeological areas. The focus is on removing architectural, cultural, sensory-perceptual, and cognitive barriers in cultural places, emphasizing the functionality of these places rather than merely analysing the characteristics of their visitors. In this context, a multisensory and multichannel approach to communication is recommended as an essential strategy, in relation to the Universal Design principles [15].

In museums, in particular, there is still a reluctance to combine real and virtual contents in the same experiential “frame”, probably due to concerns that the virtual contribution might shatter the absolute meaning of the object on display, its authenticity, assaulting it with ephemeral values.

In recent years, the concept of “authenticity”, as perceived in museums, has been investigated by some projects and articles [16,17]. Among these initiatives, the PERCEIVE European project (Perceptive Enhanced Realities of Coloured Collections through AI and Virtual Experiences [18] has studied and detailed the concept of the authenticity of the experience, rather than of the authenticity of the artifact itself, with the goal to improve the design of digital applications in the field of cultural heritage. It has been demonstrated that authenticity encompasses various dimensions beyond mere realism: the “Self”, the “Others”, and the “World” [19]. According to [20,21,22], the perception of authenticity arises when extraneous and impersonal subjects transform into something personal and intimate, touching our sphere of emotions, memories, or beliefs, and therefore involves us. In this sense, authenticity is related to “identity” and genuine self-perception, at cognitive, emotional, and sensory levels. According to [23], authenticity emerges and evolves alongside the person, growing in depth over time. Therefore, an authentic experience is perceived as deeply personal and unique to everyone, changing over time. Vannini and Franzese, in 2008 [20], discuss how the “self” is intrinsically connected to the “others” by means of explicit or implicit communication, external or internal, in presence or in remote, synchronous or asynchronous; what matters is the feeling to be part of a common thinking. Sharing the same system of symbolic values, or social practices, enhances the sense of belonging and therefore the sense of truth and authenticity.

The “world” is the external environment, the context in which the “self” is included, in this case, the museum, or the cyberspace, or an extended reality made of a combination of real and digital entities. In this case, the perception of authenticity is shaped by accessibility, credibility, content realism, and validation. Validation is often taken for granted and implicit when people trust the notoriety and scientific credibility of the author of the contents, or of the cultural institution that presents them, even in the absence of tangible and reasoned evidence of their origin. However, validation can and should be also explicit, especially in the case of virtual objects and digitally reconstructed environments, as largely discussed in the literature, especially in relation to virtual archaeology, virtual restoration, and virtual reconstruction in cultural heritage [17].

Multimedia productions for cultural places have greatly evolved in the last ten to fifteen years in terms of aesthetics, resolution and accuracy of graphical representations, multisensory solicitation, and sensors integration. This has been made possible by the progress of methodological research and technology that have implemented hardware and software solutions that are increasingly integrated, powerful, and accessible. And there is no doubt that museum managers and curators have shown a gradual openness to the potentialities of the digital, partly overcoming initial resistance. Laboratories and creative industries working in these areas have therefore increased in number exponentially, but unfortunately, the economic models of sustainability are still highly deficient, especially in the medium and long term. Strategies and investments in the maintenance, reuse, updating of content, and modernisation of technologies are lacking in most museums.

For this reason, museums are often cautious about permanently incorporating digital technologies into their exhibitions and educational programs. Most of the initiatives are temporary and their sustainability is guaranteed in the short term. Museums often join partnerships for experimental projects led by ICT institutions or industries, mostly focused on technological aims.

In the last years, new frontiers in the domain of digital metaphors, services, or virtual experiences have opened to the wider public, as marketing automation, artificial intelligence (AI), the metaverse, NFT, blockchain, which further condition the perception of authenticity, generating new value chains. For instance, the Artemisia project—carried out in 2022–2023 by the Italian National Research Council, Digilab–Sapienza University of Rome, and the company iComfort, and financed by the Lazio Region in Italy [24]—had the purpose of analysing the experience of visiting a museum by means of motion sensors and AI algorithms to understand the dynamics of users’ behaviour in relation to cultural content and architectural, organisational, and didactic elements. It was a pilot study carried out in two rooms of the Museum of Rome in Palazzo Braschi (Rome, Italy), recently renewed. Visitors’ movements, head orientation, and gender were captured in real time, by motion sensors, in a totally anonymous form. Data were processed by AI algorithms to extract paths, and the idea was to address the results to cultural marketing strategies. In fact, the final aim of the project was to study the impact of the visit with respect to cognitive learning, customer satisfaction, and accessibility, to provide a grid of indices for the improvement of future museum visits and an advanced cultural profiling model. The project remains in the research domain, given its experimental nature, with no application in the museum at present.

In 2018, the British Museum, the first national public museum established in the world in 1753, announced its entrance into the metaverse via the Sandbox [25], a virtual 3D world with its own economy based on Ethereum [26], a decentralised global software platform powered by blockchain technology. In the Sandbox, the British Museum creates and offers Non-Fungible Token [27] objects that reflect the breadth and depth of the museum’s collections and that can be acquired and collected by investors through cryptocurrency. The British Museum also announced the creation of its own immersive space within the online game world, to explore “new and innovative ways of sharing its collection and reaching new audiences” [28]. However, after initial enthusiasm, the NFT art market suffered a setback in 2022, with a slump in prices and a drastic reduction in transactions. Significant environmental impact, a lack of clear regulations, and a lower cultural and symbolic trust in the medium were cited as reasons for the decline by several websites.

Furthermore, the emergence of new social media and paradigms enhancing cultural interactions among people has led to the creation of dedicated social platforms for cultural heritage, encouraging the active participation of a wide range of stakeholders. There is a growing demand for digital frameworks open to communities, enabling accessibility, study, participatory engagement, and the sustainable management of cultural resources and assets.

As a result, the user experience design in a museum, along with the study of museum visitors, is fundamental to understanding how the museum can affect people’s lives and to let people feel involved, intimately and collectively, through a plurality of channels, to nurture a perception of authenticity. Personal expectations, intimate reactions, group identity, meaning-making processes, and memories become part of the experience, extending beyond the physical museum visit in both space and time [29].

Today, the main challenge is to create a closer synergy and interconnection between four ‘actors’: the actual collections, the digital collections, the narrative, and the public interactions, including diverse audience profiles. This integration aims to create new scenarios of cultural experience that can last and evolve over time.

1.5. Key Factors of Multisensory Museums

An artwork is an artifact created by man, using any material, endowed with aesthetic characteristics, and imitating the natural or the spiritual reality. Every cultural artifact consists of a combination of materials, colours, and shapes (aesthetic consistency) and of a convergence of expressive values, functions, and meanings (historical values) [30,31]. Art generates beauty and can arouse emotions, sensations, and feelings, pervading our souls, bringing a feeling of harmony and happiness shared by many people. However, artistic languages and paradigms are diverse, and there is no unique universal code for interpretation. Each artwork reflects the artist’s opinions, and the social, moral, cultural, ethical, or religious context of their historical period and context, which can be lost or difficult to understand today. The reconstruction of sensory and symbolic dimensions that lie “beyond” the object’s physical appearance can take the visitor into a vibrant and powerful experience. How can this be achieved? Undoubtedly, multimedia technologies are the best means to convey contents related to cultural heritage, especially audiovisual ones, because they solicit similar processes of perception, elaboration, and learning. Mixed and virtual environments enable users to learn through experience, integrating sensory-motor and interpretative faculties, to perceive and act, even in contexts that are no longer (or not yet) materially accessible today. The alternation between and the coexistence of real and virtual contents produce a cognitive anacyclosis; the redundancies and differences enhance learning [32]. Virtual contexts can be variously assembled, dismantled, and reassembled again to explore deeper connections, and can be desynchronised to become scenarios of various simulations.

The user-centred approach makes the concept of “experience”, and therefore the user experience design, one of the main issues to be pursued in museums, together with content curation. Marc Hassenzahl [33,34] describes the user experience as a merger of perception, action, motivation, and cognition, assuming a close connection between actions, thoughts, and emotions. In fact, he defines the experience as “an episode, a chunk of time that one went through […] sights and sounds, feelings and thoughts, motives and actions […] closely knitted together, stored in memory, labelled, relived and communicated to others. An experience is a story, emerging from the dialogue of a person with her or his world through action”. An experience is thus subjective, holistic, situated, dynamic, and worthwhile. It follows that contents that should be experienced must be accessible and usable, useful and original, credible and desirable; they must satisfy a need and move emotions.

Several neuroscience studies published in recent decades [35] suggest that dreams are a form of continuous stimulation of long-term memory, throughout the course of life. A recent study at New York University, led by the Hungarian neuroscientist György Buzsáki [36], found that the conversion of everyday experiences into permanent memories occurs for a significant part when we sleep. Therefore, sleep acts on the brain as a kind of memory wipe, useful in determining which thoughts to retain as long-term memories and which to discard. The brain reacts to certain experiences with ripples in the brain waves that are then reactivated during breaks or sleep, fixing them in memory. This research further highlights the importance of the stimulation of multisensory dimensions and emotions. In fact, the dream is a psychic phenomenon associated with sleep, characterised by the perception of images and sounds, which are recognised as apparently real by the dreaming individual. It is a kind of nocturnal thinking in which the utilitarian and rational pragmatism of waking life is suspended [37,38].

Human beings open their mind to the world through intuitive experience; sensing and emotions are fundamental in life experience and in the self-identification process. They drive knowledge and individual development [39]. Every important moment in our lives, stored in our memory, has been marked by emotions. “Sensing” a cultural context also means the capacity to enter into contact with those elements that let us “recognize” something and move our emotions. W. Shakespeare wrote “We are made of the same substance as dreams, and in the space and time of a dream is gathered our short life” (W. Shakespeare, The Tempest, Act IV, Scene I, 1623) [40]. When we are embodied in a mixed reality or a virtual environment, engaged in audiovisual storytelling, perhaps adopting gamification metaphors, and interacting with spaces, eras, and stories distant from our everyday lives, we enter an imaginative status.

But which are the key elements to promote in a multisensory museum to create a credible and deep experience? Based on the author’s experience, these key elements can be summarised as follows.

2. Materials and Methods

2.1. Narration

Usually, museum curators prefer to remain “neutral” regarding artifacts, and they avoid telling stories or suggesting anything else beyond the pure evidence to visitors.

In the field of Cultural Heritage, visualisation usually aims at analysing what is still existing and observable. Unfortunately, avoiding interpretations of missing or fragmentary context is not a neutral choice: if a visitor is left alone, without interpretation and “reconstruction”, even if hypothetical, he/she may deduce false and erroneous meanings.

Learning does not arise solely from reasoning, but also through curiosity, engagement, interest, and attention; in one word, through motivation. In this process, emotions play a crucial role, stimulating a feeling of self-identification, appropriation, and elaboration of meaning. Indeed, many surveys on user experience in museums [41,42], especially referring to the use of interactive digital applications characterised by technological innovation, have evidenced the crucial role of narration in this process. Narration, in fact, is more powerful than pure description. When using narration, evocation, or even dramatisation, objects become the occasional points where history “coagulates”, creating an expectation in the visitors. Narration emphasises relationships, individual perspectives, variations, liveliness, empathy, and emotions, connecting elements within the story, even in unpredictable ways, and it draws up a specific space–time dimension [43,44]. Storytelling can stimulate the ability to reflect and generate new thoughts, for example, by referencing personal, family, or popular facts, it can activate unexpected stimuli, provoke new ideas, or introduce playful dynamics.

In a multisensory museum, narration is not limited to a written or an oral text. It refers to an expressive unity where many factors converge: oral performance/recitation, layout, visual mood, soundscape, camera movements, lighting, rhythms, smell, and taste (although the latter two senses are not yet widely represented in cultural venues and in digital media in general). It is evident how important languages are in generating emotion, involvement, and well-being.

In the creation of a narrative, certain and circumstantial contents regarding the artifact are combined with plausible and probable elements pertinent to its cultural context. Usually, story construction is a long and collective process carried out by curators and creatives, continuously refined and reshaped until the final production: once the historical or archaeological information has been acquired, a process of synthesis begins, aimed at distilling the essence. Some messages are made explicit, others implicit or subliminal [45]. Through storytelling, museums can propose a “visual drama” beyond what we can see, becoming scenarios of different simulations [46].

2.2. Interaction

If the objective is to convey educational, scientifically correct, and plausible content, while at the same time arousing strong conceptual and emotional involvement, interaction also assumes a role of great importance in making the user feel decisive in the development of the experience and in the construction of the story. As early as the late 1960s, the American pedagogue Edgar Dale demonstrated that content acquired through interactive experiences settles in our minds to a much greater extent than content of which we are passive recipients [47].

Interactions in a museum are multiple: between the user and the space, between people, between the user and the collections, between the user and technological devices, between the user and the digital content. This complex framework translates into many behaviours and cognitive dimensions that determine actions and emotions. It is therefore necessary to find a good balance between free interaction and guided experience, to optimise the effectiveness of cultural transmission and both personal and collective stimuli.

Interaction can be active or passive. It is active when it is explicit, e.g., verbal with other users, or when technological devices must be used to activate content or events, or when people act together and co-create a common experience within a real or digital scenario. On the other hand, interaction is passive when one’s mind and feelings are stimulated by a suggestive story or situation, such as the sight of a beautiful picture or listening to an evocative sound, which trigger associations of ideas and cognitive excitement. This applies to each of our five senses. Clearly, all perceptions and interactions are open to subjective interpretation and, therefore, the experience differs from person to person. To make the perceptual and interactive experience positive for most people, it is necessary to make use of usability principles recognised as shared by psychologists and neuroscientists [48], and good practices in user experience design [49].

Therefore, interaction implies good visual design and physical affordances. Thus, the concept of “affordance” is fundamental because it leads us towards the attribution of functionalities to real and digital artifacts. The psychologist Gibson [50,51] defined “affordances” as all the “action possibilities” latent in the environment, objectively measurable and independent from the individual’s ability to recognise them, but always in relation to agents and therefore dependent on their capabilities. According to Gibson, what we perceive when we look at objects is mainly their affordances, not their dimensions and properties, because the physical aspect of objects allows us to understand the principles of their functionalities. Consequently, the concept of affordance is related to the concepts of perception, usability, design, interface, interaction, shape, colour, interpretation, and embodiment.

In the case of digital content and applications, active interaction should not be unnecessary or strenuous for users; instead, it should be calibrated for various types of users, whether confident or not with interactive technologies and interfaces. To achieve the greatest naturalness of behaviour and emotional involvement, interaction interfaces should be as close as possible to those of the real world, engaging primarily sensorimotor skills. A positive impact has indeed been observed in applications of gesture-based interaction [52,53] in which the visitor must perform actual actions with the body (pointing, running, jumping, grasping…) and not symbolic actions such as those performed through a device-based system like a joystick or a game console (Figure 1). Several surveys conducted on museum audiences have shown that interaction with a digital application through body gestures, in which the user finds himself/herself at the centre of a ‘performative’ space, immediately generates in him/her (as well as in the passive onlookers) the impression of being involved in a playful situation, unusual within a museum (Figure 2). The experience is therefore mostly undertaken and lived with enthusiasm, and even difficulties do not generate excessive frustration [46,54].

2.3. Progressive Deepening of Content

With the mind and senses, human beings try to approximate the reality behind the visible appearance of things. The interior of images transforms with the interest of the perceiver: the attribution of meanings grows, and memories and personal creative dimensions become associated with them [5]. The experience gradually becomes deeper and more authentic. In the extended reality of multisensory museums, the digital content associated with an object (or a group of objects) can intensify and evolve as the user retains and focuses his/her attention on it, as if to penetrate it with his/her mind and feeling.

The evolution of the contents will proceed through various steps, depending on the time of interaction, whether passive or active, of the user with the object. In the case of passive interaction, the persistence of the user’s position in proximity to the object, or the placing of his/her hand on a sensitive surface, or the exploratory movement of the pupil with respect to the image space, will be sufficient to make the object react: this persistence of attention leads the object to manifest itself in the progressive levels of interiority.

These levels can be, for instance, as follows:

• The object is identified and made readable, e.g., through the virtual restoration of the form, if it is compromised;
• The object is shown in its original context (built or natural), as well as the landscape associated with its original location;
• The object’s practical or symbolic use is represented, along with its meaning and the message it conveyed;
• The constituent material and the execution techniques are shown, in a possible journey from macro to micro;
• The invisible contents, hidden beneath its surface, are revealed, as censures, preparatory traits, and pentimenti;
• The economic and symbolic value of the object is highlighted, emphasising its uniqueness compared to other similar objects;
• The contexts, territories, and cultures it came into contact with, during its journey in time and space, are revealed, as well as the different meanings and values with which it was invested;
• The signs and traumas that the passage of time has imprinted on the object are highlighted, including its state of preservation and the events that have occurred;
• The literary, prosaic, epic, poetic, or dramaturgical memories related to the object are evoked;
• Other objects presenting parallels and similarities in form, meaning, or value are shown, even those belonging to other cultures.

In a multisensory museum, the insight of content is expressed in audiovisual form, through the evolution of multimedia or virtual events, but also through the changing of environmental parameters, such as the intensity and colour of the light illuminating the object, and olfactory stimuli, coherently with the content that is gradually manifested.

Sound participates in this progression by going from descriptive to evocative, gradually delving deeper into timbres and transforming or distorting harmonies.

Images and narratives for each level should be very concise and essential, and the message should be conveyed in an evocative manner. Content develops vertically through the levels of depth of which the exhibited object stays on top, rather than being developed horizontally with too much information on one level. This approach can help the visitor to keep their attention alive, and at the same time, it keeps the duration of the experience limited.

The object becomes alive, multisensory, and multi-modal. It activates in the visitor/traveler an inner psychic experience, arousing thoughts, memories, dreams, a sense of wonder or anguish. Thus, understanding and remembrance are strengthened.

Multimedia contents can thus be used to reconfigure the map of the museum in which they are inserted in terms of the attractiveness of objects, rooms, and paths. In fact, they can influence dwell times, the degree of collectivity of the experience, and the level of interaction and social exchange.

2.4. Media Convergence and Combination

The attempt to reconcile traditionally linear storytelling and interaction, real and digital content in museum spaces, free exploration and guided experience, and motivating and involving the public necessarily involves a convergence of languages. Virtual reality meets with theatre and cinema, with the paradigms of video games, with holographic techniques, mixed reality, psychoacoustics, and acoustic studies. Thus, the hybridisation of media is an exciting and constantly evolving domain of experimentation.

During the museum visit, digital media should dialogue with the real spaces as far as possible: even when applying gamification techniques for public engagement, the game tasks should encourage players to search for the solution in the museum spaces and among the collections (as happens, for instance, in the classic treasure hunt) and not confine the game’s dynamics and actions solely to the digital environment.

2.5. Social and Intimate Dimensions

Dialogue is a key component in the perception of authenticity, and commonly, museum communication is addressed to a community of people.

The desire to share expectations, thoughts, emotions, and reactions with other persons, before, during, and after a visit, contributes to the community building process.

However, dialogue can also be intimate with oneself, and it can determine the desire to experience personal choices and actions, being true to oneself and independent from societal conditioning. For this reason, alongside the social dimension, it is important that the cultural venue succeeds in conveying moments of intimate and personal reflection, and of deep contact with the artwork. Spaces must support the transition to this dimension of recollection. The alternation between intimate and collective dimensions also implies a variation in the optimal duration of multimedia contents dislocated in the museum. Of course, the duration of an application characterised by active interaction also varies according to the level of technological ability of the user. Especially for multimedia content provided in the presence of the collections, along the visiting paths, it is convenient to keep the duration short and the active interaction very low, to avoid bottlenecks that could impede the visitor’s flow. Even technological devices can be light, as for instance, personal mobile devices, tablets, screens, or projection mapping beside or onto the exhibited object. The holographic showcase, as implemented in the CEMEC (Connecting Early Medieval European Collections) European project, as a mixed reality environment containing the real artifact inside [55], proved to be an excellent solution to make the museum object alive and capture the visitor’s attention and enjoyment, especially if storytelling is evocative and not purely descriptive. But in the meantime, it is useful to create a spatial and thematic connection among the different multimedia stations, to enrich the perception of the development of the story linking the different artifacts.

On the other side, solitary, intimate, and reflective experiences can benefit from a dedicated space where the user can stop without pressure. In this case, interaction, active or passive, can be more sophisticated and challenging.

Technologies that foster a solipsistic, embodied, and multisensory experience of perception will be preferred. For instance, the user can enter an immersive virtual environment, and can use a tangible interface to interact with smart objects integrating sensors that activate stories or multimedia events projected in the space around. CAVE systems, head-mounted displays, and powered exoskeletons can make contents more powerful and memorable, enhancing the sense of presence and wonder.

2.6. Mixed Reality

Mixed reality and phygital experiences are the most powerful and the most difficult to implement in a museum because they potentially bring communication to its fullest expression, not only resorting to visual media but potentially involving all senses, like sound, touch, smell, taste, even if in museums these latter two are still absent.

However, they require a respectful, credible, and efficacious balance between real collections and virtual content.

Extended reality is understood here, in a broad sense, as the coexistence of real and digital content in the same experiential space. The objects in the collection, the stories and values they express, are made explicit and enriched through digital applications; conversely, the virtual contents are substantiated and reinforced by the presence of the original objects, in a reciprocal dialogue. This alternation, or combination, produces a cognitive anacyclosis that enriches and consolidates understanding and knowledge. The term ‘extended reality’ therefore does not necessarily refer to systems in which the virtual content is mapped precisely onto the real content, as is the case with augmented reality or mixed reality.

Mixed reality (MR) is intended as not only a combination, but as an overlapping and collimation of real and digital content in the same ‘scenic’ space, so to produce a new phygital environment where the real and virtual interact coherently in the space (also in real time) and are perceived as a “continuum”. According to Milgram and Kishino [56], MR includes both “augmented reality”, where digital information and virtual objects give more meaning to real-world scenarios, and “augmented virtuality”, where real objects augment the content of artificial computer-generated scenarios.

In an MR environment, the accuracy and consistency of the overlap between real and virtual content is a crucial condition to make the perception understandable and credible. A mismatch or latency would be immediately perceived by the user as disorienting and annoying. For this reason, it is not possible to disregard a preliminary topographical and volumetric survey of the real object on which the digital superimposition will intervene, to produce a 3D model on which the virtual contents will be built, to obtain a perfect correspondence between real and virtual contents in the MR scene [55]. Obviously, the urgency of such a need depends on the type of virtual content projected onto the real objects. If it consists of textual labels, the superimposition can be less rigorous than when virtual reconstructive elements are used to complete a fragmentary real object/context [57]. There are various technologies, and technical and methodological approaches to MR. Some viewers allow the user to see through the real world (See-through AR Display and Monitor-based AR Display). It is also possible to enjoy virtual content superimposed on reality without the use of any device that comes between the user and the physical world, as in the case of video-projection mapping [56,57,58]. The choice of the device (for instance, tablet or head-mounted display) influences not only the multisensory perception, the rendering quality, the interaction, and embodiment but also the duration of the experience, the interface, the kind of media, the structure of contents, and the style of the narration. For instance, if the user is stationary in a fixed place along the path of visit, just rotating his/her gaze around him, a simpler technology with an efficient geo-localisation and orientation system is sufficient. Along the path of visit, a good solution is to mark fixed points of interest in the real space where the visitor can stop and enjoy short mixed reality experiences. Storytelling should be very concise to make the experience meaningful without being redundant and forcing the user to take time out, as the mind is active on many tasks, but also to avoid queues of visitors.

If the augmented or mixed reality experience takes place on the move, the user will plausibly not use an immersive viewer but will live the experience through a monitor display, as for instance, a smartphone or a tablet. Motion tracking applied to immersive devices is an option that can enrich the sense of immersion and presence, but, along the main path of visit in a museum, it can be difficult to manage. Moreover, not all users who wear immersive devices react positively to the solicitation of motion tracking—it also depends on the technical efficiency with which it is implemented—and for some of them, it could even cause disturbance and vertigo. Hand controllers are often difficult for common visitors to use, and support from the museum staff is necessary. Immersive devices implementing motion tracking require the user to undergo a longer adaptation time; therefore, they can be used more comfortably in dedicated spaces. On the move, tracking the exact correspondence between real and virtual content is also much more complex to manage in terms of technological and technical implementation. The domain of the cutting-edge immersive technologies of MR is very challenging from a research point of view, but not easily sustainable in museums, especially if they are proposed as permanent technological infrastructure. In fact, most people are not able to use them autonomously and often need support from museum staff, and even the powerful technological apparatus needs daily maintenance that museums are often not able to guarantee.

The holographic showcase is an example of a non-immersive “see-through AR display”. These systems are equipped with a semi-transparent display that allows a direct view of the surrounding environment. The holographic showcase usually proposed in museums is based on the Pepper’s Ghost effect [59]: the augmented content, enhancing the real object exhibited inside, is projected by an ‘invisible’ device (a monitor or a projector) positioned above or below the object (according to the setup) and reflected by a transparent panel tilted at 45° (Figure 3) [55].

A large holographic showcase allows a small group of people to stand comfortably in front of it. A smaller one enables the intimate experience of a single user. Additionally, the holographic showcase based on the Pepper’s Ghost effect, when closed on the back with a transparent glass or plexiglass panel, allows the visitor to view the object from the rear, without perceiving the holographic effect (Figure 4). In this way, the installation functions as a dual-purpose design: it is a holographic showcase on the front side and a canonical showcase on the back side. In terms of attractiveness and educational impact, the holographic showcase is an efficacious, user-friendly, and robust technology, sustainable in museums in the long term.

The holographic showcase works as a small theatre: there are spot and diffused lights inside, that switch on and off to light up or hide objects, or on their specific elements, following the storytelling. The multimedia events happening inside the showcase are organised on a multichannel timeline and controlled by a dedicated software in real time. A possible conceptualisation of the experience with the holographic showcases is outlined in Figure 5a–f.

In the author’s experience, the holographic showcase has proven to be an excellent solution for creating immersive and sustainable mixed reality experiences in museums, due to the robustness and simplicity of the hardware infrastructure, its sustainability, the harmonious and natural integration into the museum layout along the visiting paths, and its modularity and scalability. Surveys conducted in several museums to evaluate the user experience included questions on usability, appreciation of narrative style, and mixed reality efficacy. The results show the following:

• Mixed reality (interaction between virtual and real contents) confirms to be appreciated as an efficacious means of contextualisation;
• Dramatised storytelling harmonises with the magic of holograms. It brings visitors into the story and brings the object to life;
• Visitors like the novelty of the holographic showcase, but its integration in the context of the exhibition is an issue (Figure 6).

Surveys also explored the users’ understanding and retention of the content presented in the holographic showcase after their visit, with highly encouraging results (Figure 7 and Figure 8).

Finally, in the case of projection mapping (projector-based augmented reality), lights, images, and videos are projected directly onto architectural structures or objects. This technique has proven to be very effective in terms of both contextual understanding and visual impact. In this case, the audience does not need to wear any devices or physically interact with technology. Usability is therefore maximised. The equipment involves professional projectors with high brightness and durability, whose number depends on the surface area to be mapped. However, the sustainability of these technologies must be carefully assessed, especially in the case of installations designed for repeated and long-term use.

2.7. Authenticity and Languages

The sense of authenticity is generated by utility and credibility, and the progressive deepening of content. It evolves through the user’s thoughts and emotions, intertwined with a self-identification process. Factors determining the sense of authenticity also include expectation and, later, the sharing of impressions.

The languages used in communication have a very significant impact on the sense of credibility and authenticity perceived by the visitors. Traditionally, impersonal academic language is associated with reliability and objectivity and descriptions rich in technical information are rarely questioned. However, such descriptions may not always ensure full comprehensibility and contextualisation. On the contrary, when cultural messages are conveyed through artistic strategies or gamification techniques, people are more likely to question their accuracy, perceiving them as subjective, partial, or even playful. Of course, the artistic or evocative approach in the communication of cultural heritage, especially within museums, carries elements of risk, as it elicits varied reactions depending on the visitors’ tastes, expectations, and educational background. Examples of such different reactions and behaviours, from both curators and visitors, were observed in the case of the holographic showcase mentioned above, experimented during the CEMEC project in five European National museums. The mixed reality installation, based on the Pepper’s Ghost technique and containing the original object inside, employed a dramaturgical storytelling style to solicit a sense of wonder and evoke the sensory dimensions beyond the pure formal aspect of the object. In that case, storytelling conveyed the same information as traditional “scientific” supports (catalogues, panels, and so on). Moreover, it was developed together with the museum curatorial staff. However, they were told in first-person, representing “spots” of real life, with characters performing actions and expressing emotions. In that case, most visitors remembered the contents presented in the holographic showcase and correctly answered the questions of the post-experience questionnaire, but a small minority expressed doubts about their credibility, judging them to be playful and childlike [42]. It has been observed that such a difference in public reaction also depends on the geographical location of the museum and the cultural education or attitude of visitors. In a sample of approximately 600 visitors, it was very interesting to observe that museum curators in Centre and North Europe were more flexible and inclined to accept evoking and imaginative languages to transmit cultural meanings, while visitors demonstrated a more conservative attitude. On the contrary, museum curators of South Europe seemed to be more resistant, while the visitors enthusiastically welcomed a more original and exciting language and, in all cases, they felt emotionally and cognitively involved [42].

Authenticity, however, is not only related to contents but also to the whole experience and adopted languages. Can an experience be perceived as authentic even if part of the content is imaginary or uncertain? Of course, this is possible because authenticity lies in the emotions that the story can elicit, and in the progressive understanding that our cognition is moving toward a deeper truth, beyond individual uncertainties. In fact, imaginary content can serve as a metaphor, as a tracker to other authentic content or concepts. Living the experience alone or with others can also alter the perception of authenticity [19].

2.8. Sense of Wonder

As mentioned above, emotional impact and a sense of presence are fundamental to the perception of authenticity, and this is true in both real and digital experiences; unpredictable and unexpected events, and emotions such as surprise, sense of wonder, involvement, and happiness also generate a deep sense of authenticity. The beauty of the layout, the soundscape, the atmosphere of the surrounding environment, the evocative style of the script, or an engaging interaction offer us the opportunity to feel like alive participants in the magic and wondrous dimension that is ongoing (Figure 9). Triggers of the sense of wonder can be the following:

• The unexpected, the unpredictable, the surprising;
• A narrative that goes beyond the boundaries of simple standardised description;
• Unexpected movement;
• Looking with eyes other than those of everyday life;
• The rediscovery of the poetic meaning of things and actions, beyond daily utilitarian and pragmatic attributions of meaning;
• Multisensory dimensions, extended reality, “phygital” worlds;
• The sense of immanence, feeling immersed with both body and mind in a real or imaginary place (embodied sensing);
• The feeling of the connection among all beings and elements surrounding us;
• The feeling of being part of a whole;
• Empathy;
• The unexpected sharing of a feeling with other people or living beings.

Figure 9. Examples of situations raising sense of wonder. (a) Virtual interactive experience among the animated characters of “The Rule Confirmation”, a fresco painted by Giotto in the upper Basilica in Assisi; created by CNR ISPC in collaboration with E.V.OCA., BCAA, for the exhibition “Giotto’s Colours” in 2010 in Assisi. (b) Sense of wonder and embodiment of a human being in a forest, in a painted chapel, and in the landscape. Image source: Eva Pietroni, CNR ISPC.

Figure 9. Examples of situations raising sense of wonder. (a) Virtual interactive experience among the animated characters of “The Rule Confirmation”, a fresco painted by Giotto in the upper Basilica in Assisi; created by CNR ISPC in collaboration with E.V.OCA., BCAA, for the exhibition “Giotto’s Colours” in 2010 in Assisi. (b) Sense of wonder and embodiment of a human being in a forest, in a painted chapel, and in the landscape. Image source: Eva Pietroni, CNR ISPC.

There are many situations that can arouse a sense of wonder, such as walking through an ancient forest; entering a bare medieval church shaped by light and flooded with the sound of an organ; being overwhelmed by the richness of colours and shapes of a surrounding pictorial decoration; entering a dark cave; listening to the call of birds; exploring the seabed; travelling back in time; looking at a historical village; and being able to grasp the intangible plots of real and imaginary lives that are hidden behind marble, stone, and concrete.

In a museum, the sense of wonder springs from the development of connections with the various levels of one’s own mind, with other people, with objects, with time, and with territory, as if the veil that keeps things separate were to fall and their unbroken continuity were to be revealed.

It is clear how important languages and forms of representation are.

Historically, the sense of wonder has been considered as an important aspect of human nature, particularly related to curiosity and the drive for intellectual exploration. Moreover, wonder can ignite an interest that goes beyond the specific experience and becomes a reason for deeper knowledge of a given reality [60]. Wonder, according to Plato [61] and Aristotle [62], is at the origin of wisdom and thus of philosophy.

2.9. Sound and Soundscape

Acoustic design in a multisensory museum is another key factor. Sound plays a central role in the formation and development of individual and social cultural identity; therefore, sound communication is a powerful vehicle for the representation and transmission of knowledge, and for experiential involvement, on both unconscious and rational levels. According to Schafer, the sound environment is the result of the interaction between sound, space, and time [63]. However, sound is also intrinsically linked to place, cultural context, body and mind, and emotion [64]. Sound is therefore essential to increase the cognitive and emotional involvement of visitors. A soundscape can evoke the cultural identity and life dimension of the contexts to which the museum objects refer, providing acoustic verisimilitude to simulated spaces (e.g., by conveying meaningful sounds, music, timbres, and harmonies of traditional instruments or songs, as well as vocal techniques), in connection with the original places where they are practiced. Such sound compositions may be the result of the application of psycho-acoustic principles (as happens, for example, in cultural video games or cinema) or scientific simulations of acoustic spaces obtained through mathematical models [65].

Various disciplines, such as archaeology, art history, anthropology, musicology, acoustemology, ethnomusicology, acoustics, architectural acoustics, and archaeoacoustics, are intertwined and combined in the study of sounds from the past, as shown by the interesting studies published in 2020 in the special issue of the journal Acoustics “Historical Acoustics: Relationships between People and Sound over Time”, edited by Francesco Aletta and Jian Kang [66]. In addition, acoustic design in museums can also enhance the sense of authenticity because it strengthens the collective dimension of the experience, fostering in the audience a process of perceptual and emotional synchronisation towards the content, a common vibration, a sense of cohesion, and shared meaning [67].

All this implies that the sounds must be well recorded, and that the museum must be equipped with good technology for high-quality audio reproduction and management. In the author’s experience, a poor audio quality immediately portrays the unprofessionalism of the audiovisual product, even more than the imperfection of the image. Of course, if the audio is derived from an original historical recording, the authenticity value of the source prevails.

Unfortunately, within museums, there is still resistance and a lack of attention towards audio technology. Sound and music have always been considered of minor importance, if not disturbing. Insufficient attention is paid to sound technologies for single or collective listening, and their day-to-day management. Few professionals, limited investments, and insufficient infrastructure have been introduced in museums, and little research has been conducted into the acoustic design of museum spaces and sonic heritage [68]. Yet audio formats and technological solutions are manifold and can meet all the needs of audiences and curators, even with the help of sensors to control their activity and pervasiveness: stereo, Dolby Digital Surround, mono, directional sound, and binaural. An article published in 2021 by the same author of this contribution [65] includes an in-depth discussion about methodological approaches and sound technologies for museums.

Of course, the acoustic properties of the museum space, and consequently the sound design, must facilitate sense-perceptive processes, and consider the needs of people with visual or hearing impairments, as recommended by the European guidelines “Accessibility Requirements for ICT Products and Services. EN 301549”. In this regard, to be as inclusive as possible, audio content should follow some basic rules: orally recited texts should be accompanied by subtitles and translations into sign language that users should be able to activate autonomously. Additionally, the audio description of visual contents should be accessible to visually impaired persons, activated by the user on a predefined specific audio channel of the device, avoiding interferences with other audio solicitations.

2.10. Tangible User Interfaces (TUIs)

Tangible user interfaces, 3D prints for tactile exploration, and capacitive and tactile sensors can facilitate accessibility and engagement, enriching the multisensory cultural experience. TUIs are based on the tactile perception and exploration of an object, to grasp the physical properties of its surface (such as material, elasticity, viscosity, flexibility), volume, form, and function, thereby enhancing understanding of its meaning. The use of a replica of an original object (printed in 3D from a digital reproduction or handcrafted) is particularly useful for blind individuals who cannot otherwise perceive the original object, which is untouchable in most cases [69,70]. For many years, the National Tactile Museum Omero in Ancona (Italy) has been promoting “the beauty of touching, of establishing an emotional relationship with things and the pleasure of contact with different materials, the joy of discovering sensory nuances, possible uses and combinations. A beauty that can be touched, a beauty that overturns all the canons of the purely visual approach to art, to rediscover a new yet primordial relationship with nature” [71,72,73].

When linked to multimedia events, the tactile interface is usually equipped with sensors which recognise the user’s touch and determine a status change in the system, allowing multimedia content to be started.

The tangible interface can also consist of a three-dimensional interpretation of two-dimensional pictorial works, where high, medium, and low reliefs correspond to foreground and background elements in the 2D image. In the HELP European project, carried out in 2003 by the CNR in collaboration with the Scuola Normale Superiore di Pisa [74], this solution was tested to help blind users perceive and interact with a 2D painting. The audio commentary was triggered by the contact of a miniaturised three-dimensional tracer that the blind user wore on his/her finger, which tactilely ‘explored’ the form, enabling the user to mentally reconstruct the work. The audio contents were of two types: (1) sounds corresponding to the colour of the painting in that point (according to synaesthetic principles), (2) very short audio descriptions of the touched element.

However, tangible interfaces are not necessarily replicas of real objects. They can also derive from creative design, as several examples of interactive installations proposed in museums can demonstrate [75,76,77,78,79].

The technology supporting the interaction with a TUI can use two different types of sensors:

• Conductive paint and/or piezoelectric sensors: This electrical method involves direct contact with the object by the user. It is based on the conductivity of certain materials/pigments of which paints are made, and on so-called ‘capacitive’ sensors that can detect touch on their surface by generating a change in electrical capacitance. These solutions require a computer, a programmable input–output electronic board equipped with microcontrollers such as Arduino or Raspberry Pi [80,81], capacitive sensors, and/or conductive paint. Objects are electrically wired to the board’s connectors to detect their capacitance. The advantage of this technique is the high sensitivity to touch, ease of operation, and simultaneous use by several hands; the disadvantage is the alteration of the physical surface properties of the paint-treated object and the decay of the conductive properties of the paint over time.
• Computer vision: This method [82,83] allows the user’s actions to be intercepted even without direct contact with interactive surfaces, for example, by recognising the action of a hand in a specific area. For these solutions, a computer (with higher performance than in the electrical method) and a camera equipped with an infrared depth sensor (such as Kinect, Leap Motion) are required. The advantage of this technique is that it does not alter the surface’s physical properties of the object; the disadvantage is the accuracy of the input, which might be slightly lower.

The result of active stimulation by the user is the reproduction of audiovisual contents, for which a monitor (or video projector), audio speakers, and a scenic lighting system are required. In general, users should be guided by lights, lines or colours, and patterns in relief circumscribing the interactive areas within which they must operate and focus their attention, making it easier for them to identify objects and interfaces.

TUIs should also be located at an appropriate height and should be easily accessible for users on wheelchairs.

Information on the TUI, such as captions describing the different elements, should also be translated into Braille.

2.11. Digitisation

The sense of authenticity, in museums, is particularly relevant when dealing with digital 3D representations of real objects, in both their present and past possible appearances. In the first case, an accurate and validated digitisation procedure is needed; in the second case, it is important to respect the criteria of reliability, truthfulness, transparency of sources, and the interpretation processes that support virtual restorations and virtual reconstructions [84,85].

A digital object, considered as a reproduction of a real object, represents an approximation of its form and appearance, and makes it possible to preserve its knowledge and memory even if the original is lost, provided that this digital object is obtained through a rigorous methodology, as accurate, detailed, faithful, neutral, and complete as possible.

In addition, through 3D printing techniques, a material replica can be obtained from the digital one, which can represent some of the properties of the lost original object and can be manipulated or relocated to restore fragmented or missing context.

However, as mentioned above, by virtue of its immateriality, digitisation should not be limited to the formal approximation of an object as such. It is useful to extend the concept of digital object to that of “digital content”, i.e., an expressive unit endowed with form and meaning, that can communicate both the function, context, and cultural value of the object and the information necessary to understand how the digitisation process was carried out (metadata, paradata).

The realisation of a digital model following such criteria of objectivity and truthfulness makes it possible to associate reliable integrations of form (virtual restoration), contextualisation in relation to the original place for which it was conceived (virtual reconstruction), and the values and meanings attributed to it throughout history (semantic characterisation, narration).

The digital twin is even more: it is a virtual simulation of a physical entity including not only the appearance and meaning but also the behaviours of the object in relation to the ecosystem. This is made possible using sensors, actuators, an internet connection, and software control allowing information exchange between the virtual (cybernetic) and physical components, to do tests, monitoring, and maintenance in real time [86]. Heritage digital twins are also understood as digital replicas of CH objects linked to all associated knowledge, interpretative levels, attributions of meaning, relations with other items, interactive processes, and digital documentation [87]. Digital twins can integrate the Internet of Things, artificial intelligence, machine learning, and data analysis. They can create digital simulation models that update and change when their physical counterparts change [88].

These actions are highly desirable to foster in-depth knowledge and understanding of the cultural asset, useful for prevention and protection, and for valorisation addressed to the public. Digital content can thus be the starting point for learning scenarios aimed at diverse audiences. A new economy could develop thanks to shared and conscious digitisation practices, such as the creation of digital libraries and services.

The digital object, accompanied by the necessary information, can thus become part of virtual collections, enriched by storytelling, following procedures of ‘loans’ between museums. Virtual collections can complement real collections already on display, complete them in cases where apparatuses or contexts are physically dismembered and scattered in several places, and establish connections or comparisons among objects of various provenance but linked by a common theme.

The digital model should hopefully adopt the principle of open science, especially when the digital resource or its derivative is used for non-commercial purposes, such as study and scientific research. Digitised data could be released, even for a fee, by governmental institutions for profit purposes, for the benefit of creative cultural industries.

This scenario requires improved curatorship to manage the continuous renewal of the museum communication strategy.

Finally, the application of FAIR principles [89,90] to digital cultural heritage, shared and promoted at the European level, is an essential condition for the sustainable life cycle of digital resources, capable of generating new cultural, social, and economic value. Data, in fact, should not only be produced, but also updated over time, shared, and properly reused, feeding the creation of new cultural content. According to this perspective, the adoption of FAIR principles is at the basis of the creation and maintenance of quality data, capable of guaranteeing easier interaction between the actors who, in different ways, play an active and creative role in the transmission of cultural heritage.

2.12. Representing the Invisible

Digital documentation, representation, and valorisation of museum artifacts of particular interest could include both visible content and elements that are not visible, hidden in the structure or in the sub-surface levels of the artifact. This can be the case of the preparatory drawing of a painting, or elements that served the execution process and remained incorporated in the structure, materials coming from a restoration work, or alterations resulting from censorships. The characterisation of the chemical–physical–biological nature of the materials at the different stratigraphic levels can reveal interesting information related to the execution technique, the conservation history, and the present state of conservation, and it helps to understand the material value of the object [91]. This approach could be innovative, aimed at creating a multidisciplinary experience with the artifact and its production context, craft skills, and workshops. All data could be integrated into a multidimensional model of the object (a 3D model to which the fourth dimension of depth can be added and explored), taking into consideration tangible and intangible values.

For such a multidimensional model, superficial information of an artifact can be captured via laser scanner, or via cameras and then elaborated through structures from motion and photogrammetric techniques. Instead, invisible content associated with the sub-surface layers can be acquired through non-invasive diagnostic techniques and sensors, such as (1) pulsed thermography, which produces images in the medium infrared range, able to reveal hidden elements or detachments beneath the surface, (2) X-ray fluorescence, (3) Raman spectroscopies, (4) hyperspectral imaging, and multi-band imaging techniques such as ultraviolet, (5) Vis-NIR reflectance, (6) reflectography [92]. These techniques are chosen and combined according to the material aspects to be investigated. Then, all the relevant information can be mapped onto the 4D model as “annotations” and as informative/semantic spots. Data interpretation and interrelations can help in the reconstruction of the complex story of the artifact. The annotated multidimensional model can then be explored through interactive installations of virtual reality or mixed reality [92].

2.13. Virtual Reconstructions and Authenticity

A virtual reconstruction in archaeology entails the digital restitution of an artifact at the time of its creation or at its successive phases of use. However, in this domain, virtual reconstructions are approximations, tools for better understanding the past and not statements of reality [93]. Usually, they are possible reconstructions, especially when dealing with lost ancient contexts. Virtual models are just simplifications, resulting from a selection of information, useful for interpretations [94]. A virtual reconstruction, in fact, can help scholars to understand structural solutions, working as a verification tool, and it can always be updated in light of new discoveries.

Again, in archaeology, a virtual reconstruction is usually the result of an integration of bottom-up and top-down approaches: the first one refers to the digital documentation and analysis of what remains on the site; the second one consists of the collection, study, and interpretation of historical, iconographic, literary sources, architectural rules, and proportion theories, including comparisons with similar case studies [93]. Virtual reconstructions let the public better imagine and understand the original context of the exhibited artifact, its function, value, and properties, giving concreteness to abstraction. As it was assumed at the beginning of this contribution, the final goal of virtual heritage is the interaction process, the semantic value, and the cognitive incitement that develop from this interaction.

Rendering techniques, contents, metadata, visualisation technologies, user interface, and investigation tools depend on the different audiences they are addressed to, even if they should always follow scientific consolidated criteria. The experience will be more analytical, with a focus on the structure and its elements, and construction materials or executive techniques, with connection to related databases, in the case of an expert audience [95]. On the contrary, it will be more dramatised, sensory, narrative, or playful, in the case of a non-expert public that must be introduced to the cultural context, the way it was used, its life dimension, and historical background. In this case, a holistic approach and embodiment prevail over analysis, and exploration tools and metaphors will be calibrated accordingly, while maintaining continuity and coherence between scientific sources, knowledge, and communication [96].

Today, the realism achieved through digitisation and computer graphic techniques can easily lead the public to believe that a virtual reconstruction is truth, especially when presented in a museum and created by a renowned author. In fact, a critical aspect with this kind of reconstruction is scientific transparency. In most cases, it is very difficult to distinguish what is original and what is hypothetical because sources and interpretative processes are not declared, nor made accessible.

Truthfulness requires a verification process. The London Charter, in 2006 [97], and the Seville Principles, in 2011 [98,99], attempted to regulate the authenticity and reliability of virtual reconstructions in archeology, recommending some rules and best practices to make the sources and the logical processes of interpretation explicit, plausible, findable, sustainable, and repeatable. Otherwise, the risk is to reduce virtual reconstruction to mere sensationalism.

Starting from those criteria, many developments and applications followed in the research domain [100,101,102]. For instance, many solutions have been found to distinguish the different degrees of reliability of single elements or portions in reconstructive models. These elements must be indicated through graphic expedients that allow the recognisability of the interpretive intervention [103,104].

Also, in the creation of a narrative related to cultural heritage, the story must be plausible and credible, as the museum is an educational cultural institution. Certain and circumstantial contents regarding the artifact will be combined with plausible and probable ones, the latter pertinent to its cultural context. To suggest the reliability of each segment of the audiovisual narrative, the author has introduced, since 2021, a kind of visualisation that recalls the methodology traditionally applied to highlight the reliability levels of the different elements in architectural virtual reconstructions [17]. This visualisation assigns a different symbolic colour to each level of certainty/uncertainty, based on the available sources (Figure 10). Thus, in this colour code of the narrative content reliability, these symbolic colours can be superimposed on the final rendering to give scholars an immediate overview.

In museums, documentary sources and logical interpretative processes followed to create virtual reconstructions must be referenced. They can be integrated into the main application for the general public, as an in-depth analytic layer [105,106], or they can be proposed in a separate, complementary application for expert visitors [107].

2.14. Embodiment

A key concept in the experience is “the sense of presence” [34], which relates to the cognitive and emotional involvement of the user in the surrounding environment. Humans open their mind to the world through intuitive experience; sensing and emotions are fundamental in the life experience and in the self-identification process, they are the engine of knowledge and development of the individuals [39].

“Sensing” and feeling “embodied” in a cultural context means the ability to connect with those elements that let us “recognize” something and move our emotions. For instance, an evocative landscape, an ancient bare church resounding with a choir, and organ and trumpets, and a multisensory museum have the power to activate a perceptual phenomenon that allows us to feel an intimate sense of happiness and enjoyment difficult to verbalise [31].

The embodied cognition approach affirms that mind and body are not separate and distinct, as Descartes mistakenly thought [108], but that our body, and the brain as part of the body, contributes to determining our mental and cognitive processes. Some classical philosophies, such as Platonism, Augustinianism, Cartesianism, and the Kantian conception of the intellect, revealed a tendency to make the mind totally other than the body, without considering their interrelationships, or even their unity, with a persistent devaluation of the body. Reflection on the corporeal and embodied aspects of cognitive and mental processes has taken place since the 1980s in various disciplines, from cognitive linguistics to artificial intelligence, from neurobiology to phenomenology, and has become central in philosophical research on mind and cognition between the 1990s and the first decade of the 21st century. The theory of the embodied mind was presented on biological grounds in the classic study by F. J. Varela, E, Thompson, and E. Rosch [52] with a focus on the conception of phenomenology [109].

The theory of the extended mind proposed by A. Clark and D. Chalmers [110] focused on the constitutive and causal role of the physical environment in the formation of mental processes. It extends the mind not only beyond the brain but also outside the skin and the body itself, at the centre of environmental interactions but also of social relations with other cognitive agents. G. Lakoff and M. Johnson emphasised the operational aspects of corporeality in the origin of mental states and language, starting with the use of linguistic metaphors. In neurobiology, the role of the bodily component in the origin of emotions, consciousness, and empathy has also been highlighted [111].

According to Daniel Mellet-d’Huart, knowledge in virtual environments comes from “enaction” and it is built essentially on sensory motor skills and direct experience in the 3D cyberspace [112]. In a museum conceived as an extended reality environment, and in virtual reality as well, embodiment can arise from the following:

• Immersion in an interactive multidimensional space;
• Multichannel and multisensory stimulation;
• Graphic realism;
• Tangible user interfaces, object manipulation;
• Creative body interaction and freedom in movements;
• Mid-air gesture-based interaction;
• Feedback from the environment in real time to user’s actions and exchange of information with the environment;
• Possibility to modify the layout/the setting/the assembly of objects in the space;
• Engaging and emotional storytelling, dramatisation, cinematographic and theatrical techniques;
• Augmented reality;
• Emotional and meaningful soundscape;
• Interaction with other users;
• Personal evolution in the digital ecosystem [113];
• Randomness and unexpected events.

The latter influence positively the sense of presence, because they enhance the realism of the interaction and the complexity of decision processes, as happens in real life. For this reason, artificial life is often introduced in videogames, and is considered a promising frontier in the domain of virtual heritage simulations [114]. In fact, through the experimentation and interpretation of new relations/differences, we can process information, and learn and develop an appropriate epistemological approach towards ourselves and the surrounding world.

The emergence of new social paradigms and media enhancing cultural interactions among people induce the creation of specific social platforms for cultural heritage that encourage the active participation of many stakeholders. There is an increasing request for digital frameworks open to the communities for accessibility, study, and the participatory and sustainable management of cultural resources and assets [115]. In recent years, many museums have created their 3D extension in the metaverse, where interactions are free and more creative, open to the co-creation of contents, multiplication of perspectives, and informal representations of meanings. In the metaverse, users can recognise themselves through avatars or through their real image, while acting in real time. The embodied communities live in a 3D space; users can perceive, meet, dialogue, touch objects, build and transform the surrounding virtual environment, make and simulate hypotheses, or perform actions following a common a purpose. The virtual collaborative environment is thus conceived as a performative space [116]. The level of engagement in the cyberspace is an essential condition to generate embodiment.

2.15. Profiling Target Audience

“Personas” are fictional characters based on real field data. Each “persona” represents a specific type of audience and serves as an archetype bringing together users with similar characteristics. Referring to this typology using a name and an image helps to make profiles concrete and refer to them in a practical and shared way. “Personas” help designers, curators, and ICT experts understand the importance of the user-centred approach [117].

Data about the public refer to the collection and analysis of information regarding visitors’ and potential visitors’ attitude and preferences. These data help museums to understand their audience, enhance visitor experiences, and tailor their services.

Data gathered from the public include demographic information (gender, age, provenance, education, occupation), behavioural information (visit frequency of museums, main interests within the museum, visitors’ movements and pathways, aptitude for lone or group visits, level of attention, timing), information about satisfaction, engagement with digital contents, ability to control digital devices and digital interfaces, level of interaction, demand for assistive technologies, and aptitude to share the experience with others.

User profiling can be performed using classical manual mapping methods, via sensors and more recently, through artificial intelligence.

Manual mapping includes surveys recording data directly at cultural sites or collecting statistical data from tour operators and tourism agencies. In museums and cultural venues, interviews can be carried out directly on visitors.

Some types of information can be collected more objectively using sensors. For instance, visitor movements in a museum can be tracked using Radio Frequency Identification (RFID) [118,119], beacons [120,121], Wi-Fi, and other infrared or Bluetooth sensors which enable automatic features to be activated as users pass through. These are various sensor systems composed of transmitters and receivers that assign an electronic identity to things surrounding us and form the basis of the Internet of Things (IoT). They allow objects, people, and content from the real and the virtual worlds to be connected, to create smart and inclusive environments capable of adapting services to the needs of those who visit or work in the museum. Examples of the application of such systems in museums can be found in [120,122].

As mentioned above, “personas” make it possible to identify the characteristics, needs, and desires of potential users. Artificial intelligence (AI) is revolutionising the field of marketing. Companies are harnessing its potential for analysing and targeting their audiences, achieving better results in a shorter timeframe and on a large amount of data (big data). Through market research and real data, AI can create detailed profiles, modelling “personas” in terms of origin, age, gender, goals, motivations, interests, aptitude, needs, habits, movements, and friendships. Promotional and orientation campaigns will be increasingly targeted to the individual, and no longer on population groups. This information derives from what we enter into the network via social media, online database entries, or dialogues with ChatGPT-4 (https://openai.com/index/gpt-4/ accessed on 26 March 2025). Artificial intelligence uses these data and connects them, analyses them, and predicts what the ‘consumer-user’ might like and proposes it to him/her. The cultural market (of which museums are part) can make use of these new developments to personalise their marketing messages and to create targeted strategies, creating customised and satisfying contents for their target “personas”.

The implications of these developments are many, interesting, and difficult to predict. On the one hand, the user is framed and treated according to the stereotypes and homogenisation logics of the consumer society, becoming a “consumer”. Personal habits and characteristics are mapped and traced, with a consequent compression of privacy. Basically, all people matching a certain profile will receive from AI similar answers to the same questions. By now, everyone accepts, more or less consciously, that an artificial intelligence decides what they will learn from journals and informative digital media. Recommendation algorithms, which were created to make it easier for users to find the information that best suits their personal interests, have also learnt over time to suggest the messages that best meet the platform’s needs, i.e., to keep people engaged with the platforms as long as possible, to gain their attention as intensively as possible, and to resell that time and attention on the market for personalised advertisements. The risk is that this could provoke the spread of various forms of social network addiction, by also using fake news and ‘deepfakes’ (a technique for synthesising the human image based on AI), produced by other artificial intelligences. Are we sure that giving the visitor what the algorithm has identified as fitting his/her profile is a benefit for the person? Or does it produce repetition and boredom, extinguishing the potential renewal of his/her cultural interests? AI does not behave objectively but tends to benefit the companies that produced it. One wonders what impact this technology will have on the economy, society, religion, and politics. Artificial intelligence can strongly influence knowledge and the relationship with truth, behaviours, and social goals and tendencies towards polarisation, collective processes of decision-making, and the relationship with freedom.

On the 17 February 2024, the Digital Service Act came fully into force in Europe [123,124]. This is the first comprehensive regulatory intervention in the world that protects users of online platforms against the abuse of recommendation algorithms.

Which use will the tourism and cultural industry make of AI in the near future? AI could enhance human creativity, being able to store networks of human and non-human knowledge, create new associations of though, and thus generate new, unexpected narratives about the world. It is possible that through an intelligent, independent, and generous use by museum curators and scientists of the opportunities that artificial intelligence offers us, a richer, more creative, and inclusive cultural communication could be developed, in which the sense of authenticity and the opportunities for individual expression, through the irreducible values of culture and imagination, can be preserved.

This is a dilemma in our society. In the field of cultural transmission, artificial intelligence is a big issue for the future. We must balance optimism with awareness of the risks that can come in many forms, either through unintended uses of new technologies or through malicious actions seeking to exploit areas of vulnerability.

Even if AI can certainly be used to offer many useful services to the citizens, it can, at the same time, turn into a weapon of manipulation and mass homologation of public opinion, now firmly in the hands of a few, gigantic, private software companies. In this regard, it is interesting to note what Mario Draghi wrote in his report on competitiveness, presented on the 16th of September 2024 to the European Commission, where he outlines a European strategy towards AI [125]: <<A critical issue for Europe will be integrating new technologies like artificial intelligence into our industrial sector. AI is improving incredibly fast, as the latest models released in the last few days show. We need to shift our orientation from trying to restrain this technology to understanding how to benefit from it. The cost of training frontier AI models is still high, which is a barrier for companies in Europe that don’t have the backing of US big tech firms. But, on the other hand, the EU has a unique opportunity to lower the cost of AI deployment by making available its unique network of high-performance computers. The report recommends increasing the capacity of this network and expanding access to start-ups and industry. Many industrial applications of AI do not require the latest advances in generative AI, so it’s well within our reach to accelerate AI uptake with a concerted effort to support companies. That said, the report recognises that technological progress and social inclusion do not always go together. Major transitions are disruptive. Inclusion hinges on everyone having the skills they need to benefit from digitalisation. So, while we want to match the United States on innovation, we must exceed the US on education and adult learning. We therefore propose a profound overhaul of Europe’s approach to skills, focused on using data to understand where skills gaps lie and investing in education at every stage. For Europe to succeed, investment in technology and in people cannot substitute for each other. They must go hand in hand>>.

2.16. Evaluation of the User Experience in Museums

It is not easy to create content and experiences to satisfy the heterogeneous museum audience. It is very important to study museum audiences in advance—establishing where they come from, their degree of cultural and technological literacy, the age groups to which they are most likely to relate—to design the best types of experiences.

It is also very important to assess and understand how, within cultural venues, the audience experiences culture. The aim of these evaluations is to explore the effectiveness of exhibition design, the attractiveness and usability of digital technologies, the level of engagement, social and individual behaviours, the educational impact of media, and finally the visitors’ ability to critically process and remember content. One of most efficient methods the authors have experimented [41] with is a structured, multipartitioned analysis on various user targets, based on three steps: observation, filling in questionnaires, and guided scenarios, i.e., tasks that the evaluator asks the user to perform, while interacting with the digital application, to understand whether he/she is able to control the system.

To be indicative of a trend, the data collected must generally come from a sample of several tens or hundreds of visitors, as this is the only way to compare the expectations of ICT designers and developers with the actual reaction of the public.

Feedback from the public helps to verify, understand more deeply, and improve the effectiveness of the content, the languages and media adopted, the interaction interfaces, the technological solutions, the layout of the space, the routes, the duration of the experience, the group dynamics, and of course the relationship between the real collections and the digital content offered in the museum. What emerges is useful not only for researchers but also for curators and museum staff, as it allows them to gain experience and awareness, to better address critical issues and extraordinary opportunities that the combination of real and virtual contents can offer.

2.17. Use of AI in the Cultural Heritage and in Museum Sectors

Artificial intelligence can be used in the cultural heritage and museum sectors for various purposes.

In the User Research studies, it can support

• Understanding target audience and users;
• Creating design brief.

AI-driven sentiment analysis is a powerful tool to analyse and interpret visitors’ comments, as natural language processing can gauge satisfaction, emotions, and key themes, providing valuable insights that would take months of manual analysis to be achieved.

Software like QoQo [126], UserDoc [127], and Design brief Generator [128] use AI to generate UX personas, journey mapping, information architecture, summaries, sitemap and wireframes, and clusters of similar ideas based on common themes and patterns; they provide tools to organise requirements with folders, labels, and relationships, or are aimed at format conversion.

In the domain of big data analyses and improvement, artificial intelligence and machine learning can offer the possibility to automate and sometimes enhance a number of manual processes involved in the collection, processing, and metadata management of texts, 2D images, and 3D models. As Europeana’s recent survey on the role and impact of AI in the digital cultural heritage sector showed [129], there are numerous projects underway with the aim of increasing the efficiency of the following:

• Digitisation, 3d models fixing, denoise;
• Resolution-enhancing processes;
• Classification;
• Restoration or reconstruction of CH assets;
• Deep learning and text analyses (transcription services, keyword extraction);
• Accessibility and automatic indexing of data;
• Archival management.

The interest in these practices is often linked to the search for new ways to facilitate the production and exploitation of digitised collections, given the potential of AI technologies to work with large amounts of data. The interpretability and generalisation of AI results are emerging issues.

In the generation of virtual scenarios, virtual objects, and digital collections, there are also several potential uses of AI. New artificial intelligence or light field rendering algorithms [130] make it possible to generate and process 3D data or to represent real objects through techniques that go beyond direct instrumental measurements on the field. These techniques focus on representing all visual properties of a scene without generating an explicit and measurable representation (as in 3D digitisation) of the scene itself. Through these approaches, several static images can be used to construct a 3D representation and generate images from new viewpoints. Technologies such as NeRF (Neural Radiance Field, based on neural networks) [131] or Gaussian splatting [132] are examples of these techniques. These techniques take photographs as input and initially perform the same orientation step as a normal photogrammetric digitisation. However, at present, they are mainly used for visualising 3D scenes rather than for metrically rigorous documentation.

The application of artificial intelligence and machine learning neural networks, combined with the human knowledge provided by the experts, also offers a significant opportunity to improve the quality of metadata and their use on a large scale, as, for instance, is expected in the European Collaborative Cloud for Cultural Heritage [133].

AI can also support the generation of storytelling and audiovisual contents.

These solutions belong to the domain of natural language processing. A natural language is any ordinary language that occurs naturally in a human community by a process of use, repetition, and exchange without conscious planning or premeditation. It can be a spoken language or a sign language, and it differs from constructed and formal languages such as those used to program computers or to study logic.

Computers have been provided with the capability to process data encoded in natural language (Natural language processing—NLP). NLP is a machine learning technology that gives computers the ability to interpret, manipulate, and understand human language, thanks to the huge amount of voice and text data coming from various communication channels such as text messages, social media, news, video, audio, and more. To automatically process these data, NLP software analyses the intent or sentiment in the message (through speech recognition algorithms, text classification, natural language understanding) [134,135,136] and responds to human communication in real time (natural language generation) [137].

The origin of natural language processing dates back to the 1950s when Alan Turing published in the magazine “Mind” an article titled “Computing Machinery and Intelligence” [138], which proposed a test (known as “Turing test”) to demonstrate the linguistic and cognitive capacity of a machine as a criterion of intelligence. It is a test of a machine’s ability to exhibit intelligent behaviour indistinguishable from that of a human, through the simulation of a conversation. The conversation was limited to a text-only channel, such as a computer keyboard and screen, so the result did not depend on the machine’s ability to render words as speech, nor on the machine’s ability to give correct answers to questions, but only on how closely its answers resembled those a human would give [139]. The breakthrough came when AI learnt to reason using a probabilistic approach, in the 1990s. Decisive improvements were progressively enabled by the availability of data, supercomputing capacity, large data centres for cloud computing, and new algorithms designed in the new millennium. In 2022, GPT (Generative Pre-trained Transformer) [140,141] was released, with the ability to put together sentences with excellent syntax and perfect punctuation. The new generation of Chat GPT, recently released by Open AI and equipped with ‘Voice mode’, is able not only to speak, but to whisper, laugh, and sigh like a human being, giving all expressive and sentimental inflections to speech and approaching any topic from various perspectives.

We can imagine that in the near future, 3D AI characters will be capable of interpreting not only our questions but also our emotions and reacting accordingly, providing information in multiple languages, conversing with us, expressing emotions in turn, performing actions, similarly to humans, thus making human–AI interaction more natural and empathic. Digital humans are already being developed; an example is given by the Algho [142], a No-Code conversational AI platform composed of proprietary technologies to create Virtual Assistants in business processes. While their use in museums has not yet been applied, a growing market will likely develop, and many uses can be envisaged for such digital humans, such as multilingual assistants, instructors or companions, or even LIS interpreters, including in museums.

But is it really intelligence? According to Viola and Biase [124], today the answer is no. Intelligence means the ability to synthesise, to invent something from nothing and not from something that exists, the ability to assimilate and then create something totally new. Language, although extraordinarily powerful, is a communication-oriented simplification of human thought. All the cognitive activities of humans are not included in language. Today’s machines cannot, therefore, be considered intelligent.

Generative artificial intelligence algorithms do not make choices based on value chains or conventions, but only probabilistic choices.

The machine imitates human language, not by doing any logical operation in deductive sense, but by setting up a linguistic operation. It considers (1) the rules of linguistics as probable combinations and (2) the data it has at its disposal to concatenate words. These capabilities are not inventive but emulative. The machine has no critical sense, it just assembles words correctly. Although the machine will certainly develop further, gaining the ability to connect logical concepts, this scenario is not possible at present [124]. Today, not only are newspaper articles increasingly written by AI, but travel and tourist guides are too, and they are often reviewed by other AIs. Today, authors using AI are those who write large amounts of content because AI allows them to save time. Perhaps asking for suggestions and ideas instead of direct answers is the best method to use AI, while declaring human or artificial authorship of content is a must. Copyright is indeed another open issue that needs to be regulated. AI tools can also provide creative teams with stories, screenplays, casting and character profiling, sounds, images and movies, and voices. Prompts provide the seeds to help ChatGPT deliver consistent, relevant, and engaging content based on the relevant use case. Practically, prompts are textual instructions or images that we provide to AI, guiding it towards the desired output. AI consists of neural networks that rely on billions of images it has been trained on to render true-to-life images, by means of different algorithms, in particular GANs and Diffusion Models [143]. The number of tools for creating images has multiplied day by day; some are free (as OpenArt or Canva) [144,145], some require payment in their advanced version (as OpenAI DALL-E) [146], some are websites (as Midjourney) [147], and others are stand-alone software (as Adobe Firefly) [148].

A similar approach is also adopted by AI for video and sound generation: available platforms translate text prompts into video or musical compositions. The description of such platforms is beyond the scope of this article, which is limited to providing a general overview of the AI potentialities in the creation of cultural, narrative, and multimedia content, which is also applicable to the museum field. There are still some technological limits. AI creates content based on existing data and patterns but it lacks real insight or originality. It cannot generate content on topics it has not been trained for, nor can it update itself without new data. In long texts, it may lose the thread of discourse, repeating or contradicting itself. Artificially generated images and videos are still imperfect in coherence and realism, as they can present anatomical distortions, and unnatural movements. The generation of video requires a lot of computational power, and continuity between frames can be compromised, leading to distortions or ‘flickering’ between images; music can be repetitive and without a well-structured harmonic progression. However, AI technology evolves very rapidly and limits will be completely overcome in the near future.

As AI technology advances, the museum industry has been increasingly integrating AI to enhance both visitor experiences and operational efficiency. In 2016, an exhibition curated by a group of anthropologists at the Musée du Quay Branly in Paris presented Joe Berenson, a robot playing the role of an art critic, created by Denis Vidal and robotic engineer Philippe Gaussier [149]. However, the robot was easily influenced by the public, as its objective was not to express its thought about the artworks, but to sample and determine, through the artificial intelligence algorithm, the reactions of people observing the artworks around it. It was an excellent tool for mediating with the public and a valuable aid for conducting surveys on the composition of museum and art gallery audiences.

Artificial intelligence also plays a role in audience engagement. In 2018, Google’s Arts & Culture launched Art Selfie, a web app that used facial recognition to match users’ selfies to portraits in museum collections [150]. AI applications have been conceived and designed for museums, to bring historical figures to life through chatbots using their writings, archives, and oral histories [151].

Beside the useful and surprising applications of AI in the cultural sector, there is also the serious risk of strengthening forms of discrimination and marginalisation. This can happen essentially because AI models are trained on historical data that may contain biases as they are collected on groups of people that exclude or penalise minorities as, for instance, women or certain ethnic groups. Furthermore, AI-based systems require an internet connection and devices with high computing power. Such expensive devices may be inaccessible to the less wealthy people or to populations living in rural areas, with poor telecommunication infrastructure. Rural populations, as well as elderly people, may not be able to access AI-based systems due to their insufficient familiarity with digital technologies. AI-generated content could therefore exacerbate social inequalities and misinformation in the interpretation and transmission of cultural and historical events. There is a need to develop practices and rules for fairness, transparency, and intellectual property. Training data must be derived from sources that are as diverse and representative as possible of the different social instances and cultural perspectives. AI development teams must be multiple, distributed, and managed by research institutes, universities, and public cultural institutions, rather than being concentrated in the hands of a few giant corporations and technocrats who, in agreement with the political and financial powers, can spread a partisan culture, undermining democratic principles on a global scale. Moreover, human control in decision-making processes in sensitive areas is an essential ethical duty, as well as in the verification of the fairness and reliability of cultural messages diffused by media and cultural institutions.

In 2019, the EuropeanaTech Community established a GLAMs Task Force (Galleries, Libraries, Archives, Museums) to conduct a horizon-scanning exercise and investigate the role and impact of artificial intelligence (AI) and machine learning (ML) in the cultural heritage sector. The final report [129] provides an overview of a survey that received 56 responses from cultural heritage and research institutions. The purpose was to create an informative basis to facilitate the innovative, ethical, and sustainable growth of AI technologies within cultural heritage. These institutions are currently increasingly interested in AI technology, although it is perceived as being too time-consuming for their organisations, with benefits coming only from the application at a large scale.

From the results, the following emerges:

<<(Meta-)Data quality is the topic for which people have more practical experience with AI (29.1%), followed by Knowledge Extraction (29.1%), Collections Management (21.8%), Discovery and Search (21.8%). (Meta-)Data Quality is also the topic in which most people are “very interested” (65%), followed by Knowledge Extraction (59.3%), Collections Management (61.8%), Discovery and Search (60.0%). The least interesting topics for our respondents are Machine Translation (20.0% “not interested”), followed by Audience Analysis (12.7%), Crowdsourcing and Human in The Loop (10.9%), and Creative or Engagement Projects and Initiatives (10.9%). […]

Additional areas suggested by the respondents that can be considered interesting for them or for which they have experience with are Layout Recognition, Photogrammetry Automation, Production of 3D Content, Data Extraction (e.g., Optical Character Recognition (OCR) and Handwritten Text Recognition (HTR)), Music Information Retrieval, Collection Content Analysis, Semantics (Linked Data, Knowledge Representation), and Visual Recognition (object, subject, colour of image/video). GLAM institutions are most interested in using AI for facilitating the exploitation (and to some extent, the production) of their digitised collections. By digitising their cultural heritage objects, they can improve the accessibility of these objects to the public, who might be able to access them via online portals. Once the objects are digitised, their metadata need to be enriched for improving findability and searchability […]. Several interviewees pointed out the limitations of commercial AI tools, in relation to the complexity of the nature of CH data and use cases. […]. The consulted professionals agree that AI has great potential within their organisation and more broadly in the GLAM sector. However, AI and Machine Learning (ML) have not been applied widely so far. The attraction for the skilled, often technical people, to work in the CH sector might not be strong enough yet. The complex nature of the collections is challenging, as the data in this domain possibly involve multiple symbolic, allegorical, and contextual interpretations>>.

3. Results

This section will present some applications and projects representative of the principles expressed in the previous sections, particularly in relation to multisensory experiences in museums that enhance accessibility, understanding, and visitor engagement with content. The author of this paper played the role of project coordinator or participant in each of these projects, as well as in those ones mentioned in the previous figures.

3.1. CNR Anniversary: 100 Years of Science

In 2023, the National Research Council, Italy’s largest scientific research institution, celebrated 100 years since its foundation. On this occasion, various initiatives were organised to retrace and present its history, the impact of scientific research on civil society in Italy and around the world, and the future perspectives and challenges. From October 2023 to September 2024, the CNR headquarters in Rome opened its doors to citizens, students, tourists, researchers, and their families, offering a multimedia experience that was narrative, playful, and educational at the same time [152]. A multisensory installation positioned in the entrance main hall was conceived and realised following the principles of inclusiveness and participation, and stimulated the visitors’ curiosity through tactile interfaces, multi-projections, sensors, and multimedia (Figure 11).

The designers, in fact, aimed not only to convey scientific content but also to interest, surprise, convey the beauty and passion for research, and make people understand that research is a vital value for society. Therefore, authors worked on an effective audiovisual narrative, not following a strict disciplinary categorisation.

Since research is an open, interconnected, and accessible space, the chosen metaphor is the celestial sphere, an open space, constantly rotating and without boundaries. Users can explore the constellations, their stars, and planets using an ideal astrolabe, an ancient astronomical instrument through which the humans oriented themselves when observing the sky and navigating across the sea. By moving the astrolabe, the star map rotates, bringing each constellation into the centre of the screen.

The constellations represent the 10 keywords that are CNR’s major research challenges: sustainability, biodiversity, ecological transition, digital transition, clean energy, circular economy, life sciences, health, cultural heritage, peace, and scientific diplomacy. The meaning of each keyword is told in a simple introductory video. These challenges require the convergence and interaction of many disciplines, making research exciting and open to continuous encounters and interactions.

By entering a constellation, it is possible to approach the introductory video and then the stars representing the most prominent projects related to that keyword. The projects are presented through short videos (approximately 1.5 min each one), integrating resources from the respective research groups. Projects allow visitors to access research environments, places, objects, and people (Figure 12).

The software system was structured to be easily implemented in the future with new projects, giving visibility to ever new research.

As research is also an interconnected space, the networks between laboratories and research centres in Italy and around the world, working together on various projects, are dynamically gathered from a database and visualised on the planisphere projected in the centre of the table.

This is also an accessible space: both the installation and the surrounding area were designed to encourage the participation of all visitors. Sensory and tactile interfaces, sloped planes, and material layout favour physical access; translations into English, Braille, LIS and IS, and subtitles favour sense-perceptual and cognitive access. A large CNR research group has worked on these aspects, and the choice of shapes, colours, contrasts, lighting, and sounds also aligns with this principle.

The installation, implemented by the Institute of Heritage Science (ISPC) of CNR, is composed of the following [152] (Figure 13a,b):

• A wooden structure designed according to physical and ergonomic accessibility criteria, to facilitate wheelchair users;
• The “astrolabe”, a rotating console that allows visitors to navigate the celestial sphere. It consists of 11 tactile, non-interactive relief shapes with symbolic figures representing the constellations/keywords, and the history of CNR. These shapes are very simple and stylised and are also recognisable by blind or visually impaired people, with Braille captions for identification;
• Proximity or stop sensors that detect when the user is present and activate a constellation or keyword;
• A tactile slider enabling the user to navigate the constellation and activate projects;
• Three screens arranged in a semicircle, to provide a panoramic view, where the slowly rotating celestial map is visualised. A demo is available at https://www.youtube.com/watch?v=5VEYI4NIEuU (accessed on 26 March 2025).

Additionally, multimedia content is accessible through a similar web app that can be started anywhere, from mobile devices, a personal computer, or a kiosk, without the need for installation [153]. The online experience is based on the open source framework ATON [154,155] developed and maintained by CNR–ISPC.

Figure 13. The installation 100 Years of Science, 2023: (a) main rotating interactive interface with 3D printed elements corresponding to constellation keywords; the slider allows users to enter a constellation and explore its star projects; (b) the implementation phase developed at CNR ISPC. Images source: Enzo d’Annibale, CNR ISPC.

Figure 13. The installation 100 Years of Science, 2023: (a) main rotating interactive interface with 3D printed elements corresponding to constellation keywords; the slider allows users to enter a constellation and explore its star projects; (b) the implementation phase developed at CNR ISPC. Images source: Enzo d’Annibale, CNR ISPC.

3.2. The Codex 4D Holographic Showcase

In 2022–2023, the project “Codex4D: 4D journey into the manuscript” was developed by the Institute of Heritage Science at CNR and the University of Rome, Tor Vergata, thanks to funding from the Lazio Region (POR FESR Lazio 2014–2020). The project concerns the definition and testing of a methodological pipeline for the digital documentation and 3D exploration of both visible content and elements that are no longer visible or hidden in the structure of ancient codices, e.g., text parts buried in the binding and stratigraphies of the illuminations, as well as characterisations of the chemical, physical, and biological nature of materials. Therefore, 3D models of some manuscripts were created, incorporating the fourth dimension of depth [91,92].

The approach is innovative, aimed at creating a multidisciplinary experience with the manuscript inside virtual and mixed reality environments, taking into consideration tangible and intangible values: form and structure; content and meaning; materials, execution techniques, and state of conservation. Surface information is associated with the visible spectrum of light, RGB, captured using a standard photo camera. RGB images were also used to produce a photogrammetric model of the object, through structures from motion techniques. The invisible content related to the sub-superficial layers was acquired via a thermal camera, capturing images in the medium infrared range. The relevant information was contextualised and visualised as “annotations” on the virtual multidimensional model, or as additional textures applied to the model.

The interpretation of such an amount of data can help to reconstruct the complex story of the artifact, including the significance of written and pictorial content, craftsmanship, workshops’ “modus operandi”, conservation history, and cultural relations.

The project was developed on three case studies at the Angelica Library in Rome:

• De Balneis Puteolanis, ms 1474, the most precious copy of a poem by Pietro da Eboli, dated back to 1258/1266;
• Divine Comedy, ms 1102, by Dante Alighieri, dated back to 1325/1350;
• Libro d’Ore, ms 459, a prayer book for secular use, produced in France in the first half of the 15th century AD.

As the ancient manuscript is usually difficult to communicate to visitors (it is handwritten, in a disused language, impossible to leaf through, and challenging to perceive in poorly lit showcases), new approaches were experimented in communication strategies, to arouse curiosity and keep the audience’s attention.

Several outputs were developed in the Codex 4D project, that differ in (a) type of users, (b) context of use, and (c) type of experience, and, therefore, it adopted different communication styles.

A multimedia website [156] was designed for in-depth educational and scientific purposes, in which the Codex 4D project, the methodologies, technologies employed, and the results of the research are recounted.

An online virtual reality environment (Web App, accessible from the web site) was designed for the scientific visualisation and analytical exploration of the manuscript 4D model, intended for more expert users (Figure 14a); a demo can be seen at [157].

A holographic showcase, based on Pepper’s Ghost technique [59], was designed as a mixed reality installation for museum and
library audiences. Since the holographic showcase resembles a small theatre, equipped with lights, scenography, buttons, and sensors, research data are presented through an emotional and playful narrative.

The showcase, in fact, introduces dramaturgical storytelling: a narrator character, played by an actress filmed on a green screen, lives in the illuminations, and tells that world from the inside. She is as small as the illuminated figures, performs actions, and uses tools to enhance the perception and understanding of the codex (Figure 14b).

In the holographic showcase, the 4D model is again enriched with annotations dealing with the iconography, iconology, materials and techniques, and state of conservation. The user can explore these elements by moving his/her hand on the model, through simple gestures. Motion capture is managed by the Leap Motion sensor [158] and contributes to raising the user’s curiosity and engagement.

In this way, the Codex 4D project aims not only to increase the scientific knowledge of the manuscript but pushes the boundaries of scientific communication into new experimental territories to attract and intrigue new audiences, introducing gamification and dramaturgy. This achievement was made possible by the integration of many different competences into the team: paleographers, codicologists, art historians, conservation scientists, physicians, chemists, biologists, 3D modellers, experts in communication and user studies, computer scientists.

The Codex 4D holographic showcase was presented to the public for the first time at the exhibition “Languages of Heritage Science: from Micro to Macro”, at the Genoa Science Festival (Villa Principe, 20 October–1 November 2022). On that occasion, a survey was conducted to evaluate the experience and impact of the showcase on the public, represented mainly by high school and university students, families, experts, and individual and group visitors. Results are presented in [159]. Contents have been enriched for the new exhibition set up at the Angelica Library in Rome, from 10 November 2023 to 8 February 2024 (Figure 15). In this venue, the holographic showcase was shown alongside the original real manuscripts, which were displayed in a dedicated showcase. Nearby, an installation featuring the website and the web app was also available for visitors.

In October 2024, the Codex 4D project won the Heritage in Motion award [160], in the category Virtual and Augmented reality, with the following motivation: «This innovative project has been recognised for its groundbreaking work in making fragile historical manuscripts accessible to both scientific and general audiences, embodying the essence of the “heritage in motion” through innovation and preservation».

3.3. e-Archeo Human Interface

Commissioned to ALES SpA by the General Secretary of the Italian Ministry of Culture, and carried out in 2021–2022, e-Archeo is a major national multi-channel project aimed at enhancing eight archaeological parks, from North to South Italy, through digital and multimedia technologies. The sites were chosen to represent the diversity of cultures that shaped ancient Italy—Greek, Etruscan, Phoenician-Punic, Italic, later integrated under the common Roman hegemony—while maintaining some distinctive features: Sirmione and Desenzano on Garda Lake, Marzabotto, Cerveteri, Alba Fucens, Velia, Egnazia, Sibari, and Nora.

Several virtual reconstructions and multimedia applications were developed for different audiences, adopting a scientific, narrative, and emotional approach, with particular attention to technological sustainability and accessibility, and in line with open science principles.

The purpose of the partnership, composed of government institutions, CNR ISPC, 10 universities, 12 creative industries, Rai Cultura, and accessibility experts, is to strengthen the capacity to work together in the valorisation of Italy’s cultural heritage. CNR ISPC played a decisive role in the overall multimedia design of the interventions and in coordinating the executive production.

Multimedia applications are accessible to the public in different ways and at different stages of the cultural experience, both on site and remotely, through online applications and site-specific installations [161]. The project provides multiple levels of representation and narration of contents, and various tools and communication styles: to the scholar, it provides all the information to follow the reconstruction process of the various environments (levels of reliability, sources, and interpretative processes) (Figure 16); to the visitor, it provides narratives adopting a more attractive and sometimes dramatised style.

Accessibility strategies such as Tangible User Interfaces (TUIs) and Universal Design principles have been applied to allow everyone, including people with visual, hearing, and motor disabilities, to enjoy the content, especially in the case of the Cerveteri site (Figure 17).

In October 2024, the project won the Grand Prix Möbius in the category ‘Mutant publishing’, dedicated to “Immersive experiences of tourism and culture”, with the following motivation: “for its high strategic value, its focus on openness, quality and accessibility of data, and for the plurality of dimensions involved, including the tactile dimension” [162].

The outputs of the project are as follows and are described in detail in [161]: (1) e-Archeo 3D; (2) e-Archeo Voices; (3) e-Archeo Tactile; (4) e-Archeo HI—Human Interface; (5) e-Archeo Video; (6) e-Archeo Website. The e-Archeo Website [163] offers access to web apps and to the Zenodo e-Archeo collection [164]. As all datasets have been published on Zenodo under the Creative Commons licenses, they can be re-used by site museums, to create new multimedia experiences for their visitors, on the sole condition that they are released under the same license.

In this paragraph, the e-Archeo Human Interface experience will be shortly described, as it offers avenues for discussion not addressed by the previous case studies.

The e-Archeo HI^® application was developed for the archaeological contexts of Cerveteri and Sirmione, based on scientific data collected by universities and presented in a non-specialist narrative. The installation is on public display at the Archaeological Museum of Sirmione and the Cerite National Archaeological Museum in Cerveteri. A narrating character, played by an actor in ancient costume, at a 1.1 scale, interacts with users and narrates the archaeological area and its history. He also introduces some of the museum’s exhibits found in the site’s archaeological context.

In Sirmione, the narrating character is the poet Catullus, who links his name to the site, although he never lived in the villa we see today. In Cerveteri, the character is Vel, owner of an important ceramics workshop in the town. The user interacts with the character as if conversing with him, vocally or by selecting topics via tablet, thanks to a narrative node structure.

e-Archeo HI^® is placed near the showcases containing artifacts of great interest for the narrative (for instance, in Cerveteri Museum, the Crater, and the Kylix of Euphronius). The lighting system dynamically switches on and off on the artifacts, synchronising with the narrative led by the character. The digital installation thus functions as a ‘narrative hub’ or an ‘extended reality’ experience, because it does not have an autonomous life, independent from the museum collections, but, instead, it dialogues with the users and the physical objects, which react by lighting up when they are mentioned in the narrative (Figure 18). The overall experience lasts about 20 min.

3.4. The Innova Patrimonio Project

One of the challenges of our time is to transform peripheral territories and small ancient villages, which constitute open-air museums of the European identity and traditions, into no longer static and closed places, where it seems that many things are being lost, but, conversely, into spaces of cultural convergence, creative energies, and exchange. This can be encouraged by the interaction of the resident population with artists and researchers, and thus by the fusion of the local cultural tradition with new perspectives and actions.

The Innova Patrimonio project [165], funded by the Lazio Region (POR FESR Lazio 2014), has created an innovative model of narration of the peripheral territory and rural areas, considering the case study of the Sabina region, located north of Rome, crossed by an ancient Roman consular road, via Salaria. The major experimentation involved the village of Fara in Sabina (RI), founded in the VI century AD, located on the top of a hill, where fewer than 200 residents live.

In a process of narration and representation of an ancient village, we questioned ourselves which strategy could best stimulate motivation, interest, and affection in both inhabitants and visitors. Certainly not the traditional and impersonal descriptions of itineraries and monuments, but, on the contrary, an original narrative built through interaction and co-creation processes, based on a convergence of personal stories, languages, and perspectives. For a community, the identification of itself and its daily-life environment within a narrative is a reason for immediate interest and attention. However, at the same time, curiosity must be activated by a new light showing those familiar places as spaces of symbolic and unexpected representations. Thus literature, cinema, theater, video-projection mapping, 3D graphics, and mixed reality come together in this unusual representation of the village. This approach enables the community to looks at itself and at its “habitat” with new eyes, which react in unexpected ways. Thus, Fara Sabina becomes a town “made of stone and marble but also of wires, curtains, and angel wings”, echoing Calvino’s tales “Invisible Cities” [166].

Actors from a major theater residing in the village, playing characters developed over many years of performances, bring local stories to life, as they walk through alleys and squares, cross gardens and cellars, and enter homes. They narrate the stories, the life, and transformation of places over time, following an authorial script. The active participation of the population adds meaning to this playful re-invention. The story springs from the encounter between the eclectic and surreal invention of the artists and the concrete contribution of the inhabitants, with their experiences, private spaces, opinions, habits, and passions (Figure 19 and Figure 20). The traditional description of itineraries and monuments is replaced by a dramaturgical representation of stories, ways of life, and imaginary dimensions that constitute the unique character of the village. To enhance this magical experience, virtual appearances and digital projections on an urban scale have been created.

Objects from the Sabine and Roman civilisations, preserved in the Civic Archaeological Museum of the village, have been digitised in 3D and projected onto the exterior facades of the buildings. They parade, carried by the hands of present-day inhabitants, who were also filmed and projected onto the facades, symbolising their role as the successors of ancient populations and present custodians of this cultural heritage (Figure 21).

The most valuable object is the two-wheeled chariot of the Sabine prince of Eretum, decorated with historiated bronze sheets, coming from a monumental tomb of the 6th century B.C. discovered in the nearby necropolis of Colle del Forno. Its virtual appearance in the sky of Fara and then its entry into the museum, in the presence of the entire population, mark the symbolic epilogue of the story (Figure 22). The chariot becomes a symbol of friendship and luck. Through these visual metaphors, access to cultural heritage is thus expanded outside the museum, to promote a wider integration within society.

The various stories have been compiled into short films, which can be accessed via an interactive map. The research has resulted in (1) an open platform that allows users not only to explore itineraries but also to modify and personalise their cultural visit; (2) a web app; (3) a native app; (4) a website [167]; (5) site-specific installation for the museum, and (6) a 1h 30′ film titled “In the sky of Fara, stones and angel wings”. Additionally, a platform based on the VVVV software [168] was experimented, to manage the direction and transmission of online multimedia events in real time, within a VR environment: in the 3D model of the square of Fara in Sabina, a video-projection mapping in real time was performed on the architecture. At the same moment, on a virtual stage located in the same virtual square, live interventions of people were played. The public could connect to this event and enjoy it, as happened on the occasion of the project review meeting. The project was presented to the Lazio Region and to the Municipality of Fara in Sabina. What has been achieved is a scalable and exportable model for the valorisation of other ancient villages, to stimulate creativity and, hopefully, to improve tourist attraction [169,170].

The project, carried out during COVID-19, gave life to new educational processes, addressed not only to inhabitants, students, and tourists, but also to creative industries and scientific institutions working in the enhancement of cultural heritage. In fact, new approaches combining science, art, knowledge, and technology were invented.

This repeated encounter has resulted in an emotional and creative ferment that has gradually spread to involve the entire local community. At the beginning of the project, the inhabitants were hesitant and a little wary, reluctant to share historical materials from their personal archives. However, as the work evolved, they showed a growing interest and desire to participate, share ideas, and open their houses. The script also includes their contributions, and their participation is evident in the film, where many appear as protagonists or extras (Figure 20, Figure 21 and Figure 22).

One of the project’s most significant outcomes was the overcoming of initial skepticism, the desire to build together, and the strengthening of social cohesion. Innova Patrimonio, realised during the pandemic, was an opportunity to share imagination, experience beauty individually and collectively, and promote it as a social value.

The 1h 30′ film titled “In the sky of Fara, stones and angel wings”, created in the context of the project, won the first award, as Best Film, at the 5th edition of the Tuscany Web Festival. This is an international festival of independent cinema which was attended by 316 works registered from 39 countries around the world, and 72 Finalists (https://www.facebook.com/yesitsyourmoment/) (accessed on 26 March 2025).

3.5. The PERCEIVE Project

The PERCEIVE (Perceptive Enhanced Realities of Colored collEctions through artificial Intelligence and Virtual Experiences) European project [18] addresses the need to better perceive, understand, preserve, and communicate coloured artworks. Coloured works are extremely fragile and vulnerable, as colours tends to degrade and change over time, until they are lost. For example, the colours of textiles fade in just a few decades; ancient statues now retain very few traces of their original brilliant hues; similarly, the colours of analog photography can be subject to substantial fading due to aging.

At the same time, the PERCEIVE project aims to conceive and develop strategies for enhancing the visitor experience with a variety of colourful digital collections, as well as natively digital artworks, and to transmit and communicate them in the most effective and correct way to future generations. Two core concepts of the project are the “sense of care” and “sense of authenticity”.

The project is still under development and expected results include the following:

• The reconstruction of the original perception of the coloured artworks through simulation tools and the production of new digital images;
• The prediction of the future evolution of colour changes;
• The development of tools, services, and interactive prototypes for on-site and online access;
• The elaboration of methodological guidelines for the exhibition of coloured collections;
• The creation of a design toolkit addressed to designers, educators, and curators, aiming at creating new multimedia, VR, and AR applications.

The project aims to use artificial intelligence and machine learning to automate several manual processes for reconstructing the original colour, thereby facilitating and speeding up the work of scientists and creative industries. A primary requirement is feature extraction, which is an essential task in the process of image analysis. Since computing power is increasing and acquisition techniques are improving, the feature extraction process could be carried out by AI-based techniques more effectively. Initial results show that learnt AI-based descriptors, on average, outperform traditional hand-created descriptors. However, these techniques revealed a disconnection between the mathematical representation of extracted features and any meaningful parameters perceived or understood by humans [171,172]. The risks associated with the use of AI that the PERCEIVE project aims to mitigate are as follows: (1) lack of shared methods and tools; (a) absence of AI adoption for 3D or more complex collections; (b) limitations related to the training dataset and lack of a shared data source; (c) use of local infrastructure; and (d) little or no evaluation of results. The project, started in 2022, lasted 36 months and represents a good occasion to investigate the potential of artificial intelligence in this frontier domain.

4. Discussion

The museum should be the place where the visitor lives perceptual experiences that are unique and not equally repeatable outside the museum. In other words, the virtual visit inside the museum should not offer the same content and the same communication registers that a smartphone can offer in any other context of use.

The pervasive presence of smartphones in every moment and place of our lives risks compromising the opportunity that museums can seize from multimedia: bringing spaces to life, animating them, dramatising them, recreating a sensory dimension behind the exhibits. Delegating the entire digital offer to the smartphone, asking the visitors to constantly direct their gaze and interaction on this small display, risks weakening the exceptional nature of this experience. This is undoubtedly an easy and functional solution, since it evades the problems of the daily management of technologies; however, it is not an exciting solution to lead the visitor into a lively experience.

Multimedia content can be used to reconfigure the map of the museum in terms of the attractiveness of the objects, rooms, and routes since they influence the time of permanence, and the level of interaction and social exchange.

The design of the museum experience must consider both digital content and the exhibition layout: the real and digital should dialogue, to focus the visitor’s attention and emotion on the place in which he/she is currently immersed (Figure 23).

The great effectiveness of extended and mixed reality consists in the fact that the real and virtual no longer exist in separate spaces and times but communicate not only conceptually but also ‘physically’, becoming synergetic components of our perceived and experienced space. Thanks to the coherent superimposition of the virtual, physical reality becomes multi-layered, stratified, and is in continuous potential transformation, a ‘canvas’ on which forms and meanings can be reconstructed, representing interpretative contents (Figure 24).

Through mixed reality, it is possible to give new light to those aspects or meanings of the artwork that have been erased, or are no longer perceptible, for conservative or ideological reasons (damnatio memoriae, or modernisation) or because they were modified by the artist. These contents can be documented by diagnostic analyses revealing the material aspects hidden under the surface, and they can be visualised through a digital projection onto it. A new convergence of sciences, arts, languages, and technologies is possible today, and the fusion between the real and the virtual seems to prefigure the perceptive and cognitive dimension of our daily life in the future.

Another fundamental aspect of exhibition design is the correct relationship between virtual content and real collections: the virtual should not overpower the real, which remains the primary focus of the museum on-site experience. It is also necessary to avoid over-reliance on digital tools, as too many multimedia close to each other can conflict in terms of audio and tolerance. Efficient visitor flow, visibility, clarity of display, evidence of the concept to be expressed, harmony, relaxing conditions of use, and aesthetic and cognitive stimuli need to be balanced correctly to generate a feeling of well-being. Lighting, acoustics, seating, and order in the flow of visitors are essential to create a comfortable environment. Certain ‘iconic’ elements or objects that convey the main concept transmitted by the digital installation can be replicated in their material aspect and placed nearby. Similarly, original artifacts included in the digital storytelling can be highlighted, as in the case of the e-Archeo Human Interface project, to enhance the experiential context.

Thus, physical and virtual enjoyment become part of the same knowledge process, combined in conveying the material and immaterial value of the cultural asset: they reinforce and complement each other, according to a holistic and multifunctional approach. For this reason, it is mandatory to mention multimedia in the catalogue or in the museum guided tour, to inform users about their presence and role in the exhibition. For instance, in the case of the CEMEC exhibition in Bonn, where the holographic showcase was presented and the user experience evaluated, visitors appreciated the new way to tell and dramatise stories, as well as the overlapping of real and digital content to contextualise objects. However, they complained that the role of such an installation in the exhibition was not clear, as it was not mentioned in the written nor audio guides, or in panels (Figure 6).

One of the key objectives of research should be the experimentation of multiple ways to represent stories, and shape languages and levels of interaction to create engaging experiences addressed to a diverse audience. Different narrative forms and approaches (whether linear or interactive, predefined or co-created, transmitted by 3D characters or by human narrators) might prove to be more effective in certain contexts and for specific users. For this purpose, various professionals should cooperate: cultural heritage experts, scriptwriters, directors, set designers, graphic designers, musicians, user experience designers, psychologists, cognitive scientists, neuroscientists, doctors. Digital interventions in museums should be prepared by preliminary surveys on museum visitors, to analyse their provenance, background, average age, preference to conduct group or individual guided tours, and initiatives for children or users with special needs. Prototypes should be tested and evaluated at different stages of development, to verify and improve their usability, content intelligibility, efficacy, utility, comprehension, and remembering. Some scholars argue that museums learn more from the formative evaluation than from the summative one [173]. Both of them were carried out during the Etruscanning and the Codex 4D projects, as well as along the experimentation the CEMEC holographic showcases in museums, already mentioned in this article. Results guided our team towards an improved user experience design and a better use of digital media [42,54,159]. In fact, formative evaluations were instrumental in harmonising the expectations of designers and ICT developers with the real needs of museums and visitors. For instance, they helped define the ideal interaction complexity level, the optimal length of storytelling, the comprehensibility of content and interfaces, and the integration of digital installations along the paths.

Finally, the technologies used within museums should be user-friendly, robust, and sustainable over time, as their maintenance remains one of the most critical aspects today. To ensure maintenance and sustainability, it is crucial that digital application developers provide museum staff with live training, as well as a detailed set-up guide, including visual support such as diagrams and screenshots. This guide should outline the necessary maintenance, such as software updates, hardware recalibration, and periodical replacement of perishable materials like sensors, lamps, and air filters in video projectors. It should also highlight the need to regularly clean electronic equipment. Museums often cannot handle complex repairs themselves, but they should be able to communicate the required information to qualified technicians.

An efficient internet connection, whether always active on installations or activated when needed, can facilitate remote interventions and updates by experienced teams. However, it is essential that funding from public and private tenders is allocated not only for the creation of new products (which undoubtedly gives greater media visibility to financiers), but also for maintaining and improving existing ones, as well as reusing prototypes for new content. The short lifecycle of digital applications in museums and cultural venues, driven by consumerist logic, is one of the most critical issues. This challenge should be addressed through new, robust management strategies, staff recruitment efforts, and funding policies that promote long-term sustainability. The adoption of criteria and practices for digital data management is of paramount importance, as it ensures the long-term maintenance and development of digital assets. Compliance with the FAIR principles is essential, particularly regarding interoperability and metadata of digital contents. The publication of data produced through public funding, in accordance with the Open Science principles, is a prerequisite for the life cycle of public information. This approach aligns with the general guidelines of the Horizon2020 Programme (Horizon 2020 Open Research Data Pilot and Data Management Plan) and Horizon 2021–2027 [174,175].

The growing participation and sharing of cultural information, tied to ‘citizen science’, co-creation, and crowdsourcing initiatives, is a sign of the unstoppable digital transition underway. It is therefore necessary to adopt rules, services, tools, and good practices to ensure quality control and the preservation of this great amount of digital heritage which can generate new cultural, social, and economic values.

Collaboration with the school sector is also strategic. Curators and educators should be encouraged to involve students in creating new content that can enrich the museum experience, bringing new perspectives, an exchange of knowledge, and innovative storytelling.

5. Conclusions

The creation of a multidimensional and multisensory experience in a museum, understood as extended reality in which physical and digital contents are harmoniously integrated, implies a complex design.

In this paper, methodological aspects were discussed, and case studies were presented, which can help to assess the following:

• Who the current or potential museum visitors are, and what their expectations are;
• How the museum can help them meet their needs, offering a profound and multisensory experience of comprehension and well-being;
• What strategies can be adopted to attract their curiosity towards the contents, but also to retain attention and focus it progressively, so as to foster emotional involvement, critical processing, and remembering;
• How to integrate real and virtual content in the museum space to create extended realities;
• What the principles for good storytelling and useful interaction are;
• How to increase embodiment in physical and digital spaces of experience;
• Which good practices in cultural heritage digitalisation should be followed, how to codify and express the levels of trustworthiness and reliability of virtual content;
• How to represent and transmit the invisible aspects of the artworks, hidden beneath the surface;
• How to increase the sense of authenticity of the cultural experience;
• How to create inclusive and accessible museums;
• How artificial intelligence works and what perspectives it opens up in the field of study, creativity, and cultural transmission, especially in museums;
• How to encourage social exchange while at the same time resonating with the deep emotions of the individual entering in contact with the arts;
• The importance of the link between museum, territory, and communities;
• How to ensure the life cycle of digital resources.

Different solutions and methods, among the wide range of possibilities presented in this paper, can be chosen based on educational and narrative goals curators want to achieve. Different experiences can be provided to different target groups, in relation to the kinds of artifacts, the location along the visiting path, the storytelling structure, the interaction level, the estimated length of the digital experience, the environmental conditions, whether the engagement is as a single user or as a group, and, last but not least, the cultural and technological literacy level of visitors. Digital communication in museums can enhance the experience of many visitors, but it does not necessarily attract those ones from traditionally underrepresented groups, such as people from low social classes, with limited education, or ethnic minorities. The mere presence of engaging digital elements is not enough to overcome the barriers these groups face when visiting museums, especially if the whole conception of the museum is not welcoming and inclusive. If the goal is to improve accessibility, a broader transformation of the entire museum is essential: interaction and narrative metaphors, languages, and media should be tailored to the different target groups. A meaningful dialogue with users, schools, and families, and improvements in democratisation and participation strategies are essential if museums want to remain relevant in 21st century evolving society [176,177]. The museum becomes not only a place for learning, but also a place to receive new cultural and spiritual stimuli, to relax and recover from the stress of daily life, a “restorative environment” [178]. Therefore, the design of the entire physical and digital environment in which the experience takes place is of paramount importance in persuading the visitor of the credibility, authenticity, and value of what is being offered. Many factors can condition visitors in decision-making processes, even unconsciously. Space design and lighting elements, and information and interaction modes can guide the users’ behaviour and concentration, without restricting the individual freedom of choice. Digital nudging strategies are already widely used in healthcare, e-commerce, or online privacy and security [179,180]. They could be usefully employed in museums to make the environment more welcoming, to pace the experience, and to attract users to specific areas of interest, reflection, and well-being.

When, during the 19th century, photography made it possible to capture artworks, there was a fear that the public would no longer have a reason to visit museums. A similar concern and debate arose in the late 1990s with the spread of digitisation practices and the development of virtual museums. These worries have largely been overcome by evidence showing that a greater circulation of knowledge, allowed by digital technologies, actually stimulates the public’s curiosity to visit museums and see the original artworks, thereby promoting accessibility to knowledge [181].

Digital technology, and specifically virtual reality, cannot replace the real experience of visual and bodily perception of the original objects, which are embedded in the museum context and atmosphere. The museum experience turns out to be not linear but cyclical, with emotions playing a crucial role at every stage [182].

Mixed reality and phygital experiences are the most powerful and the most difficult to implement in a museum because they potentially bring communication to its fullest expression, not only resorting to visual media but involving all senses, like sound, touch, smell. However, a respectful, credible, and efficacious balance between real collections and virtual contents is recommended. Physical and virtual contents reinforce and complement each other, thus becoming part of the same knowledge process. They convey the material and immaterial value of the cultural asset and can increase the authenticity of the experience. The expressive quality of the multimedia content is recommended, both from an aesthetic point of view and in terms of historical documentation and cognitive value.

The primary task of digital and multimedia along exhibition routes is to multiply the levels of perception and symbolic reconstruction of the meaning of cultural objects, to increase their communicative potential and make them more comprehensible and accessible. Through digital media, it is possible to integrate their shape and volume (virtual restoration), and to simulate their context of provenience (virtual reconstruction) and their semantic and thematic connections (storytelling). In fact, the virtual allows us to give form to abstraction, to represent something that does not exist, that is no longer visible or cannot be experienced. Although a restoration, as well as a virtual reconstruction, can present elements of uncertainty with respect to the real original state of the object, they are nevertheless fundamental vehicles of understanding: without them, the abstract mind could be induced into greater error, that is, into interpretations far from any historical plausibility.

Another task that multimedia technologies should perform, both inside and outside the museum, is the multichannel dissemination of cultural information, i.e., the ability to convey content through diverse languages and technologies tailored to users with a variety of cultural backgrounds, needs, technological abilities, origins, and ages, including persons with disabilities, for whom the difficulties of accessibility to the original cultural asset are amplified. Audiovisual representation, interaction, and gamification within cultural venues can also attract young audiences, involving them in collaborative experiences and in the creation of shared storytelling that makes them feel like participants and protagonists [183,184].

Such a perspective makes it necessary to complement traditional curatorial teams—consisting of experienced archaeologists, art historians, architects, and conservators—with professionals who have so far remained outside this context, such as writers and scriptwriters, directors, set designers, graphic designers, lighting experts, musicians, user experience designers, digitalisation experts, sociologists, and computer scientists. A fundamental aspect for museums and cultural sites is the economic and management sustainability of the technological infrastructure over time. In fact, practical barriers can include costs, maintenance, and staff training. Museums emphasise the need to work on a daily basis with technologies that are easy to manage, simple, and do not require extensive training for museum staff and users. Sometimes a disconnection arises between the demands of research institutions and laboratories working in technological innovation, whose main interest is devoted to experimenting with the most sophisticated and cutting-edge technologies, and the everyday challenges of managing digital technologies within museums and archaeological parks. Nevertheless, when developing permanent digital infrastructures for public use, usability, usefulness, and content effectiveness are of the utmost importance. Staff training is undoubtedly necessary to guide and support visitors towards the digital experience: technologies cannot be left to function on their own; human mediation remains crucial.

As discussed, machine learning and generative AI are opening new scenarios in management practices, image processing, pattern recognition, text analysis, natural language processing, decision-making, and in the dissemination of cultural heritage [124,129,185]. These scenarios need to be understood and well driven through the integration of new professional skills. Artificial intelligence can increase the accessibility of museum content through multilingualism, virtual assistants and sign language virtual interpreters, and by providing information about the artworks. Before AI, it was impossible to answer any visitor’s curiosity or question in real time. The answers were necessarily static and predefined, and this could lead to minor public involvement. With the development of generative AI, a major challenge is emerging: how to control the data used by AI to prevent misinformation to visitors. It is crucial to inform users about what they can realistically expect from AI. More than in any other sector, the cultural institution must carefully consider the ethical implications of AI, to ensure that these algorithms are used in a responsible and inclusive way.

Transparency is the first key point: users must be informed about where and how AI has been used, and where information comes from. Intellectual property is another ethical issue to be considered, together with the fair representation of content across different cultural contexts, avoiding offensive interpretations or the marginalisation of people. Finally, AI should be used to complement human expertise, enhancing and not replacing the role of the museum staff in visitor engagement.

The combination of culture and technologies can produce new economies, but culture must not become a mere product of the market. Technology can help humans to automate, connect, control processes remotely, and shape creativity, but with the aim of affirming a new humanism that places the human being at the centre without burdening him/her with anxieties.

Future Perspectives

In future research, it will be useful to investigate further which factors most influence the sense of well-being, involvement, motivation, and embodiment in museums.

For the evolution of museums towards multisensory communication, the involvement of all senses is desirable: touch, smell, and taste, in association with the audiovisual experience, in real as well as in VR and MR environments. For this kind of experience, digital technology and robotics are not yet ready to offer usable, agile, and effective solutions. However, physical objects with sensors, which can be experienced tactilely, or dispensers of scented or flavored essences can easily be integrated into the interaction space.

An often-missing experience for museum visitors, aside from specific activities addressed to schools, is the practice of making; using one’s hands to go through some of the processes that led to the creation of the artworks.

This multiplicity of stimuli helps consolidate understanding and enhance the sense of authenticity in the experience.

It will also be extremely useful to measure which factors most affect the perception of the credibility and authenticity of the narrative, also considering the spread of generative AI. Undoubtedly, a convergence between heritage sciences, digital humanities, and neurosciences could facilitate the measurement of neurophysiological parameters indicative of well-being and thus the understanding of their activators, both at an individual and collective level. These results could translate into the adoption of increasingly mature strategies in the cultural offer and in heritage valorisation, suitable for all audiences.

Technologies and paradigms change, but the ultimate goal of this research does not change: to nurture understanding, affection, and thus a sense of caring for our history, while also embracing the creative reworking of our cultural heritage.