Exploring User Engagement in Museum Scenario with EEG—A Case Study in MAV Craftsmanship Museum in Valle d’Aosta Region, Italy

Abstract

In the last decade, museums and exhibitions have benefited from the advances in Virtual Reality technologies to create complementary virtual elements to the traditional visit. The aim is to make the collections more engaging, interactive, comprehensible and accessible. Also, the studies regarding users’ and visitors’ engagement suggest that the real affective state cannot be fully assessed with self-assessment techniques and that other physiological techniques, such as EEG, should be adopted to gain a more unbiased and mature understanding of their feelings. With the aim of contributing to bridging this knowledge gap, this work proposes to adopt literature EEG-based indicators (valence, arousal, engagement) to analyze the affective state of 95 visitors interacting physically or virtually (in a VR environment) with five handicraft objects belonging to the permanent collection of the Museo dell’Artigianato Valdostano di Tradizione, which is a traditional craftsmanship museum in the Valle d’Aosta region. Extreme Gradient Boosting (XGBoost) was adopted to classify the obtained engagement measures, which were labeled according to questionnaire replies. EEG analysis played a fundamental role in understanding the cognitive and emotional processes underlying immersive experiences, highlighting the potential of VR technologies in enhancing participants’ cognitive engagement. The results indicate that EEG-based indicators have common trends with self-assessment, suggesting that their use as ‘the ground truth of emotion’ is a viable option.

1. Introduction

In museums and heritage sites, technology is gaining momentum. Some innovations, such as projection mapping, binaural audio, digital twins, holographic displays, and visitor flow tech, have moved from proof of concept to standard. Most state-of-the-art museum technologies impact and enhance the visitor experience, while location-based intelligence helps curators to understand how people experience their exhibits and organize their collections in a way that better connects with visitors’ needs.

Among emerging technologies, Virtual Reality (VR) offers new opportunities for creating immersive and interactive experiences in the cultural heritage scenario. Generally speaking, the applications in which VR has been involved are numerous: from education [1] to pure entertainment, but also in marketing, as well as human–computer and human–robot interaction [2], psychology and medicine [3,4]. In the last few years, there has been a considerable increase in the use of VR in museum environments, and some of the most important museums in the world have already embraced technological innovations and adapted to the challenges of the digital era to offer different and immersive experiences to visitors. Indeed, museums aim to bring collections to life, and VR is an excellent tool to achieve this aim. Take, for example, the Victoria and Albert Museum in London [5], which, in the summer of 2021, opened ‘Curious Alice’, an exhibition exploring the origins, adaptations, and reinventions of Lewis Carroll’s classic, or the Louvre Museum in Paris, which, in October 2019, launched ‘Mona Lisa: Beyond the Glass’, a VR experience that explores the Renaissance painting as part of its Leonardo da Vinci blockbuster exhibition.

Especially after the COVID-19 pandemic, the advantages of VR became evident to enhance museum exhibits by creating virtual tours, making exhibits interactive, putting objects in context, and showing their real scale. However, while this immersive technology has some unique traits, such as the ability to create a first-person perspective and the sense of presence, immersion, agency and embodiment [2], it also has some drawbacks. If, on the one hand, VR has been employed in various industries (e.g., aerospace, automotive, and biomechanics) to reduce cost and time by replacing physical mock-ups with virtual ones [6,7,8], on the other hand, the adoption of more advance VR tools, such as Head-Mounted Displays (HMDs) and Cave Automatic Virtual Environment (CAVE), could be very expensive. Current commercial VR HMDs span from low-cost (e.g., smaller brands, usually running off smartphones), average-cost (e.g., Oculus Quest and HP Reverb G2), to high-priced (e.g., HTC Vive Pro 2 and Valve Index), with different effects to the user’s immersion degree [9]. Although, even on a small scale, VR is an innovative and versatile tool that museums can adopt to increase users’ engagement and improve visitors’ experience, making information more accessible and enjoyable, providing new possibilities to explore content through the process of “learning by doing” [10].

VR has also been acknowledged for its potential to raise and study emotions, which are essential for making a difference in memorable experiences [11]. In general, VR has been adopted with this purpose in various contexts: for therapeutic uses to induce relaxation and feelings of emotional well-being [12], to treat phobias and post-traumatic stress [13], or to stimulate mood changes [14]. Indeed, thanks to its completeness and flexibility, VR is the perfect environment where semantic and sensory elements, dynamism, and interaction can contribute to arouse a specific emotion [2,15]. The immersion, quantified by the sense of presence, i.e., the subjective experience of being in one place while physically located in another, is linked to the emotional response [16], especially regarding arousal [17].

In this sense, one of the most promising technologies for studying users’ emotions and perceptions from a neural perspective during real and immersive VR experiences is electroencephalography (EEG) [18]. EEG records the electrical signals generated by the brain, providing a direct window into the neural processes associated with perceptions, emotions, and attention. Recent technological advances, including portable and wireless devices, have enabled its integration with diverse settings such as museums and galleries, allowing researchers to explore visitors’ affective states during artwork exhibitions [19,20]. EEG measurements have an excellent time resolution, typically less than a second, and wearable devices allow for continuous recording while people participate in real or virtual experiences. Still, EEG is susceptible to motion artifacts and, thus, requires controlled experimental settings; in this sense, besides EEG applications in real-world environments, VR allows researchers to explore different scenarios and conditions and obtain reliable EEG results while ensuring ecological validity [11]. In addition, studies have shown that EEG-based metrics can provide objective measurements regarding emotion interpretation, such as engagement, valence, and arousal, which are useful for understanding human behavior [21,22].

The effectiveness and the applicability of the EEG analysis can benefit from the adoption of machine learning (ML) techniques for processing and classification purposes. Particularly, the capability of ML to find out patterns in data makes this approach suitable for the study of the brain’s activity, both in terms of emotions and affective indicators. Support Vector Machines (SVM) [23,24,25,26,27], K-Nearest Neighbor [28], Naïve Bayes [29], Linear Discriminant Analysis (LDA) [30], Quadratic Discriminant Analysis (QDA) [28], and Decision Tree (DT) [31] have been commonly used. Moreover, solutions based on ensemble learning such as Random Forest (RF) [31], Bagged Tree (BT) [32], AdaBoost [33] and Extreme Gradient Boosting [34] have been proposed in the same context for the classification of EEG processed data. SVM is one of the most common data-driven approaches, and it is considered a valuable alternative to statistical analysis in affective studies thanks to its capability of handling multi-dimensional data on the basis of multi-variate patterns [35,36]. One drawback is that SVM is prone to overfitting, leading to unsatisfactory results when dealing with small, noisy, and complex datasets [37,38]. From this viewpoint, bagging and boosting ensemble learning offers valid alternatives [32,33,34]. RF has been used to construct predictive models of mental states such as meditation and concentration, with classification accuracies around 75% [39], outperforming 90% accuracy when deep learning was used for feature extraction [40]. AdaBoost was successfully applied to classify human emotions in the 2D valence–arousal space, reaching on the DEAP dataset [41] 97% accuracy [42] and outperforming 88% accuracy for binary classification in the dominance dimension on the same dataset [33]. On the same dataset, BT showed high performances with over 97% accuracy in 2D valence–arousal space [32]. From the analysis of the literature, it emerged that XGBoost is less commonly used in emotion recognition, even if noteworthy results were obtained for EEG analysis of the DEAP dataset [34], achieving almost 95% accuracy [43]. Concerning complex data such as EEG, Neural Networks (NNs) represent a valid opportunity both with shallow and deep algorithms [44]. The recent literature shows that more than an alternative to traditional ML approaches, NNs are obtaining remarkable results when a combination of the two approaches is applied, particularly NNs for feature extraction and ML for the classification stage [45,46]. On the other hand, NNs require a substantial amount of data that when combined with the complex nature of EEG signals could lead to high computational costs.

In this work, we explore the adoption of EEG-based metrics of arousal, valence and engagement to quantify the visitor’s emotional state while s/he interacts with five physical (experience 1) and virtual (experience 2) handicraft objects belonging to the collection of the Museo dell’Artigianato Valdostano di Tradizione (MAV). The MAV, located in Fénis (Italy), is a museum dedicated to the traditional craftsmanship of Valle d’Aosta region. While the first experimental session (counting 33 participants) was held physically at the museum, the second (with 62 participants) was a virtual tour in a VR environment which was designed ad hoc as a complement to the traditional museum visit. In both experiments, EEG was used as emotional monitoring technology; the EEG-based engagement indicator was computed relying on previous literature, labeled according to the administered questionnaire, and classified adopting an XGBoost classifier.

The research conducted in this work was possible thanks to MEDIA (Museo Emozionale DIgitale multimediale Avanzato) project, which was funded by the Valle d’Aosta region.

The paper is structured as follows: Section 2 explains the methodology and experimental setup, providing a description of the experiment and the selected handicraft objects (Section 2.1), of the VR environment designed for the VR experience (Section 2.2), of the participants taking part (Section 2.3), of the questionnaire for collecting emotional feedback (Section 2.4), of the EEG affective monitoring (Section 2.5), and of the machine learning XGBoost classifier (Section 2.6). Section 3 presents the results divided into questionnaire replies (Section 3.1), EEG emotional assessment (Section 3.2), and XGBoost classification (Section 3.4). The paper ends with discussion (Section 4) and conclusions (Section 5), where future works are proposed.

2. Materials and Methods

The MAV, located in Fénis (Italy), is a museum dedicated to the traditional craftsmanship of the Valle d’Aosta region, carrying a wealth of symbols, knowledge, identity values, and extraordinary creative processes of which the objects are the custodians. About 800 objects are exhibited inside the MAV, including everyday artifacts and sculptures, which witness the evolution of the local artisan tradition. The collection of this museum, particularly referring to five specific representative handcrafted objects, has been chosen as a base for carrying out the experiment described in this section.

2.1. Description of the Experiment

The experiment was divided into two experiences united by content and structure but differentiated by the technology used. With the help of the museum curators, five objects of the museum’s collection, shown in Figure 1, were carefully chosen based on their state of preservation, suitability for successful scanning, and the presence of engaging anecdotes that facilitate contextualization. In particular, the following apply.

• The cockerel: born as a game for children, it was made with a forked tree branch that outlined the body on one side and the tail on the other. After applying the color to this artifact, its evolution received widespread public approval, and the cockerel became the symbol of craftsmanship in the Valle d’Aosta region;
• The crib: handed down from generation to generation, the cradle was given as a gift by the godparents to the unborn child and used on the day of baptism. It was decorated with geometric carvings such as rosettes and religious elements, offering indications about its provenance;
• The butter press: the various types of butter presses tell how a simple object can differentiate itself between the side valleys and within the municipalities of a small region, undergoing modifications in both its symbolic representation (whether natural or heraldic) and construction techniques (including elements affixed with nails rather than carved);
• The goat collar: it consists of a sheet of wood folded by immersion in boiling water to create the appropriate curvature to be worn by goats. Once folded, the foil was tied at the ends and left to dry. Following this, the skilled artisans move on to the intaglio decoration, incorporating colored elements;
• The ‘tatà’: it is a carved wooden horse with wheels. Its name derives from a childish onomatopoeic expression indicating the noise the wheels make when dragged along the ground.

The experimental setup is shown in Figure 1 and is composed of two experiences, which are explained here below. While taking part in the experiment, participants’ brain activity was measured using a 14-channel EEG headset: the Emotiv EPOC X.

Real experience (EXP1). A designated space with a table and seat was set up at the museum to let visitors visualize the handicraft works and interact with them physically (Figure 2a).

VR experience (EXP2). Users are invited to navigate an ad hoc developed virtual environment in which objects have been inserted and contextualized with the possibility of interacting with them. Participants used an Oculus Quest, a VR headset developed by Oculus that can run games and software wirelessly under an Android-based operating system. The adoption of this HMD device mounted on the head does not preclude brain activity monitoring through the EEG headset [47] (Figure 2b).

For both experiences, the objects were randomly presented to the participants at two different levels of interaction, which are called ‘phases’ and explained here below.

Visualization and contextualization phase (VC). The selected objects, one at a time, are shown for 30 s and then contextualized, i.e., explained in terms of history and significance, to the participant for another 30 s. The description was provided using a combination of written text and recorded explanations for both experiences (Figure 3a).

Interaction phase (INT). After the first phase, participants could interact with the physical objects by touching them (real experience) or with the virtual twins (VR experience) via VR controllers, which ensure natural interaction in a virtual environment (Figure 3b).

2.2. VR Environment

The virtual environment developed for the experimentation is meant to represent a traditional village in the Valle d’Aosta region, where the selected objects are inserted and contextualized in four distinct areas. The original handicraft objects have been scanned through a structured-light 3D scanner, a 3D scanning device for measuring the three-dimensional shape using projected light patterns and a camera system. The patterns projected by the light source from the scanner head, when light projects onto the object’s surface, become distorted enabling 3D data acquisition, while the camera enables texture acquisition. The VR environment is available in three languages: Italian, English and French.

Unity (Unity 2021.3.15), i.e., the cross-platform game engine developed by Unity Technologies, has been used to design and develop the VR environment. Starting with adding a Terrain GameObject to the Scene, textures, trees, and details like grass, flowers, and rocks have been created to obtain the desired landscape features (Figure 4). Then, the four macro-areas intended to host the objects have been defined. Following the indications provided by the museum curators, a carpenter’s workshop (Figure 5a), a log house (Figure 5b), a stable with cows (Figure 5c), and a farmhouse with goats (Figure 5d) have been designed to contextualize the cockerel, the crib and the tatà, the butter press, and the goat collar, respectively. In addition, some sounds downloaded from Mixkit, a free gallery of stock video clips with no watermark, music tracks, sound effects and video templates, have been inserted into the virtual environment that recalled the object or its manufacture. In particular, a hand saw tool sound has been selected for the carpenter’s workshop, a lullaby has been chosen for the crib, while for the tatà, the sound of wooden wheels on the floor has been played. Finally, for the two areas with animals, cows and goats, it was decided to reproduce their sounds, together with the noise of pouring milk.

Unity supports the C# programming language natively, and some scripts have been created and attached to GameObjects to control their behavior, such as triggering game events, modifying component properties over time and responding to user input in different ways. Indeed, some scripts have been used mainly to define the 30 s phases, i.e., visualization and contextualization, and interaction, for each object, to activate/deactivate canvas, sounds, lights, and cursor lock, and to print and save timestamps in order to synchronize the experience with the acquired EEG data.

Android, the target platform for Oculus Quest, has been set, all necessary additional settings have been configured, and the Oculus Touch Controllers have been arranged and connected to allow interaction during the INT phase. After all these steps, a build has been generated and run on the headset, which is connected to the computer over a USB-C cable.

2.3. Participants

The study involved a total of 95 participants between the ages of 19 and 72, mostly Italian, who voluntarily took part in the experiment. In particular:

• EXP1 involved 33 participants (21 women and 12 men);
• EXP2 involved 62 participants (31 women and 31 men).

In order to maintain each interaction with the objects novel and avoid biases from participants interacting with the same objects twice, we randomly split the participants into two groups (EXP1 and EXP2); this consideration also allowed us to offer the participants a comfortable experience in a reasonable time. Before starting the experiment, each person was asked to read an informative document and sign an informed consent to receive enough information about the data usage and the research scope in accordance with the principles of the Helsinki Declaration.

2.4. Questionnaire

Based on the subdivision of the experience into the two different interaction levels with the selected handicrafts, a questionnaire comprising three questions was developed to collect information about the emotions felt and the degree of involvement of the participants.

In order to evaluate the overall user engagement, some studies drawn from the literature on the representation of emotional dimensions have been considered. In particular, Russell developed the Circumplex Model of Affect [48], which is an emotional space wherein emotions are expressed through two independent dimensions: valence and arousal. The first dimension, valence, represents the positivity or negativity of emotions and varies between the extremes of unpleasantness and pleasantness [49]. The second dimension, arousal, concerns the degree of emotional arousal and ranges from deactivation to activation [50]. Another critical indicator in interpreting emotions is engagement, i.e., the level of attention and involvement during an activity [51,52]. These three indicators, i.e., valence, arousal and engagement, were evaluated at the end of the interaction with each object for the two phases and for the two experiences, using a 3-point scale. The questionnaire is summarized in

Table 1. Questionnaire for collecting emotional feedback for every object, phase and experience.

Questioned Indicator	Question	Answers
Valence	How do you rate the emotion evoked by the handicraft?	negative	neutral	positive
Arousal	How did the handicraft make you feel?	calm	neutral	excited
Engagement	How do you rate your level of engagement?	not engaged	neutral	engaged

.

2.5. EEG Affective Monitoring

Measuring visitors’ perceptions and engagement is crucial for museums seeking to enhance their exhibitions and create appealing experiences. Traditional behavioral and affective analysis methods often rely exclusively on self-reporting, which can be subject to limitations in expressing the complete emotional panorama. To address these challenges, physiological measures have emerged as a promising complement to objectively assess participants’ global experience and affective responses [52]. The use of EEG in the museum context provides an objective measure of the visitors’ experience and a way to characterize their emotions, particularly in terms of valence, arousal, and engagement. This provides valuable insights regarding the emotional influence of the artworks, in this case, handicrafts, which will lead to assessing the visitor perceptions with the final goal of optimizing their experience.

In this study, the EEG monitoring was carried out using a wireless headset consisting of 14 saline electrodes placed in accordance with the International 10–20 System [53]. The sensors are located at AF3, AF4, F3, F4, F7, F8, FC5, FC6, P7, P8, T7, T8, O1, O2 and two additional CMS/DRL reference channels at P3 and P4. Different studies have highlighted the usefulness of EEG-based indicators in evaluating and quantifying valence, arousal, and engagement, considering the association between emotions and brain frequencies [18,21,54,55,56]. The human brain’s theta ($\theta$), alpha ($\alpha$), beta ($\beta$), and gamma ($\gamma$) frequencies are influenced by several cognitive processes, including emotions. Theta is associated with relaxation, learning, and memory formation [57,58,59], alpha is associated with calmness and creative processes [57,60], beta is associated with alertness and concentration [22,61], and gamma is associated with attention and information integration [62,63].

Cognitive engagement is defined as the level of psychological investment that a subject makes while performing a task and the willingness to accomplish it [22,64]; considering that an increase in beta power is related to a cognitive activity increment during a stimulus and that increases in alpha are related to lower vigilance, derived from the EEG literature on attention, the following EEG-based engagement index has been systematically reviewed and implemented in emotion monitoring studies [65,66,67,68,69]:

$$Engagement=\frac{\beta }{\alpha }$$

(1)

Based on comparable research [22,70], the engagement will be assessed by aggregating the measurements from all the electrodes.

Regarding the valence, an indicator of the pleasantness or unpleasantness of the perceived emotion, researchers have found that the most analyzed EEG electrodes for the definition of valence are in positions F3 and F4 (frontal lobe). Furthermore, the frontal asymmetry has been studied as an expression of emotional states by comparing the difference between the natural logarithm of the left hemisphere alpha power ($\alpha$F3) and the right hemisphere alpha power ($\alpha$F4); the magnitude of this difference indicates the positivity of the emotion, with higher values corresponding to more positive emotions [18,54,71]:

$$Valence=\alpha F4-\alpha F3$$

(2)

On the other hand, arousal represents the intensity of emotion and how reactive or proactive someone is to a stimulus. Since relaxation is associated with alpha waves, while alertness is linked to beta waves, researchers have found that the ratio of beta and alpha measured in the frontal lobe (F3 and F4) can be an expression of a person’s arousal [18,72,73]:

$$Arousal=\frac{\beta F3+\beta F4}{\alpha F3+\alpha F4}$$

(3)

Due to the individual-dependent nature of physiological studies, there are no fixed maximum or minimum values for valence, arousal, and engagement. To address the variability caused by individual differences, this research utilized a within-subject design experiment commonly employed in similar studies [22,70,74]. The electrodes placement was carried out by trained personnel experienced in EEG data collection and familiar with the 10–20 system [53]. We adjusted the headset and electrodes comfortably for each participant, minimizing discomfort during the experiment. Furthermore, we monitored the EEG signals in real time throughout the data acquisition to ensure the electrode contacts remained stable. If any issues were detected, we made immediate adjustments to maintain signal quality. For each participant, the EEG data were separated into the baseline, recorded during eyes-closed condition, and the brain activity during the stimuli, in our case, during the visualization and interaction phases. By analyzing the brain activity difference between the baseline and the activated response during the stimuli, it is possible to identify the emotional indicators variations.

2.6. XGBoost for Engagement Classification

In consideration of the type of stimuli concerned in this study, a classification was performed on the engagement indicator. In fact, among the affective indicators selected and considering the two types of experiences the participants underwent, engagement is the most suitable to evaluate differences between real and VR experience for its relationship with immersion [75]. Moreover, craftsmanship implies the involvement of the craftsman with the material on both a physical and mental level [76], which seems reasonable to look for in the users as well. To this aim, EEG-based engagement feature vectors computed for each participant were labeled adopting questionnaire replies, which have been used as ‘ground truth’, according to the type of experience. An ML rather than a statistical approach was preferred in order to empirically verify the existence of a difference in emotional responses to the two experiences. ML does not require creating a mathematical model, as the model is created by the algorithm based on the data of the training set [77]. Indeed, for the type of experiment, conducted with no pre-specified interactions, it seemed reasonable to test the possibility of predicting membership to the first or second experience rather than finding relationships and interactions between variables and relying solely on the feature vector of engagement.

Among the different ML approaches to classification, Gradient Boosting was the one adopted. It is a widespread supervised approach in multiclass classification tasks. As one of the implementations of ensemble learning, the prediction of more than one model is involved to obtain the final output [78]. Gradient Boosted Decision Tree (GBDT) sequentially adds Decision Tree classifiers aimed at correcting the prediction made by the previous classifiers and outputs a weighted average of the predictions. To correct the previous predictions, at each training step, the correct observations have a lower impact than those misclassified. Considering the approach, these Decision Trees are considered as weak classifiers; their predictions are combined for votes or average, and the final output is weighted on the contributions of each model based on its performances. For each tree, nodes are added to optimize a non-linear objective, in this case, the square error.

Specifically, the Extreme Gradient Boosting (XGBoost, xgb Python package version 1.7.4) implementation of GBDT [79] was chosen for the present analysis. This choice was made considering that XGBoost offers some extensions to GBDT as the sparsity awareness, which allows handling missing values in data without imputation first [79,80]. Thus, it is particularly suitable for the purpose of EEG analysis, as during the process of removal of artifacts, some data could be missed. The Hessian is used to manage the non-linearity of the objective since the second-order derivative allows a more precise approximation of the direction of the maximum decrease in the loss function.

3. Results

The results are described in this section by grouping them into questionnaire replies, EEG-based emotional assessment via indicators and the classifier.

3.1. Questionnaire

The answers to the questionnaires were collected, grouping them by the following:

• Experience, to analyze differences between the real experience and the VR experience;
• Phase, to analyze differences between the degrees of involvement with the objects.

Figure 6 shows the responses to the questionnaires divided by experience (Figure 6a) and by phase (Figure 6b). Negative (negative, calm, and not engaged), neutral and positive (positive, excited, and engaged) feedback is represented on the graph in orange, gray, and green, respectively, with the relative percentages, while the median is represented with a yellow line. The mode, i.e., the number in a set of numbers that appears the most often, is always the positive class except for the arousal emotional indicator in the visualization and contextualization phase, where it is the neutral state. Comparing the responses to the questionnaire grouped by phase, the three indicators show an increasing trend passing from the visualization and contextualization phase to the interaction one, particularly for the arousal. Indeed, ranging from deactivation to activation, it is a good indicator for analyzing the difference between phases. The highest positive responses percentage has been registered for the engagement indicator in the interaction phase, with 71% of positive and 21% of neutral answers, meaning that participants declared being strongly involved while interacting with the real objects and the virtual twins. The valence indicator received fewer negative responses (≤3%) than arousal and engagement. Overall, on a displeasing–pleasant scale, participants did not report negative emotions during the experience.

As a further analysis, the main effects plots (MEPs) for each object, considered individually and collectively, have been reported in Figure 7 for valence, arousal, and engagement, for qualitatively comparing EEG and questionnaire data. When the line is not horizontal, there is a main effect, and the steeper the slope of the line connecting the response means for each factor, the greater the magnitude of the main effect. In this case, the graph highlights that the two phases affect the responses differently, with an increasing trend from the visualization and contextualization phase to the interaction phase.

3.2. EEG Emotional Assessment

After completing the experiences, the EEG data were processed (detailed preprocessing procedures are provided in Appendix A), and found artifacts were removed. The EEG activity was recorded at a sampling rate of 128 Hz and the bandwidth was between 0.2 and 45 Hz. To isolate the frequency bands, Fast Fourier transforms (FFTs) were calculated for the following spectral ranges: $\theta$ (4–8 Hz), $\alpha$ (8–12 Hz), $\beta$ (12–25 Hz), and $\gamma$ (25–45 Hz) for each phase of the experiment. Before performing the FFT, a high-pass FIR filter with a cutoff frequency of 0.18 Hz was applied. Then, a Hanning window on the EEG stream was implemented to improve the accuracy of the spectral analysis and reduce artifacts. In alignment with comparable studies in the field [18,22], we used a Hanning window size of 256 samples and slide this window by 16 to create a new window for each analysis epoch. Then, we applied an FFT to the most recent 2 s epoch of EEG data and averaged the FFT-squared magnitude across the frequencies in each band. Also, given that we used a relatively low number of electrodes (14), we opted for an artifact manual inspection and removal process. We visually inspected the EEG data to identify any remaining artifacts, major artifacts such as ocular (EOG) and muscular (EMG) were excluded from analysis; in cases where significant artifacts were present, we decided to exclude the entire subject’s dataset to avoid potential biases and ensure the reliability of the results. As a result, due to excessive noise or incomplete data, three subjects were excluded from the real experience (EXP1) and 17 were excluded from the VR HMD (EXP2), leaving a total of 30 (EXP1) and 45 (EXP2) artifact-free datasets for consideration. The raw EEG signals were recorded in two distinct phases: the first phase consisted of visualization and contextualization (VC), each lasting 30 s, and the second phase involved interaction (INT), also lasting 30 s. Upon completion of the preprocessing steps, a varying proportion of the original signal was excluded. This exclusion ranged from approximately 5% to 30% of the total recording. The variation in the removed signal percentage was influenced by multiple factors, such as the type and extent of artifacts and the applied filters; this means an average effective duration of the processed signals between 21 and 29 s.

Considering the conditions of the experiment and that the participants were the actual visitors who voluntarily took part in the experience, for data analysis, we decided to maintain the groups’ heterogeneity without discriminating on age or gender, and in this way, achieve consistent results with the museum’s real context.

Afterwards, for both the experiences, valence, arousal, and engagement indicators were calculated for each participant during each phase with the handicrafts, and the baseline data were subtracted from the post-stimulus response to assess the impact of the experiences and phases on the emotional indicators. Equations (1)–(3) were employed for emotional metrics assessment.

The overall activation responses in terms of band power, for each experience and phase, are presented in Figure 8.

Concerning the phases of the experience, during the first phase of passive visualization and contextualization (VC), participants experienced a receptive state while observing the handicrafts. In fact, the results show that participants’ theta activity incremented compared to the baseline; in particular during the VR, participants exhibited a higher theta when passively visualizing the handicrafts without any interaction.

Moreover, during this phase (VC), participants listen to explanations and contextual information about the handicrafts, enhancing their understanding and appreciation. Indeed, as the participants shifted their attention and became involved in active listening and comprehension during the contextualization, the demand for focused attention resulted in a lower alpha power compared to the baseline. This decrease in alpha activity indicates a shift from a relaxed state to a more attentive state [69,81].

Likewise, during the interaction phase (INT), participants had the opportunity to interact and manipulate the handicrafts, promoting a sense of agency and involvement. Beta and gamma frequencies are associated with cognitive engagement and active mental processing; the results show that participants presented a higher beta and gamma activity than the baseline during the interaction phase for both real and VR experiences.

Regarding the experiences, the real experience (EXP1) presented a physical and tangible interaction involving multi-sensory activities. When passively visualizing the real handicrafts, participants exhibited higher theta power, indicating a relaxed and reflective state. Also, higher theta power may be correlated with sustained attention and processing of the aesthetic characteristics of the handicrafts, especially during the interaction phase of the real experience [82]. Likewise, increased alpha power indicated focused attention and comprehension of the information provided. Similarly, beta and gamma increased during the interaction phase, reflecting engagement, attention, and active mental processing while manipulating the handicrafts. The real experience (EXP1) showed a steady increment of the frequency bands power during the development of the experience, reaching the highest value during the interaction phase. In particular, the increase in theta power during the interaction phase suggests engagement and cognitive demand during the handicrafts manipulation. This could be due to the increased mental effort required to explore and interact with the physical objects, leading to enhanced theta activity [83,84].

VR offered an immersive experience (EXP2) and a sense of presence inside the virtual environment [85,86]. During the first VR phase (VC), participants showed a slightly higher beta and gamma activity than the real experience; this could be due to the first impact with the VR, which increased cognitive engagement. Afterwards, beta and gamma remained steady.

Additionally, the difference trend in alpha power between real and VR experiences may be due to the cognitive load experienced in each setting. The real experience required participants to actively manipulate, touch and feel the handicrafts and listen to live contextualization. These cognitive demands can contribute to increased alpha and beta power as the brain involves in information processing and task control. In VR experiences, the cognitive load might be slightly reduced as the virtual environment is more controlled and guided.

Overall, the first phase’s (VC) impact was higher during the VR, particularly for alpha, beta, and gamma. While for the second phase (INT), the results were more evident during the real experience, especially for theta and beta. Regarding the orders of magnitude, the VR behavior showed similar patterns to those observed during the real experience. This can be attributed to the cognitive engagement, attention, and active mental processing associated with the immersive nature of the HMD as well as the sense of presence and embodiment that may lead to enhanced brainwave patterns.

Regarding the EEG-based emotional indicators and their relation with the questionnaire answers (Figure 7), the phase analysis revealed a consistent pattern in participant preferences and EEG outcomes.

Examining the questionnaires and comparing the two phases of the experiments, participants’ responses, as well as the EEG emotional indicators, consistently showed a preference for the interaction phase for all the handicrafts. While the visualization phase allowed participants to appreciate the handicrafts and gain knowledge about them, their responses indicated that it might have been a less engaging experience. Conversely, the interaction phase appeared to significantly enhance participants’ perceptions and engagement. The EEG data further corroborated this trend, revealing lower values for valence, arousal, and engagement during the visualization phase and higher values during the interaction. The alignment between self-report preferences and EEG outcomes suggests that active involvement and manipulation played an important role in elevating the overall emotional experience. The act of interacting with the handicrafts allowed the participants to create a deeper connection, developing a greater sense of enjoyment and emotional engagement.

In relation to the real and VR experiences, the EEG-based emotional indicators results revealed distinct patterns in the participants’ cognitive responses (Figure 9).

Notably, the VR experience (EXP2) produced the highest cognitive outcomes, eliciting higher levels of arousal, valence, and engagement. This finding highlights the immersive nature of the HMD VR, which successfully captured participants’ cognitive attention and generated positive emotional responses.

In terms of the real experience (EXP1), it came in second place but always showed positive results with respect to the baseline, indicating that participants generally experienced positive emotions and were engaged.

Considering that the cognitive engagement reflects the focused mental effort in understanding complex ideas and tasks [64], it is important to note that the cognitive engagement during the VR experiences could also respond to the novelty associated with the VR compared to the real visit. Participants might have been less familiar with this technology, making it a more fresh and intriguing experience.

Overall, the results indicate that the VR experience provided the most evident cognitive outcomes. These findings highlight the potential of HMD VR as a promising tool for providing immersive and emotional experiences. On the other hand, the real museum setting is able to offer a unique in-person experience that holds the importance and significance of the physical interaction with the handicrafts; the sensations produced by the physical senses, such as visual stimulation, olfactory, and tactile stimuli, are elements that can only be fully experienced in the real context. By taking advantage of the strengths of both worlds and combining the interactive capabilities of VR, museums can enhance their traditional visits, offering additional virtual content that enriches the understanding and emotional engagement with the artworks.

3.3. Measures of Variability of Power

To illustrate the variability of power across subjects, for the Tatà, one of the most representative handicrafts, we have calculated the standard deviation, the mean, the variance, and the coefficient of variation (CV) as the ratio of the standard deviation to the mean, which is expressed as a percentage for each power band across participants for both Phase 1 and Phase 2 and for both the real museum (EXP1) and the VR (EXP2).

Table 2. Measures of variability for the Museum Experience (EXP1) Tatà—Phase 1.

	Mean_Phase1	StdDev_Phase1	Variance_Phase1	CV_Phase1	Median_Phase1
Theta	8.38	7.60	57.79	90.73	5.50
Alpha	2.5	1.61	2.60	64.09	1.93
Beta	2.39	1.13	1.28	47.50	2.02
Gamma	0.68	0.38	0.14	55.69	0.62

and

Table 3. Measures of variability for the Museum Experience (EXP1) Tatà—Phase 2.

	Mean_Phase2	StdDev_Phase2	Variance_Phase2	CV_Phase2	Median_Phase2
Theta	12.71	13.69	187.51	107.74	7.33
Alpha	4.41	4.68	21.88	106.16	2.57
Beta	4.71	3.47	12.05	73.73	3.87
Gamma	2.26	2.92	8.52	129.02	1.49

show the measures of variability for the Museum Experience regarding the Tatà handicraft for Phase 1 (VC: Visualization and Contextualization) and Phase 2 (INT: Interaction).

In Figure 10, there are the box plot representations of theta, alpha, beta, and gamma variability measures for the Museum Experience. Plus signs represent outliers based on the Interquartile Range (IQR).

The observed changes across the power bands between the two phases align with the expected increase in cognitive and motor activity during Phase 2, and they support the idea that Phase 2, involving direct interaction with objects, is associated with greater cognitive engagement and complexity. The high CV values in both phases (VC and INT) may indicate a significant variation in how visitors individually respond to the stimuli. This could be interpreted as the intrinsic diversity in visitors’ cognitive engagement and emotional responses to the handicrafts.

Similarly, we have calculated the measures of variability for the VR Experience (EXP2), for both Phase 1 and Phase 2, for the Tatà, in

Table 4. Measures of variability for the VR Experience (EXP2) Tatà—Phase 1.

	Mean_Phase1	StdDev_Phase1	Variance_Phase1	CV_Phase1	Median_Phase1
Theta	9.09	6.80	46.29	74.87	7.08
Alpha	3.25	2.29	5.24	70.48	2.68
Beta	3.38	2.02	4.07	59.66	3.01
Gamma	1.23	0.95	0.91	77.86	0.97

and

Table 5. Measures of variability for the VR Experience (EXP2) Tatà—Phase 2.

	Mean_Phase2	StdDev_Phase2	Variance_Phase2	CV_Phase2	Median_Phase2
Theta	8.90	11.77	138.44	132.13	4.13
Alpha	3.32	3.48	12.10	104.86	2.25
Beta	3.50	2.52	6.37	72.05	2.69
Gamma	1.29	1.06	1.13	82.52	0.99

as follows:

In Figure 11, there are the box plot representations of theta, alpha, beta, and gamma variability measures for the VR experience.

In the VR experience (EXP2) and Phase 1, the relatively high mean values across the power bands may indicate an engaging initial experience in the VR. The CV percentages (especially for theta and gamma) indicate a variation in response, possibly reflecting different levels of engagement with the virtual content. In Phase 2, the slight decrease in theta mean and the increase in the CV may indicate less consistent cognitive engagement during the interaction with the virtual handicrafts. The other bands’ values remain relatively stable, but the increased CV in alpha and the decrease in beta suggest more individual variation in experiences during this manipulation phase.

The results from both environments highlight some essential differences between physical (EXP1) and virtual (EXP2) experiences with the museum’s handicrafts. The VR experience seems to offer a more consistent and a more immersive introduction (Phase 1) but shows a different pattern of cognitive engagement during the interaction (Phase 2) with less pronounced theta activity.

Both sets of results offer insights into how visitors respond to these two different modalities, providing a basis for further investigation into how to use virtual and physical spaces for museum contexts.

3.4. XGBoost Classification

A binary classification was conducted to empirically assess a difference between the engagement experienced by participants in EXP1 and that experienced by participants in EXP2. The f1-score metric was adopted to evaluate the classification results. In this way, the imbalance in the number of observations in the two classes was taken into account. In detail, the f1-scores on the test set (0.3 of the complete dataset) were 0.65 (precision = 0.81, recall = 0.54) for the classification of the EXP1 and 0.81 (precision = 0.74, recall = 0.91) for the classification of EXP2. The confusion matrix is reported in Figure 12. A higher prediction accuracy is reported for EXP2. However, also for EXP1, the majority of predictions are on the diagonal of the confusion matrix, and the f1-score is higher than 0.5.

4. Discussion

Our study used an EEG-based engagement index to measure how engaged people are when they are interacting with a handicraft in a museum context. These kinds of metrics have been used successfully before for distinguishing brain attentive states and determining the cognitive workload, helping researchers to understand how focused and interested people are in different situations [68,74,87,88]. In the specific naturalistic situation addressed, this measure helps us see how much attention and interest a person has in a specific piece of handicraft. Other studies have shown that this approach can even predict what pieces of art people might be most interested in [21,89]. In this regard, these metrics are a good fit for understanding how people experience a museum and connect with the art they see.

The real experience provided a multi-sensory real environment involving visual perception, spatial awareness and tactile interactions. The initial theta power during the passive visualization of the real handicrafts can be attributed to the sensory stimulation and cognitive load associated with processing the onsite location. The immersive nature of the VR experience, on the other hand, may require less cognitive effort during passive visualization, leading to lower initial theta power. Previous studies linked theta waves to various cognitive processes, including working memory, mental effort, and the maintenance and updating of information [58,59,90]. The power of theta activity tends to increase with a higher cognitive load when a task requires sustained attention and selective focus, reflecting the involvement of cognitive resources and information integration. In this sense, considering the design of the proposed experiences, when the cognitive load increased theta tended to increase, this is clearly confirmed in the real experience, where participants needed to remember and process multiple pieces of information. Indeed, both phases of the real setting context, i.e., visualization and contextualization (VC) and interaction (INT), were conducted by a guide; in particular, VC took place with the guide personally sharing stories and explanations about the actual objects. Quite the opposite, in the VR context, the VC was conducted using audio recordings without any physical intervention of the guide. This distinction and the opportunity to touch the authentic handicrafts produced a more evident multi-sensory elicitation in the real experience, which is reflected in a stronger activation of theta waves compared to the VR counterparts. On the other hand, the HMD experience produced brain wave patterns similar to those of the real visit. The HMD’s ability to create a sense of being present and part of the virtual environment evoked similar outcomes to those experienced during a physical museum visit. The EEG emotional indicators and questionnaires analysis showed a clear preference among participants for the interaction phase, revealing higher levels of engagement and positive emotional experiences. The classification of the engagement showed differences in the patterns of the two experiences. For both classes (Exp1 and Exp2), f1-scores higher than 0.5 were found, showing that the classifier was able to find differences between the engagement feature vectors of participants to the two experiences. The highest f1-score resulted for the VR experience (EXP2), suggesting that EXP2 produced more particular results in the engagement experienced by the participants. These findings suggest that the virtual environment designed ad hoc to show and contextualize the selected handicraft objects is a valuable technology for creating immersive and interactive experiences in the considered cultural heritage scenario. Compared to the real experience, participants in the VR experience have been able to observe the four macro-areas hosting the objects, listen to sounds that recall their manufacture or intended use, and move around the mountain village. The head-mounted device to display the VR environment and VR controllers to allow interaction with the virtual twins have increased the sense of presence, i.e., the feeling of “being there”, boosting cognitive processing and providing a highly emotionally and cognitively stimulating environment. On the other hand, by analyzing participants’ self-reported assessments, positive perceptions toward the real visit have emerged. This suggests that while VR may elicit higher cognitive engagement, the tangible and concrete aspects of the physical museum visit present a unique appeal to create enjoyable experiences for participants. The empirical results of the classifier support the consideration that can be made on the aggregated analysis of the affective indicators that highlighted significant differences in the cognitive attention between the real visit and the VR experience. These results should be considered along with the self-reports overall outcomes, where a slight preference for the real experience emerged. In this viewpoint, the result supports the suggestion that an integration between the two types of experience can enhance the quality of the museum visit where a multi-sensory elicitation due to the physical presence in the museum space can be completed by virtual interaction with the objects. In fact, in a real scenario, objects can not be physically available to museum visitors, but a virtual solution based on HMD could be successfully adopted. Moreover, individual differences can also impact the emotional characterization within each modality. Factors such as personal preferences, prior experience with VR, previous knowledge of traditional handicrafts, and the specific design and presentation of the experiences can all contribute to the emotional responses observed. Also, background factors may contribute to the observed differences. For example, participants’ familiarity with VR technology and their prior experiences with these handicrafts could influence their emotional responses. Additionally, the specific characteristics of the objects, such as their uses, crafting, traditions, or personal relevance, might impact the cognitive and emotional responses. While the analysis of these aspects was not the primary focus of this research, it presents an exciting opportunity for prominent future investigations in this field.

5. Conclusions

In summary, the findings of this study highlight the significance of active interaction and hands-on engagement with the handicrafts in evoking positive emotional responses and fostering high levels of cognitive engagement. The initial impact experienced during the visualization phase, coupled with the contextual understanding, contributes to the participants’ overall emotional journey enhancing the interaction phase. By integrating these findings into the design of museum experiences, curators and designers can create immersive and engaging environments that leverage the power of interaction, contextualization, and emotional connections to enhance visitors’ overall experience and appreciation of the handicrafts. These findings have exciting implications for future developments in virtual museum experiences in creating engaging virtual experiences that closely reflect the real setting providing visitors with immersive and authentic encounters. It also highlights the potential of HMD-based VR as a valuable tool for enhancing museum accessibility and reaching broader audiences. Additional research avenues in the field of museum experiences could be employing advanced immersive VR technologies, such as the CAVE system, to increase the museum’s affective exploration. Additionally, extending our research to involve museums with distinct themes or cultural backgrounds can enrich the understanding of how different settings and styles may influence visitors’ engagement and emotional responses. Furthermore, integrating a multi-modal approach with additional physiological metrics, such as Galvanic Skin Conductance (GSC), heart rate (HR), and facial expressions (FER), can provide a more comprehensive assessment of visitors’ engagement, perceptions, and overall satisfaction.