Evaluating Uses of XR in Fostering Art Students’ Learning

Abstract

This paper addresses the potential of extended reality (XR) to foster art students’ learning and creativity with specially developed applications for the creation of art exhibitions. This study is based on the EU-funded research project ‘Scaffolding Creativity of Arts Students: Framework, Toolchain, and Educational Material on how to Create their Own Virtual Exhibitions’ (CREAMS). CREAMS develops interconnected virtual reality (VR), augmented reality (AR), and mixed reality (MR) indoor and outdoor exhibition creation applications. The article puts emphasis on the issue of what evaluation methodology is more adequate to assess the efficacy of such applications, and pertinent challenges, as well as related research, are discussed. In this context, ways in which Fine Art School undergraduates can benefit from the creation of XR/VR exhibitions are explored. Such uses of XR are primarily examined in terms of their potential to enhance learning, and foster students’ skills. The potential of adding multimodal resources that contextualize exhibited artworks is examined to foster viewers’ meaningful engagement. Art students’ ability to communicate the underlying concepts/ideas of their art through XR apps is also addressed. Moreover, this paper investigates how XR technologies can foster the collaboration of tutors and students in Fine Art Schools through specially developed platforms embedded in the XR applications. The ability of the CREAMS VR application to foster such cooperation between students and their tutors is also evaluated. The scientific contribution of this paper relates to the evaluation methodology of XR art exhibition applications that have an educational role.

1. Introduction

Emerging technologies, such as extended reality (XR) applications, provide new opportunities but also challenges in relation to their adoption as learning tools by Higher Education Institutions (HEIs) in the Art and Design domain [1,2,3,4]. Art schools explore the capabilities that XR heralds and adapt ICT tools to their needs and requirements. This paper refers to the characteristic case of the EU-funded research project CREAMS (full title: ‘Scaffolding Creativity of Arts Students: Framework, Toolchain, and Educational Material on how to Create their Own Virtual Exhibitions’). An essential aspect of harnessing the potential of emerging technologies for HEIs is to enhance students’ ability to use such tools effectively.

The challenge of exploiting the VR medium’s expressive capacity, as [5] posits, is a key factor. Arts-related interfaces should be innovative, inspiring and creative themselves.

Gifreu-Castells in [5] delineates the objectives of a VR art exhibition, which, depending on the specific curatorial aims and priorities, can significantly vary. According to the author, their first function is to attract new audiences. Moreover, they may aim to reduce the so-called digital gap. Another aim is to place emphasis on different curatorial possibilities. The medium’s expressive capacity represents a significant challenge for art schools as a VR art exhibition should exploit as much as possible. The same holds for the ability of a digital platform to foster curatorial practices through experimentation and communication between students and tutors or even amongst peers. The above points delineating prospective benefits and possibilities for art students are reflected upon the metrics and evaluation criteria, as well as the questions or discussion topics used (respectively in questionnaires and focus group sessions) that are further analyzed in this article. At the same time, beyond qualitative aspects that relate to the specificity of a visual arts-related set of applications, we concurrently assessed with the use of a mixture of more pragmatic and commonly evaluated aspects, namely, usability and user experience. This paper offers an overview of the actual CREAMS EU-funded research project in the following section and provides an overview of methodological considerations, methods employed, the actual findings, and a discussion thereof before delineating some concrete suggestions for both practitioners in the field of XR evaluation and stakeholders in the field of tertiary art education on the basis of the projects’ consequent phases of formative and summative evaluation.

The paper presents the outcomes of an evaluation study that, in turn, draws on an ever-increasing body of published research and reviews in the field of evaluation methodologies pertinent to extended reality with a focus on virtual cultural experience. The work of Kabassi [6], who published a comprehensive systematic review, has informed both our research and this paper. In a similar vein, Morales et al. [7] map out pertinent publications in an extensive survey. Morales et al. [7] provide the key methods employed in XR-related evaluation studies that often use a methodological approach similar to the present study (i.e., mixed methods with the combined use of the SUS and UEQ questionnaires, which will be explained in the Methodology section of the present paper), such as the work of Barricelli et al. [8]. Likewise, another pertinent review by Garcia et al. [9] presents evaluation studies that often address culture-related material, such as the publication by Campoverde-Durán et al. [10].

This article puts emphasis on the uses of extended reality/XR (with a focus on VR) as a tool that enhances learning and creativity in art schools through the creation of virtual art exhibitions and, as mentioned, examines optimal ways to evaluate the efficacy of such applications. In a recent publication by Vital et al. [11], there is a comparison of extended reality platforms and tools for viewing and exhibiting art, as well as a thorough discussion of the impact of such technologies on teaching and learning in art schools. Vital et al. [11] explain that there is ‘a growing interest in scientific production regarding the use of VR applications of virtual reality in higher education for art education’ [1]. Other examples include Huaman et al. [3], who created a virtual learning environment based on a VR museum that fostered active learning according to the results of their research. As Vital et al. [11] mention, other uses of VR exhibitions, such as those presented by Qiu et al. [4], put emphasis on promoting the work of the art students, something that also contributes to their professional skills and overall training. In a similar vein, Song and Li [12] posit that VR can help to create effective technology-based teaching environments, to improve students’ professional skills, to assist them in better understanding theoretical knowledge by an improved teaching quality, and finally to better integrate into society. However, they note that teaching content and teaching is the key to the adaptation of virtual reality to their courses [11,12]. This is the key aspect that this paper seeks to capture in this evaluation of the CREAMS research project, which envisages the creation of XR tools that are, by design, meant to be embedded in art school modules to enhance learning and art students’ skills. As stated, the main contribution of this article is the methodological framework that has been developed to undertake this task.

2. CREAMS Project: An Overview

The main CREAMS application, which is the virtual reality platform, is a web-based application. It includes a variety of tools that enable students to manage their digital artworks and create and manage virtual exhibitions, and that enable instructors to manage students’ exhibitions and assess the students’ artworks and exhibitions. Key components (Figure 1) include: (i) the curatorial component, which serves as a communication interface between instructors and students for receiving feedback on the selection and grouping of artworks and reasoning of the students’ selections before the actual virtual exhibition creation takes place; (ii) the virtual reality workshop, which consists of a set of tools and functionalities enabling students to create their virtual reality exhibitions by assigning their digitized artworks in a custom-made exhibition space customized by the students. Digitized artworks include 2D and 3D artworks, as well as videos and sounds. Additional tools enable students to adjust their digital artworks, assign and manage associated material related to the main artworks, and apply and adjust the lighting in the virtual space (ambient and spotlights); (iii) teaching services, enabling instructors to assess the exhibition projects created by the students; (iv) dissemination services, enabling visitors to share exhibitions in their social media; and (v) personalization and recommendation services, which recommend best-fit exhibitions to visitors based on interest-based modeling and collaborative filtering mechanisms. Personalization is mainly based on keywords associated with exhibitions that enable recommendations for users who may wish to see similar art shows.

The CREAMS research project aspires to foster educational exchanges amongst various stakeholders in the Higher Art Education domain. To this effect, a main platform that serves as the virtual reality exhibition editor and viewer has been created. This VR editor is the main CREAMS platform and is related to other additional augmented and mixed reality applications that enable art students to show their 2D or 3D work in virtual spaces, as well as in indoor and outdoor actual spaces, respectively. An auxiliary digitizer application complements these apps. This paper focuses on the VR platform as it has a pivotal role. Students can upload artworks onto the VR platform with the help of the digitizer app. Students can view a dashboard containing information about the assigned exhibitions (Figure 2) and the uploaded artworks (Figure 3).

Moreover, students can create an exhibition space or choose an available gallery template, then insert and arrange the artworks (Figure 4). They gradually engage in curatorial choices with the help of their tutors with the use of appropriate interaction provisions. Together, they specify the overall exhibition narrative or underlying concept by adding texts, information, and multimodal material associated with their work. Moreover, space-related choices are an essential aspect, as both the focus groups’ discussions and the reviewed bibliography indicated.

3. Evaluating XR in Tertiary Art Education: Methodological Considerations

Learning in the field of tertiary art education, where art students develop their skills, gain insights, and acquire knowledge, is far from being a solitary affair: their learning processes are mediated through interaction with others as well as with tools and resources. According to Vygotsky [13], objectives form one of three elements from which to start designing learning, the other two being the subject (the learner) and the tools for learning. In the context of tertiary art education, innovative digital tools that enable art students to enhance skills critical for their development as visual artists are, therefore, of great significance. In turn, the ability to establish a sound and dependable methodology to evaluate the impact of new technologies, such as that of XR applications on art students’ learning, is key.

The development of XR technologies for the needs of HEIs is a reflexive process: Researchers investigate ways to foster learning and teaching through XR based on students’ and other stakeholders’ feedback on the efficacy of applications designed to meet their needs. This paper focuses on the case study of the CREAMS evaluation methodology to inform respective efforts in the sector of Fine Art Schools. The paper, moreover, discusses the potential of extended reality (XR) to foster art students’ creativity by using specially developed applications that draw on the findings of the CREAMS research project. The development of the tools and the respective evaluation methodology were based on initial needs analysis regarding art students. Early insights provided guiding principles informing the structure and characteristics of the main VR exhibition platform. In turn, the accrued tools were assessed and reconfigured to reflect the evaluation findings. The digital tools incorporate the amendments from the feedback received in two separate formative evaluation phases. The principles that informed our methodology comprise the core scientific contribution of this paper, as opposed to the findings that accrued at the present phase of the application’s development. Moreover, concerning the novel characteristics of the VR exhibition editor application, the aspect that differentiates it from existing ones is its focus on educational functionalities and role. This is another area in which this paper seeks to offer inspiration to researchers in the broader field, by presenting an educational use of VR/XR exhibition platforms.

During the needs analysis phase of the CREAMS project, we conducted in-depth discussions, employing focus groups and doing desk research involving a review of the pertinent literature and case studies. Beyond the needs analysis evaluation, phases involved focus group discussions; more specifically, three separate focus group discussions with about eight participants (eight art students, as well as one or two coordinating tutors), one in each of the three participating art schools, in each of the three evaluation phases, i.e., first, second formative, and summative evaluation, so in total nine focus groups sessions took place. Comments gathered through these semi-structured focus group discussions provided an outline of how XR can support learning in art schools. The key benefits of the XR exhibition’s creation for art students are enhancing their ability to share, curate, and contextualize their art practice with innovative means. More specifically, art students, using such tools, can experiment with curating choices in terms of the spatial arrangement of artworks and, most importantly, explore ways to communicate concepts or narratives underpinning their (virtual) art show. Art students can create a virtual gallery space through a modular floorplan design or choose a digital clone of existing galleries in their art school. Moreover, they can exploit digital tools to associate contextual information and multimodal resources with exhibited artworks. Thus, they can better situate their practice in terms of inspiration, influences, and related art/research and reflect on their creative process. The ability of virtual exhibitions to embed resources that frame artistic practice in ways that are not feasible in actual gallery spaces opens a gamut of possibilities. However, art students will have to acquire additional digital skills and adapt the use of digital tools to their aims. This involves the ability to put artworks, multimodal resources, and spaces in synergy.

Learning is never tool-free [13], and is conditioned by the qualities, abilities, and affordances of given or new tools. Virtual exhibitions can offer multiple layers of contextual material that frame the artwork, thus offering opportunities for art students to reflect upon, conceptually frame, and communicate their artistic practice and research. Such contextual material can itself be aesthetically and conceptually enticing, challenging, and inspiring, which becomes, in a way, a digital extension of the artwork. These essential qualitative aspects of learning and acquiring skills are often elusive and quite challenging to evaluate in terms of establishing adequate metrics and methods to gather data.

3.1. Related Work

Publications such as the recent survey by Sylaiou et al. [14] not only address the increasing importance of extended reality (XR) in art exhibitions by mapping the existing terrain but also put emphasis on the evaluation of such applications. The evaluation methodology, pertinent metrics, and means in the art- and culture-related XR applications have increasingly become the topic of extensive research. A characteristic example is a thorough survey [6] that reviewed pertinent publications on the evaluation of virtual exhibitions. This survey shows that criteria vary considerably depending on technologies employed (e.g., immersive VR environments, as opposed to websites or web-based applications) or evaluation methods (namely, empirical or inspection methods). More specifically, as Kabassi [6] notes, relevant evaluation studies typically employ empirical methods, i.e., questionnaires or interviews that involve end-users. On the other hand, inspection methods involve specialists who offer insights through cognitive walkthroughs or other gathering data methods. Often, researchers employ a combination of empirical and inspection methods, as is the case with the CREAMS evaluation methodology.

Moreover, criteria in all the above cases also depend on the researchers’ priorities, and therefore different authors in published research foreground specific (sets of) criteria that often do not overlap. More specifically, as Kabassi [6] shows in a table included in her survey (p. 9), which describes museum-related VR environments’ evaluation criteria, foregrounded by eight researchers in 10 publications, there are significant discrepancies. For example, specific criteria such as the sense of presence or immersion were employed in some publications [15,16], and likewise, the factor of enjoyment [15,17] or, for the same matter, narrative [18] in others. Concerning VR environments in which cultural content is exhibited, some criteria are almost universally accepted (albeit not without exceptions), such as ‘Orientation-Navigability’ and ‘Usability’. Methods (e.g., questionnaires, discussions in small focus groups), type of data gathered (qualitative or quantitative), and, as mentioned, metrics or criteria do not seem to adhere to a rigid methodological framework that is commonly agreed upon, but instead accrue from the specificities of the researchers’ aims and priorities, as well as the specificity of the VR/online experience under research.

Firstly, regarding the sense of presence, especially in VR environments, the capability of a system to support the feeling that users are ’there’ in front of an artwork is assessed. As Lombard and Ditton [19] put it, in immersive experiences, users have the illusion of non-mediation. Realism and plausibility of the environment are considered key [20,21], to draw and keep users’ attention, and foster engagement. A comprehensive overview of the concepts related to and affecting presence is offered by Skarbez et al. [22]. The ability of a system to sustain meaningful attention and cognitive engagement apart from realism and fidelity of sensory input (two closely related aspects of a VR/XR experience) are guiding to what can be termed as cognitive presence as the central and pivotal characteristic of an arts-related experience. This relates to the fact that users will inevitably compare the level of similarity and, thus, congruence between real objects (in this case, artworks) and virtual representations of them [15]. Beyond the sensuous aspect of art perception and appreciation, viewers will have meta-thoughts on the virtuality of the experience, as described by Hofer et al. [23]. They will also compare their ability to engage with the artwork within a VE to real-world visits to an actual gallery space. According to Weber et al. [20], presence can also be associated with the plausibility of narratives embedded in or underpinning a virtual experience, which is crucial for art exhibitions that typically are formed around a concept or a narrative in the broader sense of the term. The term ‘perceived realism’ is described as the user’s judgment about the degree of realism of the VE in terms of (1) virtual objects, sounds, and scenes, (2) credibility and plausibility of the narrative, and (3) naturalness and ease of the interaction with the VE [20]. Narratives (e.g., virtual humans reciting a story about an exhibit) play a crucial role in perceived realism and the sense of cognitive presence [24].

Finally, a pertinent evaluation method comes from published research on an MR museum experience by Hammady et al. [25]. This research introduces a comprehensive and relevant theoretical scheme as well as a methodological framework for evaluating a museum-related XR experience. Although the Role of the Guide (virtual human) is central in the methodology proposed in this study, the overall structural interrelations of the other factors and evaluation metrics proposed (which jointly contribute to the main desired outcome, which is users’ Intention to Use), provided inspiration to the CREAMS evaluation, as outlined in more detail in the following section.

3.2. Methods and Tools

The CREAMS framework for evaluating virtual exhibitions was initially informed by stakeholders’ needs analysis and a thorough review of pertinent studies. The three major areas that define the quality, effectiveness, and usefulness of an XR experience in the fields of the arts and cultural heritage can be summed up as follows: usability [26,27,28,29], sense of presence [30,31,32,33], and learnability [16,34]. These key areas comprise the main evaluation topics concerning XR experiences in the arts and visual culture domain.

The methodology of evaluating the applications, e.g., the VR platform (exhibition editor), is based on two pillars: chiefly, questionnaires, and secondly, focus groups. During the second formative evaluation phase, 81 art students participated in total and 198 art students in summative by responding to questionnaires. The sample size is deemed representative of the art schools that participated in the research project itself, namely Aristotle University of Thessaloniki, Greece—Department of Visual and Applied Arts; SHENKAR, Israel; and NTNU—Norway, a number that covered the largest part of the fourth-year art students who prepare their final degree show. The selection process of the participants in the evaluation was based on the students’ degree of familiarity with the system, as answering the majority of questions required that the responders had a hands-on experience with the CREAMS VR platform, having used it to create a VR exhibition through its main application, within the context of taught modules in cooperation with the help of tutors who participated in the CREAMS consortium. This study focuses on this specific category as the participants were both users and content creators, while, moreover, the application itself was primarily developed to answer their needs. Bias (and in particular positive bias) has been controlled by constantly communicating to participants the need for commentary and feedback through the evaluation process that would allow developers and other researchers to improve the tools in accordance with the actual needs, recommendations, and critique, as opposed to expecting responses that would provide a more positive image of the application(s) assessed. Moreover, questionnaires during the three consequent phases of the evaluation (two formative and one summative) were reviewed and improved when the wording of questions appeared to encourage positive bias, e.g., by using the initial phrase ‘how would you rate…’ as opposed to wordings that could bolster positive bias.

Furthermore, 33 visitors responded to (different) questionnaires about the VR platform alone (2nd formative valuation) and another 32 during the summative evaluation phase. Additionally, smaller numbers of art teachers and ICT specialists with experience in XR apps took part. Moreover, focus groups involved 8 participants plus the coordinator. Semi-structured interviews and discussions with participants in focus groups allowed us to gather qualitative data. Art students were initially familiarized with the VR platform in the context of taught modules such as curatorial studies and art education specific classes.

The questionnaires comprised mainly of Likert-scale type questionnaires (with five levels of scale, and seven, in the UEQ sets of questions described below), as well as three or four open questions depending on the participants’ category (i.e., three for art students and visitors, and four for art teachers and ICT specialists) to which responders could provide written answers. Questionnaires were conducted online, and responders were invited via email. Questionnaires were composed of three main parts: firstly, questions adapted to the specificity of the art exhibition editing tools, which were chiefly inspired by presence questionnaires. A set of questions deriving from the System Usability Scale (SUS) [35] was also used, while another set of questions corresponding to the User Experience Questionnaires (UEQ) [36,37,38] formed the third cluster of questions. As the evaluation methodology progressed during the research project, UEQ has been identified as an adequate tool given that it puts emphasis on both pragmatic aspects (e.g., usability) and subjective, personal responses to the VR/XR experience. UEQ is commensurate by assessing the experience of users who view or create virtual exhibitions, given that practical aspects (e.g., reliability) and so-called hedonic aspects [36,37,38] (i.e., stimulation and innovation) are crucial for the overall attractiveness and efficacy of such applications, as Figure 5 illustrates.

The overall structure of the art students questionnaire was as follows: the first set of ten questions (based on a 5-step Likert scale) asked the participants to rate the key functions and processes mainly from a usability point of view. The following set of ten questions (Q11 to Q20, also based on a 5-step Likert scale) corresponded verbatim to the System Usability Scale (SUS) The following set of questions (Q21 to Q27 for the second formative evaluation) put emphasis on the more qualitative aspects of the user experience in relation to learning, and possible enhancement of curatorial and reflexive or critical skills. These questions also used a 5-step Likert scale. At this point, three open questions enquired about which aspects of the platform were most or least liked, as well as suggestions for improvement. The questionnaire closes with the UEQ set of questions (main results are outlined in Tables 4 and 5, as well as in Figure 6 and Figure 7), which is based on a 7-step Likert scale. Lastly, the questionnaire invites participants to rate their familiarity with pertinent applications.

The methodological approach proposed by Hammady et al. [25] pertains to the CREAMS evaluation process as it highlights key areas of interest, thereby informing the questionnaire structure as well as outlining a categorization framework for written commentary in open questions and answers obtained during focus group discussions. More specifically, the terms presented by the authors are the following: enjoyment, immersion, multimedia and UI, storytelling (where applicable), usefulness, ease of use, interaction, lack of distractions (i.e., content is not distracting), independence (visitor gets privacy/more independence), overall satisfaction, and finally, the willingness for future use [25]. For example, ‘Independence’ in the way it is framed is a crucial aspect, considering the emphasis put on the ability of the CREAMS users to engage with the exhibits conceptually, which requires unintrusive interface provisions, thereby enabling viewers to concentrate. This has been described as the system’s ability to support cognitive presence and is, therefore, seen as a significant aspect of CREAMS to be evaluated. The publication of Hammady et al. [25] consolidated our focus on qualitative aspects of users’ art-related experience as conveyed by the choice of questions described in Table 2 (questions 21 to 22).

3.3. Results of the Second Formative and the Summative Evaluation

A key finding was the need for users to have a more intuitive design in relation to navigation. Participants asked for more guidance on how to navigate virtual gallery spaces. Users frequently criticized the parallel use of arrow keys and mouse as confusing. In post-evaluation meetings, we decided to address this issue with enhanced guidance provisions. Moreover, a widespread wish for an even more realistic rendering of the spaces and artworks alike has been recorded. Lastly, a few responses addressed some lingering technical issues related to the limitations associated with open-source software. The main contribution of this paper is on the evaluation methodology. Thus, the results presented in this section are simply indicative of the insights that may be gained through the methods employed.

Regarding demographics, the visitors’ experience in generic Virtual Environments was slightly over the mid-point, or average value, and their experience of arts-related VEs was slightly lower (about 3 on a scale of 1 to 5). The interesting finding is that the responders’ interest in contemporary arts is exceptionally high (4.5), whereas this app’s effect on potentially fostering (further) this interest was lower, standing at 3.5. This can be seen as a clear indication that irrespective of the platform characteristics, art exhibitions with high aesthetic and curatorial standards should be used to measure the potential of such apps more effectively.

Art students mostly provided a much more favorable assessment of the CREAMS VR editor platform. This can be attributed to their interest in using a cutting-edge tool, which generates both excitement and expectations. The main point identified that needs further improvement is firstly the perceived lack of realism. Moreover, students required a more clear-cut interaction context, i.e., the ability to organize associated material to artworks, e.g., through a user-friendly menu.

The Art Student VR Exhibition questionnaire, as mentioned in the Methodology section, is a composite concatenation of separate clusters that are either adapted (sets of) questions drawing from existing XR-related questionnaires or blocks of questions corresponding to the User Experience Questionnaire (UEQ, based on a 7-step Likert scale) as well as the System Usability Scale (SUS), based on a 5-step Likert scale. The adapted questions (also based on 5-step Likert scale) focus on qualitative aspects of CREAMS’ applications.

The first set of ten questions (i.e., Q1 to Q10; see

Table 1. Usability related questions on VR platform, art student responses.

QUESTION	Second Formative Evaluation	Summative Evaluation	Explanation of 5 Likert Scale Steps
Q1. How would you rate the sign-up interface in terms of effectiveness?	4.16	4.05	points 1 to 5 stand for the following: (1) Very difficult/cumbersome to use; (2) Somewhat difficult/cumbersome to use; (3) Neither easy nor difficult to use; (4) Fairly easy to use; (5) Very easy to use.
Q2. How would you rate the interface regarding the creation of the exhibition in terms of effectiveness and ease of use?	3.72	3.98
Q3. How would you rate the interface regarding the uploading of your artworks on the platform in terms of effectiveness and ease of use?	3.7	3.97
Q4. How much helpful is the functionality for inserting an artwork (image) in the exhibition?	3.81	3.93
Q5. How would you rate the interface providing access to the exhibition(s) assigned to you in terms of effectiveness?	3.9	4.01
Q6. How would you rate the sizing functionality regarding the artworks?	3.7	4.14
Q7. How would you rate the lighting functionality?	3.84	4.22
Q8. How much helpful are the provisions of virtual rooms to create art exhibitions in terms of spatial context?	3.67	3.79	point 1 stands for: Not helpful at all; point 5: Very helpful
Q9. How much helpful are the provisions of curatorial function (e.g., arrangement of artworks in groups, description of groups and addition of related narrative)?	3.9	3.96	point 1 stands for: Not helpful at all; point 5: Very helpful
Q10. How much satisfactory is the rendering (appearance) of the artworks in the virtual exhibition?	3.82	4.08	point 1 stands for: Not satisfactory at all; point 5: Very satisfactory

, below), address the functional aspect of the CREAMS VR editor. The following table (Table 1) presents, comparatively, the results from these questions, from the second formative and the summative evaluation phase.

Art students wish to exploit the creative and expressive capacities of VEs and deemed the use of the platform beneficial for their artistic development. Furthermore, in terms of incorporating chat function(s) and enhancing the social aspect of the application, while feedback from tutors is a very welcomed feature, there is a modicum of reticence in relation to receiving commentary from viewers, especially when this takes place in a publicly visible environment. We present below the results of questions 21 to 27 (formative evaluation) or (21 to 26 for the summative) immediately after the first set of questions. These two sets comprise especially adapted or created questions that correspond to the specificity of the CREAMS project. The results of questions 21 to 27 that relate to more specific issues pertinent to tertiary art education are shown in

Table 2. Especially adapted questions on VR platform, art student responses.

QUESTION	Second Formative Evaluation	Summative Evaluation	Explanation of 5 Likert Scale Steps
Q21: Did the creation of the exhibition encourage self-reflection on your creative process?	3.75	3.51	In Q21 point 1 stands for ‘Not at all, I Strongly disagree’; point 5 for ‘Very much, I Strongly agree’
Q22: Did framing your exhibition with text/other material help you situate your practice (i.e., identify connections between your artistic research and issues, themes and relevant artworks)?	3.81	3.70	In Q22 point 1 stands for ‘Not at all, I Strongly disagree’; point 5 for ‘Very much, I Strongly agree’
Q23: How helpful was the experience in acquiring skills in curating and presenting art?	3.87	3.87	In Q23, point 1 stands for: Not helpful at all; point 5: Very much helpful
Q24: How much important do you think is to include resources in a creative way that matches the artworks exhibited?	4.09	Question deducted	In Q24, point 1 stands for: Not important at all; point 5: Very important
Q25: What do you think about receiving feedback/comments from viewers, visible only to you through a chat function?	3.81	3.89	In Questions 25, 26, 27, point 1 stands for: Not a desirable function at all; point 5: Very desirable function
Q26: What do you think about receiving feedback/comments from viewers, visible only to you through a chat function?	4.09	4.13
Q27: What do you think about receiving feedback/comments from your tutors, through a chat function?	3.46	3.44

. Lastly, we present the results of two sets of questions that were especially created for this project, together (irrespective of question number order), and present afterwards those that relate to the pre-existing SUS and UEQ questionnaires, for better cohesion.

Furthermore, participants were asked to fill out the SUS questionnaire (Q11–20) between the above two sets of questions, which shed light on usability issues. SUS follows a complex calculation procedure to procure the score, which, in turn, is measured against a benchmark chart. As previously mentioned, this is a scale that addresses finished products; thus, the (under development) CREAMS VR editor is measured against a yardstick regarding commercial and finished products. Nevertheless, findings during the second formative evaluation (a total score of 63.1) indicated that usability still needed significant improvements, thereby emphasizing this aspect with tangible improvements as the findings of the final summative evaluation phase suggest (71.6). The results of the SUS questions are shown in

Table 3. System Usability Scale (SUS) questions, art students’ responses on VR platform.

QUESTIONS:	2nd Formative Evaluation Results	Summative Evaluation Results
Q11: I think that I would like to use this system frequently.	3.57	4.03
Q12: I found the system unnecessarily complex.	2.61	1.98
Q13: I thought the system was easy to use.	3.58	3.90
Q14: I think that I would need the support of a technical person to be able to use this system.	2.60	2.23
Q15: I found the various functions in this system were well integrated (e.g., tutorial, virtual guide, artworks functionalities).	3.69	3.99
Q16: I thought there was inconsistency in this system.	2.51	2.02
Q17: I would imagine that most people would learn to use this system quickly.	3.82	3.87
Q18: I found the system difficult to use.	2.39	2.03
Q19: I felt confident using the application.	3.43	3.66
Q20: I needed to learn a lot of things before I could get going with this system.	2.73	2.43

.

3.4. UEQ Findings and Analysis

The benchmarks of UEQ (as is the case with SUS as well) are created with finished (and commercial) products in mind, which should be factored in during the formative evaluation phase of a research project. The dependability scale proved to be an area that requires more attention in the final development phase. However, novelty had the lowest numerical score. This is an indication that contemporary art applications are expected to be cutting-edge in relation to the aesthetic characteristics of the platform itself.

The User Experience Questionnaire has been a significant tool that helped streamline the analysis and the procurement of meaningful conclusions (Figure 6 and Figure 7). The automated UEQ benchmark is created based on a large body of research conducted with this tool.

Figure 6. Art students’ mean scores per UEQ scale (category), including confidence intervals during second formative evaluation phase. Result scales from −1 to 2.5 correspond to the seven Likert scale steps.

Figure 6. Art students’ mean scores per UEQ scale (category), including confidence intervals during second formative evaluation phase. Result scales from −1 to 2.5 correspond to the seven Likert scale steps.

Figure 7. Art students’ mean scores per UEQ scale (category), including confidence intervals during final summative evaluation.

Figure 7. Art students’ mean scores per UEQ scale (category), including confidence intervals during final summative evaluation.

Here, perspicuity, which relates to the degree to which an app is understandable and easy to grasp, suffers considerably (in both evaluation phases) and becomes a focal point for (further) improvement. At the same time, an equally frail area is dependability: users need to feel more confident using the system. The results (−3 to 3) correspond to the seven Likert scale steps. In

Table 4. Numeric values of UEQ art student responses during second formative evaluation.

Attractiveness	Pragmatic Quality	Hedonic Quality
1.32	0.99	1.24

, we present the mean value results from the second formative evaluation phase.

In

Table 5. Numeric values of UEQ art student responses during summative evaluation.

Attractiveness	Pragmatic Quality	Hedonic Quality
1.79	1.29	1.79

, we present the respective mean value results from the summative evaluation phase. A comparison suggests that significant progress was made in terms of user experience.

Table 6. Detailed values of UEQ art student responses on VR platform, per scale (summative evaluation).

UEQ Scale	Mean	Variable	Comparison to Benchmark
Attractiveness	1.791	0.70	Good
Perspicuity	1.133	1.31	Below Average
Efficiency	1.578	0.87	Good
Dependability	1.173	1.24	Above Average
Stimulation	1.808	0.74	Excellent

provides details on the exact values obtained in the UEQ scales used.

The following chart (Figure 8) presents a combinatory overview of the second formative and the summative evaluation results for easier comparison. It shows in dark color the existing score from the second formative evaluation phase, and in lighter shade of the same color the additional progress made, as documented by the summative evaluation (except for the ‘complicated/easy’ metric, which remained static). One key area for further improvement is the (un)predictability factor, as users expressed in questionnaires and during focus groups a sense of being unable always to predict how the system will respond or what they should do next.

The following chart (Figure 9) provides a more detailed overview of the UEQ results from the summative evaluation phase, with numeric values and an additional column (titled ‘Scale’) showing which questions were used to determine the results per scale (UEQ uses the term scale to convey the categories that, in turn, comprise the pragmatic and hedonic aspects of the evaluated application).

4. Discussion

This section provides a discussion that focuses on the findings from art students’ responses in relation to the VR platform. Firstly, usability and practical issues are outlined as follows.

Art students commented on the need to improve the adjustments procedure, during which a user must select from a drop-down menu (see Figure 10) which element they want to adjust. Still, oftentimes, it is fairly confusing to locate the right item through this route as one may be unsure of the name given to each item, and in any case, it is a cumbersome procedure that is not very intuitive. The ability to just (right) click on an element and access a user-friendly adjustment menu is highly desirable.

Some indicative written responses in relation to what improvements they wish to see or what comments they want to make are as follows:

• Simplification in moving between the options of settings [this is a characteristic comment on user interface]
• The artworks may look good on the platform, but the background should get more real space vibes [this is a characteristic comment on spatial realism]. On the flip side, another participant comments: ‘It looks kind of artificial, but it doesn’t bother me that much’. This is an indication that realism in the rendering of the space should not necessarily strive to achieve commercial-type levels of life-like appearance, as the main idea is to provide functional and flexible spatial manipulation capabilities even within a minimal aesthetic, as the artworks in the last analysis should be the protagonists rather than aim for a cinematic experience, as pointed out in a group discussion with experts. During focus group discussions, participants insisted on the importance of exploiting the XR medium’s capacity to support creative spatial configurations and make the most out of virtual spaces in terms of expressive capabilities and aesthetic potential. For example, focus group participants, as well as written responses in open questions, suggested the need for supporting creative uses of the space or navigational functions, e.g., by introducing transparent walls or even offering the possibility to create virtual exhibitions that defy the rules of gravity by allowing the viewers to hover between exhibits.
• Another responder comments: ‘It is still restricted. I would like to have more choices’. Then, he or she suggests ‘Expanding it yet keeping it easy’. This brings us to a key point: often, art students expressed their need for simplicity and intuitiveness through a clear-cut and straightforward UI (user interface).

In a nutshell, the insights gained in relation to users’ feedback suggest that enhanced realism, an intuitive navigation interface, user-friendly interactivity provisions, and dependable support and guidance should be given priority. Furthermore, according to the feedback, the VR application necessitates simplicity, intuitiveness, and more support with well-placed links to tutorials, guidelines, and help messages. These were areas that transpired during the first evaluation phase as well. Even though much has been achieved in rendering the VR editor more user-friendly, this appears to be a key aspect for users. Concrete steps and measures have been taken, in the light of the first evaluation phase, ranging from improving the semantics of tabs, e.g., what was initially called ‘Sketchbook’ (function) see Figure 11, changed into ‘Associate Material to Artwork’ (see Figure 12), to the mechanism of inserting artworks or managing the exhibitions’ spaces with the introduction of a modular floor plan functionality for designing the gallery.

Moreover, in terms of limitations and areas for further improvement in relation to usability and user interface, as the abovementioned chart (see Figure 9) clearly illustrates, the learnability (perspicuity) of the system is a relatively low-scoring aspect of the app, as users still deem the VR platform to be somewhat complicated. This finding, in conjunction with feedback from focus group discussions, highlights the importance of clear, concise, and readily available manuals, as well as other provisions (e.g., informative text or Frequently Asked Questions/FAQs provisions in the form of drop-down menus or of interactive widgets) for the support of the users. A recurring idea in conversations within focus groups is that there should be some links within the applications rather than externally positioned manuals in the knowledge repository section of the CREAMS site, rendering access to useful guides easy and straightforward. Unpredictability is also linked, as we gathered from conversations by the lingering existence of some bugs or glitches that, despite the improvements made, still occur unexpectedly due to the limitations posed by the fact that the app is based on open-source code and, therefore, has some inherent limitations in this respect.

Furthermore, beyond the usability-related and aesthetics-related issues (e.g., in relation to realistic rendering of the environment, or lack thereof), responders commented on more specific aspects of the platform’s capacity to support learning in the fields of curatorial frameworks as well as fostering self-reflection upon and their ability to contextualize their art practice. Virtual gallery spaces offer immense opportunities for interaction with additional resources in imaginative ways (e.g., by hovering clusters of artwork-related material around the viewer) as suggested during focus groups’ discussions online and in person by participating art students.

Moreover, the ability to frame exhibited work with texts, multimodal resources, and other informative or aesthetically charged material matching the gist of the exhibition itself has been seen as highly desirable throughout focus group discussions and pertinent questionnaire answers. The positive disposition towards XR’s capacity to support students’ ability to contextualize their artworks or, for the same matter, curate their virtual exhibitions is reflected in the specially adapted questions that are presented in Table 2. More specifically, questions 21 to 23 show a positive approach in the abovementioned aspects, as the numeric values are above 3.5 and gravitate towards 4 out of 5. This is a key and distinguishing feature of virtual exhibition, which empowers artists to add contextual resources in ways that are not possible in conventional gallery spaces by exploiting the potential of XR for interactive, personalized, and active engagement with multimodal material.

While we included different sets of questions, some verbatim, adopting existing questionnaires (e.g., SUS) and others adapted from other publications to match the specific characteristics of the CREAMS project, we chose not to generate a comprehensive scheme, mapping the interrelations between the different elements such as the one proposed by Hammady et al. [25] or, for the same matter, comparative charts of metrics and other evaluation aspects in the form of a systematic review as in Kabassi et al. [6]. This evaluation interrelated quantitative data (from sets of questions) and qualitative findings (from written comments or focus group discussions) with the aim of highlighting areas in need of improvement to meet art students’ expectations, as expressed during the needs analysis phase of the CREAMS project. Beyond the pragmatic role of this evaluation process that informed the development of this XR platform, in hindsight, future research should aim to generate a more comprehensive mapping of how different elements, aspects and methods of XR exhibition-creating applications can be integrated into a schematic and holistic approach. This may be a limitation of this study, that nevertheless inspires future research that will not inform pertinent methodologies, solely by offering an instance of practice, but by suggesting a thorough framework with concrete schemes, addressing both practical and theoretical facets of XR evaluation.

5. Concluding Remarks

Last but not least, we include a short discussion on the specificity and the distinct characteristics that differentiate learning and teaching processes in the visual arts domain. This, in turn, is reflected in ways to assess learnability in Fine Art Schools.

Art exhibitions are not lessons, neither for those who create them nor for those who visit them. At the same time, they offer opportunities for learning, which means creating meaning through the mediation of sense and emotion, but in such a way that the outcomes of learning are not predetermined or easily assessed. This is deeply rooted in the nature of artistic practice and its function as a fulcrum for thinking and feeling differently. In relation to learning in the field of visual arts, as Addison notes [39], assessable and non-assessable learning operate dialectically, and if one or the other ceases to exist, the benefits of both would be negated. While such extracurricular research-driven initiatives and settings offer opportunities for enhanced learning and reciprocity in student–faculty relationships, they are not readily available to all.

CREAMS as a research project and more specifically its evaluation methodology undertook to provide and optimize tools such as virtual exhibitions that encourage and at the same time require negotiation, reflexivity, and relational encounters between learners and teachers. This is reflected in this paper, which aspires to offer inspiration to stakeholders who work in the interstices of XR technologies, culture, and art education.

Furthermore, the need to exploit the possibilities offered by new media to support a dynamic and in-depth provision of associated contextual material is prevalent. As Vergo states [40]:

The notion that works of art in particular should be left to speak for themselves takes no account of the fact that such works are, for most visitors, remarkably taciturn objects. Left to speak for themselves, they often say very little; and sometimes quite considerable effort is required on the part of the historian, art historian, critic or the viewer to coax them into eloquence.

This quotation underlines the value of framing visual works with a nexus of informational or even polysemic material, which befits contemporary artworks that are typically open to multiple interpretations. After all, the virtual museum “is not a mere reflection of the real; it has developed a presence/life of its own” [41], according to [42]. This signals the importance of creating virtual exhibitions that exploit the potential of digital tools rather than being digital replicas of brick-and-mortar gallery spaces. Learning through the arts and using digital tools to create exhibitions as tools for enhancing creative, critical, and practical skills is a highly personal, heuristic, and reflexive process [43,44,45] that may be elusive, multifaceted, and hard to capture in all its dimensions [39]. This has been a central challenge for this evaluation process outlined here and a source of inspiration for developing a concrete evaluation philosophy that matches the intricacies of assessing the efficacy of XR tools for learning and acquiring diverse skills in this context.