TinajAR: An Edutainment Augmented Reality Mirror for the Dissemination and Reinterpretation of Cultural Heritage

Abstract

The use of augmented reality (AR) in cultural heritage (CH) applications opens a whole set of possibilities, including the virtual transformation of CH elements. This paper presents TinajAR, a mirror-based AR application designed to serve both as an edutainment application in the field of CH and also as an artistic expression. As an edutainment application, TinajAR features a multi-marker video-based AR application designed to show virtual ceramic pieces and explain the pottery process through virtual avatars. As an artistic expression, TinajAR seeks to reinterpret an ancient type of cellar called calado, which was used in the past for storing wine in northern Spain. The reinterpretation consists in giving a different but meaningful use to the space. TinajAR was used by around 1800 people during a ceramics exhibition in La Rioja, Spain and was assessed at the satisfaction level with 56 users by means of a system usability scale, giving very satisfactory results.

1. Introduction

Virtual and augmented reality have been used in cultural heritage (CH) applications for a long time. On the one hand, virtual reality (VR) is a technology that allows performing virtual reproductions of CH elements. For this reason, it is suitable for simulating or recreating elements or buildings, which are either distant from the current physical space in which the user is located or that are inaccessible or have been destroyed. On the other hand, augmented reality (AR) can enhance/enrich an existing CH element or site with additional information that is not currently present in the real world. This could be interesting, for instance, to appreciate the original aspect of a building or painting, on top of the current state. Therefore, whereas VR allows performing virtual travels, AR requires that users physically visit the site/element in order to trigger the virtual CH content. This physical relationship between the user, the AR application and the CH element where the application is used, creates a strong spatial connection between these elements that make users more aware about where they are. This spatial connection is especially important to perform a meaningful interpretation of CH elements.

CH most followed principle is “through interpretation, understanding; through understanding, appreciation; through appreciation, protection” [1]. In this regard, AR is a very useful tool to achieve CH interpretation and it is also very appealing for children and young people. The attitude of these future generations towards CH will be key for the preservation of our cultural legacy. For this reason, edutainment applications, in which education or dissemination of knowledge is performed in a fun way, are effective tools to engage young audiences. This is one of the reasons why serious games (SG) and AR applications, like the one presented in this paper, are two of the most important trends in CH applications [2].

Furthermore, as AR requires interaction between the user and the application, this feature has been utilized to create interactive art, a form of art in which the spectator is part of the artistic element, either because they trigger part of the artwork or because they provide meaning to the work. Although art and science are disciplines that are constantly looking in different directions with moments of proximity and moments of distance [3], the intersection of art and science through interdisciplinary contributions could lead to unexpected artistic contributions.

Following these previous considerations, the aim of this paper is to present TinajAR, an AR system designed to serve both as an edutainment application in the field of CH and also as an artistic expression for the reinterpretation of an ancient type of building. On the one hand, as an edutainment application, TinajAR features a multi-marker video-based AR application designed to show (virtual) ceramic pieces. These masterpieces can be inspected as many times as desired without risk of damaging them. TinajAR is an AR system based on the augmented mirror metaphor [4], by which the image of the user is augmented with virtual content. The idea behind TinajAR is that users could choose one or several AR markers that will trigger virtual content in the form of virtual pottery jars. This way, users will see themselves carrying a pottery piece that is not present in the actual world.

On the other hand, as an artistic expression, TinajAR seeks to reinterpret an ancient type of cellar called calado, which was used in the past for storing wine in northern Spain. The reinterpretation consists in giving a different but meaningful and coherent use to the space, transforming the way people interpret this ancient space, which visitors unavoidably associate with wine. In this regard, the TinajAR concept involves turning the wine barrels into elements that participate in the creation of virtual pottery jars. In fact, the name of the system comes from the combination of the word tinaja (meaning pottery jar in Spanish) and the acronym AR.

TinajAR was designed, developed and installed as part of a ceramic exhibition in the region of La Rioja, Spain, with great success, as around 1800 people visited the exhibition. In order to give an insight of the amount of satisfaction, a preliminary test was performed, reporting a high degree of satisfaction.

The paper is structured as follows. Section 2 reviews the related work on the topic. Section 3 describes both the concept of TinajAR and the system implemented to achieve this concept. Section 4 describes a user satisfaction test that was conducted to analyze the response of the public. Finally, Section 5 draws some conclusions and outlines possible future work.

2. Related Work

Information and communication technologies (ICT) have reshaped our lives and forever transformed the way we communicate and perceive reality [5]. Cultural heritage is no exception since ICT allows us to look into the past through a different enhanced perspective while providing multiple benefits—such as digital documentation, improved dissemination, interpretation, and an increased availability—leading to a higher degree of understanding and appreciation of CH items. For this reason, the use of augmented reality applications at CH sites is now quite common. Since the first AR application was designed by Caudell and Mizell [6] and the properties and features of these systems were defined by Azuma [7], many AR systems have been proposed in the CH sector [8].

Most AR-based CH applications follow the mobile AR paradigm, where users transform their mobile phones or tablets into doors to an enhanced environment, such as in [9,10,11]. More complex setups can be proposed with holographic devices or see-through head mounted displays (HMD), such as in [12,13] or in [14], where an HMD and a tracking system were used to create an outdoor AR archaeological guide in ancient Olympia, Greece. Spatial AR has also been used to project information over real CH items [15], a technology that can also blend the real and the virtual world seamlessly.

The augmented mirror metaphor [4] has also been used for CH, although very few cases have been reported in the academic literature. The work proposed in [16] shows an approach that can be considered similar to the one presented in this paper, since it is used for indoor exhibition purposes and provides a multiuser augmented reality mirror (ARM). In this system, a webcam, a projector and a white screen are used to create a mirror image of the user, and AR markers are employed to blend the exhibition environment with multimedia and virtual 3D objects that visitors can manipulate. A Kinect sensor is utilized to provide depth information and perform occlusion handling. At the exhibition, environment users were able to walk over a map of the Valencia region, containing several hot points that refer to different CH sites. User interaction with these hot spots allowed them to virtually visit these places. A very similar work, but using a map of Slovakia, is presented in [17].

A different mirror-based approach is the AR-Jazz system [18], where AR is utilized to ‘visualize’ sound and movements in live jazz performances. This kind of system can be considered also as an AR-based artistic expression, because music is reinterpreted (an artistic goal shared with TinajAR) and transduced into images, something that provokes emotional reactions in spectators. Another mirror-based approach for artistic exhibitions is Faces [19], where facial recognition is used to propose an identity change. With this system, users literally change their look for faces of other individuals, causing different emotional reactions.

The use of technology in artistic expressions is certainly not new, and it did not start with AR technology. The mix of real characters and cartoons in movies such as Who Framed Roger Rabbit? (1988) or Mary Poppins (1964) is an early example of this [3]. However, AR implies interaction, and therefore it can bring the user into the artistic element, opening endless possibilities to the artistic expression, such as activism, environmental awareness, aesthetics, narration, perception shifting, etc. [20]. Examples of interactive art using AR can be found in [21,22,23]. Although the interaction and gaming capabilities of AR are very appealing in order to create interactive art, AR can also be used for painting and sculpture in a more traditional way [24]. The extent of the use of AR as a mean for art creation is so profuse that a complete review lies outside the scope of this paper.

On the other hand, the use of AR or VR in the ceramic sector is also not uncommon, and some examples can be found such as AR Pottery [25], an interactive AR application in which users can live pottery design experiences. With this tool, users can create pottery models simulating the traditional way of pottery making. Another similar approach is the work presented in [26], where a Leap Motion device is used to track user’s hand gestures that can be used to create a virtual pottery piece. This project, called Pottery Go, is mainly used to assist beginners in learning the gestures needed to use a pottery wheel. Most works using ICT in the ceramics field use this “virtual pottery” paradigm [27,28,29,30,31], where the process of making pottery pieces is the focus of the system, and not the description of existing pieces and the importance of this cultural legacy. A work in this latter line is presented in [32], where an AR application is designed for the exhibition of pottery finds in archaeological museums. The TinajAR application presented here focuses in this idea of CH dissemination while serving also as an artistic expression; it uses an augmented mirror paradigm for edutainment and reinterpretation purposes, something that is very unusual in CH. Although the use of these mirror-like paradigms has not been widely explored, the contribution of TinajAR is mainly focused on the concept more than in the technology.

3. The TinajAR System

3.1. The TinajAR Concept

TinajAR is conceived both as an edutainment application to disseminate CH knowledge and as an artistic expression through technological means. As an edutainment application, it serves to show the different ceramic pieces of the region where it is installed, as part of the Spanish National Pottery and Ceramics Fair (NACE). Since augmented reality is a technology that is very appealing for children, dissemination of this ancient cultural tradition, which is endangered by the lack of young people in the field, can reach wider audiences. For this reason, in addition to the interactive AR application, TinajAR includes a narrator who tells the story about how pottery jars were created, highlighting the different parts of the process: pottery wheel, baking, and painting.

As an artistic expression, the aim of TinajAR is the reinterpretation of a special type of building. In this regard, TinajAR is designed and meant to be installed in ancient Spanish cellars called calados. A calado is a special kind of underground cellar designed as a place for the storage of wine barrels at constant temperature. These places were so important in the Spanish region of La Rioja that from the XVI century, municipal regulations limited the passage of horse-drawn carriages and wagons through those streets in which these cellars were present. The rationale of this decision was to preserve the final quality of the wine, since it was thought that the constant rattle disturbed the peace and tranquillity wine needed. As it can be seen, these buildings were very important some centuries ago. However, their use was discontinued when modern cellars were built by the wine industry. Some of them have been recently restored in an effort to preserve them as remarkable CH elements and use them to illustrate our cultural legacy.

TinajAR was installed and tested in two of these buildings: the Calado del Conde, in the town of Navarrete, La Rioja, Spain, and the Calado de San Gregorio (see Figure 1), in the city of Logroño, capital of the same Spanish region, world-famous for its high-quality wine. The former is a XVII-century building, whereas the latter is a 30-meter long XVI-century space featuring a tunnel vault shape.

Figure 1. Calado de San Gregorio, Logroño, La Rioja, Spain, before the installation of the TinajAR system.

One of the goals of TinajAR is to reinterpret these places (calados) and transform them into virtual ceramic exhibitions, making use of existing elements of the space. The relationship between pottery jars, which were also used for drinking and storing wine, and the wine cultural heritage, makes this reinterpretation easier. TinajAR was installed within the context of a pottery fair (NACE), in which many traditional fair stands were deployed. Therefore, the introduction of a virtual ceramics exhibition within an old-fashioned traditional pottery fair creates an additional clash of cultures that is also in the aim of this artistic expression.

To fulfil this space-reinterpretation goal, wine barrels are used in TinajAR for several purposes, different from the original use. First, they are used as picking place for the AR markers of the system (see Figure 2). Instead of picking wine glasses, users have to pick AR markers, which are, in turn, virtual ceramic pieces when users look at the augmented mirror. This artistic purpose is interactive because spectators are who fulfil the artist’s intention, transforming the site from a wine cellar to a pottery cellar. Second, one of the barrels is used to host and conceal the computer that runs the application. Finally, the camera was also conceived to be mounted on one of the barrels (see Figure 2). However, this setup did not provide a good mirror perspective and was reconsidered.

Figure 2. Sketch of the TinajAR concept for the Calado del Conde, in Navarrete, La Rioja, Spain.

In order to create a more appealing user-centered application, the mirror paradigm is utilized, in which users can see a reflected image of themselves, augmented with CH elements (virtual pottery jars). The use of AR instead of other immersive and interactive technologies, such as virtual reality, is motivated by two important restrictions of the system: (i) the system is meant to reinterpret an existing site, something that with VR would be much more difficult; (ii) the goal of TinajAR designers is to make users participate actively and make them feel they are the centre of the system, while integrating the application in an existing space.

3.2. Description of the TinajAR System

In order to fulfil the aforementioned concept, the TinajAR application features a video-based ARM system, in which users can see themselves in a mirror-like perspective. The mirror image is augmented by means of AR markers that are used as trays on top of which 3D pottery jars are visualized when they are recognised by the software system. Other options could have been possible, such as mobile AR, but the mirror perspective is more engaging since users feel that the system communicates with them when the marker is presented or changed, providing the sensation that they (and the ceramic pieces) are the only protagonists of the system. The use of a video-based ARM is also motivated by the simplicity of the set-up and the orientation to shared/group experiences of this paradigm, unlike other solutions that are less group-oriented, such as see-through HMDs.

The main components of the system are a laptop computer, which hosts the AR application and where the virtual models are stored, a white projection screen, a projector, and a small camera, necessary to capture the mirror image that needs to be augmented. Figure 3 shows the setup of the system at the laboratory level.

Figure 3. Laboratory setup of the TinajAR system, at the IRTIC laboratory of the University of Valencia, Spain.

As the system was intended to blend with the environment in ancient cellars, a seamless integration with these architectonical jewels is essential. For this reason, in the final setup an ultra-short-throw projector with frontal projection was employed, since rear projection did not provide enough brightness. The projector was concealed behind a banner and a 2 × 1.5 m white screen was utilized. As previously explained, a wine barrel was used to conceal the laptop. Because of the necessary low light conditions of these sites, big 30 × 30 cm solid (printed on cardboard to avoid bending) AR markers were utilized, so that they be as visible as possible for the AR recognition system.

Figure 1 shows one of the original spaces of the exhibition before the installation of TinajAR took place, whereas Figure 4 shows the same space with the final TinajAR setup. As it can be seen, there is a seamless integration with the environment, since one of the goals of TinajAR is to reinterpret the meaning of the site by means of virtual elements without fundamentally changing the existing physical space.

Figure 4. Final setup of the TinajAR system at the Calado de San Gregorio, Logroño, La Rioja, Spain.

The software running TinajAR is not complex, yet it uses state-of-the-art technology. It consists of a Unity3D-based application written in C# that is programmed to recognise a set of 15 different AR markers. Each AR marker is transformed, when recognized, in a different ceramic piece over a virtual tray. AR-Toolkit is utilized to recognize the markers and place the virtual models. For this reason, multi-marker situations are properly handled, and the system is able to show several pieces at the same time.

One of the most challenging parts of this system was the reconstruction of the ceramic pieces in a realistic way, since they are the most important part of the system. For this process, 3D models of the selected pottery jars were created with the ‘Lathe’ tool of 3D Studio Max, which allows creating virtual models in a way that is similar to the operation of a pottery wheel. Pottery pieces were modelled from high-resolution pictures and measures of the original pieces provided by NACE (see Figure 5), and later inserted and visualized in Unity3D with Phong shading (Figure 6).

Figure 5. A snapshot of the process of creating a 3D model of a pottery jar with 3D Studio Max.

Figure 6. 3D model of a pottery jar integrated and visualized in Unity3D.

In addition, virtual avatars (see Figure 7) were utilized in selected parts of the exhibition (they are also triggered by a particular AR marker), showing a potter creating a pottery jar or an artisan baking the jar in order to create the solid piece. These 3D avatars were utilized by the narrator of the exhibition to provide integrated explanations about the process of creating ceramic pieces. Therefore, the dissemination of this CH knowledge is performed in an appealing way, integrating interactive elements and a proper storytelling, which is an essential part of any CH application.

Figure 7. An animated avatar showing how to use a pottery wheel (laboratory setup).

4. User Tests

TinajAR was installed and tested during a ceramic art exhibition in La Rioja. As previously explained, two different but similar places were used for the installation and testing of the system. The first one in Navarrete (see Figure 8) within the context of the NACE pottery fair and the second one in Logroño (see Figure 9 and Figure 10).

Figure 8. The TinajAR system during the exhibition in Navarrete, Spain.

Figure 9. Artist Bibiana Martínez shows TinajAR to the mayor of Logroño, Spain.

Figure 10. Children enjoying TinajAR in Logroño, Spain.

As one of the goals of TinajAR is to serve as an edutainment application, a satisfaction user test was conducted during the exhibition. Fifty-six (all of them from the cellar of Logroño) of the approximately 1800 visitors that enjoyed the exhibition filled a 10-question survey with answers in a 1–5 Likert scale. Users were explained that a value of 1 meant strongly agree and a value of 5 meant strongly disagree. The questionnaire follows the system usability scale (SUS) developed by John Brooke [33], which has become an industry standard. According to [34], usability tests are one of the five evaluation types applied to AR systems.

The questionnaire included also a brief profile section with questions about age, gender, academic degree (formal education), hours of daily use of multimedia devices and type of use of these devices. The ages of the users ranged from 8 to 72 years, with an average of 42 years; 66% of them were men, and 34% were women. The 10 questions of the SUS survey are shown in Table 1.

Table 1. The 10 questions of the SUS questionnaire.

Number	Question
1	I would like to use TinajAR frequently.
2	I found TinajAR too complex.
3	The system is easy to use.
4	I need the support of a technician to be able to use this application.
5	I think that the different functions of TinajAR are properly integrated.
6	I think there are too many inconsistencies in this application.
7	I imagine that most people are able to learn to use TinajAR rather quickly.
8	I believe this project was too cumbersome to use.
9	I felt very confident using the system.
10	I needed to know many things before using TinajAR.

Results show that users were quite satisfied with the application. Averages for positive questions (questions 1, 3, 5, 7, 9) are 1.609, 1.043, 1.088, 1.087, and 1.130, whereas averages for negative questions (questions 2, 4, 6, 8, 10) are 4.696, 4.174, 4.826, 4.783, and 4.652. Table 2 and Figure 11 show the results for the 10 questions. In Figure 11, average values are represented by vertical bars, and the standard deviation is plotted as white vertical lines. The value for the standard deviation is small (lower than unity) for questions 3, 5, 6, 7, 8, and 9, and is only high in question 4 (“I need the support of a technician to be able to use this application.”). This question received an average value of 4.174, with a standard deviation of 1.497, being the question with the lowest satisfaction. Nevertheless, we can conclude that users have positive views about TinajAR, as the values for positive and negative questions are clearly clustered around 1 and 5, respectively. No significant differences were found for the two different genders.

Table 2. Results of the SUS questionnaire

Question	1	2	3	4	5	6	7	8	9	10
Average	1.609	4.696	1.043	4.174	1.088	4.826	1.087	4.783	1.130	4.652
Std. Dev.	1.158	1.020	0.209	1.497	0.288	0.491	0.287	0.850	0.626	1.027

Figure 11. Results of the SUS questionnaire.

Despite the results of the questionnaire are satisfactory, it is important to highlight that this is a preliminary evaluation, and the aim of this paper is primarily to show the proof of concept of TinajAR and how this AR application could serve for both CH dissemination and artistic expression. Moreover, this second element (the artistic part) was not evaluated, since the artistic expression of TinajAR reflects the personal perspective of its creator and it is meant to express an intimate artistic message. Although users were not asked about the artistic expression, they were part of the proposed art system as users are those who provide a meaningful reinterpretation of the space, through its use.

5. Conclusions and Future Work

In this paper, we have presented TinajAR, an augmented reality application for the dissemination of pottery art and the reinterpretation of ancient cellars. The fusion of tradition, culture, history, and legacy with future, technology, innovation and interaction provides an appealing system that serves both as an artistic expression and as a tool for the dissemination of CH knowledge. In addition, the use of the augmented reality mirror paradigm alongside pottery art provides a significant level of novelty. Although this specific aspect has not been individually assessed and compared with other possible solutions, we believe that the ARM technology suits the purposes sought with the project, since the third person view and the mirror images offered by this paradigm provide a broad perspective and make people be more aware of their actions, making them the true centre of the application.

As an interactive artistic expression, the TinajAR concept involves giving a different use for the elements of a cellar, mainly the wine barrels. As a CH application, users can appreciate the uniqueness of the pottery process (and its result) both by inspecting different virtual pottery jars and by means of the narration supported by the use of virtual avatars showing different aspects of the process.

The system has been successfully assessed by a sizeable group of people, giving satisfactory results, which means that users have positive views about this way of portraying and explaining CH elements. Besides the positive evaluations, personal feedback from the users was very satisfactory and several users visited the exhibition more than once. It is important, however, to emphasize that the main contribution of the paper is not the evaluation of the designed AR application, a task that can be performed in different ways depending on the specific aspect the assessment intends to highlight. The aim of this paper is to present the novel TinajAR concept, a system designed to serve both as an edutainment application in the field of CH and also as an artistic expression.

Future work includes the extension of TinajAR to other cultural elements. Another interesting suggestion is the transformation of TinajAR into a serious game, so that the system can be more engaging and fun. It would be also important to perform a more detailed evaluation analyzing the impact of the ARM paradigm, the significance of the proposed setup with respect to other possible technologies, such as an HMD AR device (e.g., HoloLens), the engagement with respect to awareness about CH, the transferred CH knowledge, etc. Finally, different immersive approaches—such as spatial AR, holograms, or even immersive VR applications could be used—possibly with different objectives and features.