Identification of Usability Issues of Interactive Technologies in Cultural Heritage through Heuristic Evaluations and Usability Surveys

Abstract

Usability is a principal aspect of the system development process to improve and augment system facilities and meet users’ needs and necessities in all domains. It is no exception for cultural heritage. Usability problems of the interactive technology practice in cultural heritage museums should be recognized thoroughly from the viewpoints of experts and users. This paper reports on a two-phase empirical study to identify the usability problems in audio guides and websites of cultural heritage museums in Vietnam, as a developing country, and Australia, as a developed country. In phase one, five-user experience experts identified usability problems using the set of usability heuristics, and proposed suggestions to mitigate these issues. Ten usability heuristics identified a total of 176 problems for audio guides and websites. In phase two, we conducted field usability surveys to collect the real users’ opinions to detect the usability issues and examine the negative-ranked usability. The outstanding issues for audio guides and websites were pointed out. Identification of relevant usability issues and users’ and experts’ suggestions for these technologies should be given immediate attention to helping organizations and interactive service providers improve technologies’ adoptions. The paper’s findings are reliable inputs for our future study about the preeminent UX framework for interactive technology in the CH domain.

1. Introduction

Visitor experience in cultural heritage (CH) sites has undergone positive changes, including enhanced audience engagement and improved learning experiences influenced by the evolution of interactive technologies in the domain [1,2]. The affordance of interactive technology is recognized by cultural organizations to encourage creative interaction with spaces and artefacts, provide inventive ways of engagement, and extract new kinds of meaning from previously inaccessible archives [3]. It is no doubt that the recent increase in interactive applications in the cultural spaces allows the visitors to experience the artefacts, visits, and exhibitions in more effective and attractive ways. CH interactive applications consist of smartphone apps and games, location-aware audio guides, VR/AR (virtual/augmented reality) technologies, online virtual worlds, multi-touch screens, and different types of interfaces that transcend CH content to users [4].

CH spaces have a long history of adapting mobile technologies, including audio guides. Together with using the app physically at museums, websites are also an important means to offer museum services to visitors/users. However, like other areas, the objectives of CH spaces to enhance the visitors’ experiences cannot be achieved if the critical usability issues that pertain to both interactive applications and websites are not adequately resolved. To meet the high user expectations, designers have applied user center design methodology by bringing users into the design process on creating products or services to interpret and effectively satisfy users’ needs. It is no exception in the field of CH. Usability and user experience (UX) have always been the predominant concern of software products. Usability and UX are indicators for successful interaction designs.

From the existing work, the interactive technologies’ situation in CH spaces in developing countries have been partly shown. VR/AR has been deployed to preserve and enhance the CH in some developing countries such as Indonesia (Banten E-heritage) [5], Africa (The tales of the Tokoloshe) [6]. However, there are remarkable gaps in CH budget distribution and management in developed and developing countries. For example, the limited financial allocation at the Indian national museums makes them uninspiring, hardly drawing any visitors [7]. Donatella and Bertacchini [8] showed the vicious circle in the developing nations, in which the low level of development obstructs culture promotion. In contrast, the unsuccessful exploitation of domestic heritage impedes the chances of culture-based development strategies. To analyze the challenges and opportunities of CH management in developing countries, Altschul showed that frameworks maintaining a balance between cultural heritage and economic development in those nations do not exist or are ineffectual [9]. In general, the situations of how the interactive technologies are applied in national CH museums in the developing country and their usability issues have not been mentioned officially in existing work. However, via those papers, we can somehow understand the financial circumstance in deploying the technology in CH spaces in developing countries.

Vietnam’s culture is a colorful picture of “unity in diversity”, the essence and convergence of unique cultural values of 54 ethnic groups. Therefore, like other CH of other countries, they should be preserved and promoted because of their high values and uniqueness. Vietnamese museums nowadays are keen on presenting their objects in the most appealing and exciting way in order to magnetize visitors. Some national Vietnamese CH museums, including the VNMH, the Museum of Vietnamese Ethnic Culture, and the archaeological museum, have piloted interactive technologies such as touch displays, audio guides, and 3D content. Such interactive technology infrastructures capture visitors’ attention and motivate them to learn and engage in exhibits, resulting in a memorable and immersive learning experience [10]. However, this statement is based on the general observation; there had no statistic number from an interview or survey. Vietnamese CH institutions lack necessary interactions with visitors. Besides the appealing display of museum artifacts, ways to make museums are competitive with other forms of entertainment are Vietnamese museums’ concerns [11]. There is still minimal and intermittent interactive technology implementation in the Vietnamese CH domain [12]. Therefore, the Vietnamese government has introduced policies to enhance usability and visitors’ experience by adopting more interactive technologies in the museums displays and exhibitions [13].

In a developed country, Australian museums have a long-standing framework that other nations should benefit from the current system [14]. Lehman [15] highlighted a strong demand for immersive technologies, such as storytelling, audio guide, and VR expressed in the strategic or master plans of museums, such as New South Wales museums and galleries [16], and Australian museums [17]. Australian museums have successfully deployed modern technologies that other countries can learn, with no exception for Vietnam [14]. In the context of developing countries, an ICT evolution in CH organizations is critical. Additionally, to propose a preeminent UX framework for interactive museum technologies, especially during and after COVID-19 pandemic. The interactive technologies and their shortcomings should be understood and evaluated. Therefore, usability problems that the UX experts and actual users have experienced for developing and developed countries are essential and necessary to recognize.

The term “interactive technologies” used in this study refers to audio guides and museum websites/virtual museums. Although the audio guide is quite dated technology that has been utilized in CH venues, it is one of the primary devices conveying the message from the artefacts and exhibitions to the Vietnamese CH national museums’ visitors. Besides physical visits, online visits via websites are the common way to connect the community to museums. Therefore, audio guides and websites were chosen as interactive technologies to conduct the surveys.

Our research was carried out in two independent phases with the primary aim to examine the interactive technologies’ usability issues (phase 1) and what usability was ranked negatively (phase 2) from national museums in developing and developed countries. In phase 1, we conducted a HE with five UX experts examining the CH audio guides and websites/virtual museums to identify their usability problems and collect suggestions to improve these interactive technologies at CH locations. For phase 2, we conducted two usability surveys with 205 visitors visiting the museums physically. The purpose of the surveys is to examine the opposing ranks (including strongly disagree and disagree rank from the five-point Likert scale) of interactive apps and websites in Vietnam (as a developing country) and Australia (as a developed country).

This paper presents the results from the two phases and concludes with the usability issues of these technologies from the viewpoint of UX experts and real users. Additionally, the overview considerations for the interactive technology deployment in CH museums are presented. The reasons to conduct the two-phase study are the combination of the two techniques helping identify more usability problems of the systems [18]. Our method meets more attributes of positive cultural UX, including usefulness, ease of use, efficiency, accessibility, and identification [19,20], and increases the reliability of the data collected [21].

In phase 1, we used the checklist, mainly from Nielsen [22] with heuristic explanations [23] to conduct the HE. It involves the insight of five UX experts to assess the usability of museums’ audio guides and websites based on a set of predefined heuristics.

In phase 2, we developed the questionnaires based on the comprehensive questionnaires for multimedia guide experience from Othman [24], which were constructed on the variety of studies and professional organizations from the UK, and the criteria for evaluating the usability of the interactive device [25]. The synthetic questionnaires were used for another research on designing good UX for CH interactive technologies in developed and developing countries. These usability surveys are parts of that research.

Commonly, large national museums frequently have some supplies or funds for employing professionals to design their online museum effectively, while small museums face difficulties in developing and maintaining their official websites [26]. Vietnamese and Australian museums also have faced the same situations. The government’s investment in museums is uneven. It is widely noticed that the small local museums are not receiving the necessary financial support from the national government [26]. Therefore, the reasons these museums were chosen as the spaces to conduct the research are easy to interpret. Three national museums were chosen to examine their interactive technologies, including two national museums in Vietnam, namely the Vietnamese National Museum of History (VNMH) and the Hanoi Temple of Literature (TL), and one in Australia, the National Museum of Australia (NMA). Those three museums were selected as they share similar traits suitable for our study: government-owned museums; and provide an interactive application(s) and virtual exhibition(s) via the website.

2. Literature Review

The existing papers have shown how the usability and usability evaluations of CH products/apps are essential; and how these evaluations were conducted in the CH context.

2.1. Usability, User Experience and Heuristic Evaluation

Usability is part of the broader term UX and refers to the ease of access and or use of a product or website. Usability has been a principal aspect of the system development process to improve and augment system facilities and meet users’ needs and necessities. According to Nielsen, usability frequently means how well users can utilize the system’s functionality [27]. Unusable user interfaces are probably the single most prominent reasons for combining of interactive systems—computers and people, fail in actual use [28]. Usability is defined as the degree to which specified users can achieve specified goals in a particular environment, with effectiveness, efficiency, and satisfaction and in an acceptable way [29].

UX is related to the users’ emotions and perceptions leading to the use or the anticipated use of the product [30]. UX has also gone beyond traditional usability engineering, including users’ feelings, motivations and values, which are given more emphasis in addition to the efficiency, effectiveness and basic satisfaction, considered as the traditional meaning of usability [31].

In the relationship with usability, UX is tightly coupled with usability. According to Nigel Bevan [32], there is no elemental difference between usability and UX; however, there are some differences regarding task performance and pleasure in the development phase. UX has also gone beyond traditional usability engineering, including users’ feelings, motivations and values, which are given more emphasis in addition to efficiency, effectiveness and basic satisfaction. These aspects have been defined as the traditional meaning of usability [31]. Usability and UX, nowadays, have been the success ingredients of the software products, and indicators for successful interaction designs.

A usability assessment is defined as evaluating the user interface to identify usability issues in a design [33]. The usability evaluation’s objective is to identify as much as possible potential issue that the users could have with an application ([34] as cited in [35]). UX research, however, may include usability metrics from the viewpoint of visitors, such as effectiveness, efficiency and satisfaction.

The data collection to measure different usability metrics can ultimately contribute to better software development decision-makers, whether they are software developers, team managers, or usability developers [28].

Heuristic evaluation (HE) is critical for finding usability defects. Heuristic evaluations are prevalent, quick, and inexpensive techniques used for analyzing usability. HE is defined as an informal approach of usability analysis where several evaluators are presented with an interface design and inquired about commenting on it [21]. The evaluation has been performed in computer games, e-learning, mobile map applications, health social networking websites, gestural interaction, and heuristics with cultural aspects, to name but a few [36,37,38]. HE is one of the most actively used and researched usability techniques from numerous related studies. Among the related theories, interaction refers to an abstract model by which humans interact with the computing device for performing a given task. Nielsen [22] proposed a guideline for conducting HE. A minimum of three to five individuals are selected to lead the HE that can identify an average of 74% to 87% of the problems.

The methodologies of usability heuristics vary, including literature reviews, usability problems; mixing processes; guidelines, principles, or design recommendations, interviews, and theories [39]. The authors also illustrated that many researchers and practitioners have successfully used Nielsen’s heuristics; however, some researchers found it necessary to modify them in some ways. Likewise, Silva et al. [40] also showed that Nielsen’s heuristics are still the most preferred by the many previous researchers. Additionally, although being developed some new criteria, the new heuristics from TMD Heuristics [41] or SMASH heuristics were evaluated to be essentially like Nielsen’s [42].

In general, the heuristics of Nielsen could be applied in multiple contexts and audiences. In the next section, we will present an overview of the research about usability and its evaluations in the CH domain. Most studies about usability evaluations are HE.

2.2. Heuristic Evaluation in Cultural Heritage

Several researchers have proposed sets of heuristics based on Nielsen’s usability heuristics for websites in the CH environment. For example, Tehrani et al. [43] and Pallas and Economides [26] conducted research on virtual museum, and art museum websites. Additionally, heuristics for interactive applications in CH have also been addressed by many studies. For example, Gómez et al. [23] and Inostroza et al. [41] researched on usability heuristics of mobile devices, standard technologies in CH. The following two sub-sections provide closer views about how usability heuristics have been conducted in applications and websites of the CH domain.

2.2.1. Heuristic Evaluation for Interactive Applications

There are various techniques and methods which can be employed in the usability assessment of software, including mobile guide, AR, or VR. Gómez et al. [23] proposed a compilation of HE checklists taken from the existing bibliography but readapted to new mobile interfaces. Selecting and rearranging these heuristic guidelines provides a useful tool for both evaluation and a best-practices checklist. This experiment shows that the proposed checklist is very useful to avoid usability gaps even with non-trained developers. The resulting framework includes 13 sub-heuristics for mobile (adapted from 10 Nielsen usability heuristics and three new ones). The three new sub-heuristics are skills, pleasurable and respectful interaction, and privacy (See Figure 1). Inostroza et al. [41] proposed the 12-item usability heuristic set, which was also based on Nielsen’s heuristics and modified a new attribute, physical interaction and ergonomics. According to these researchers, traditional methods for usability measuring do not fit the nature of interactive devices. Both above studies used the same approach: usability heuristics developed using a mixing process involving the existing heuristics (Nielsen’s heuristics) in the mobile context.

Figure 1. 13 sub-heuristics for mobile devices adapted from Gómez [23].

Unlike ordinary software applications, in the case of AR, numerous devices can be employed, such as handheld, head-mounted, large screens with projecting devices and sound devices and interfaces that the user interacts with, using movements or gestures. The usability evaluation of AR interfaces becomes an essential procedure that should be part of their development process, as in the case of common software. Likewise, Murtza et al. [44] introduced new sets of heuristics that can be used to carry out usability inspections of VR systems via the HE method.

In conclusion, there are almost no research studies investigating usability evaluations about interactive technologies in national CH museums of both developing and developed countries. Ten usability factors of Nielsen can be modified and adjusted to suit the new generation of products. For audio guide’s usability evaluation in this study, ten usability heuristics of Nielsen [22] with the detailed explanations from Gómez’s et al. [23] should be considered as the best new sets of heuristics to form a checklist to measure the usability of interactive applications and websites in CH environment.

2.2.2. Heuristic Evaluation for Websites

The CH of a nation symbolizes an invaluable inheritance for both citizens to understand and explain the origin of customs and traditions and for the tourism sector to attract foreign visitors. Internet and technologies have enabled CH institutions to provide access to their collections in multiple ways, both on-site and online, and attract even more audiences. Fotakis and Economides [45] showed that museums have used the Internet to make their presence known, exhibit their artefacts virtually, and communicate widely with the community during recent years. Thus, many museums have focused on improving their virtual presences and services. To make CH websites more effective, researchers have measured the effectiveness of users’ interaction and websites’ attraction. In this context, user needs in the CH domain also vary. Usability evaluation assesses the ease of using a website’s functions and how well they enable users to perform their tasks efficiently [46]. Besides, CH has been a challenging domain of application in the information and communication technologies area. The use of modern technologies was recognized to be one of the principal methods to approach the mass audience and therefore to effectively promote and show the CH of nations. It is proved that technology-supported natural HCI is a critical factor in enabling access to CH assets. Advances in ICT provide the best conditions for visitors to access collections online and better experience CH on-site. Designing effective interfaces for CH websites is the destination that the organizations desire to achieve. Various institutions provide access to their collections both on-site and online, aiming to attract more comprehensive users than those that visit the physical sites. The most effective methods which can be deployed to make successful CH displays consist in attracting the visitors’ attention and improving their engagement. Lazarinis et al. [47] evaluated the technical capability of tourism and cultural websites to recognize the available options provided to users. Lazarinis et al.’s conclusions that besides the rich multimedia content, websites need to support users more efficiently by providing more services or customizing e-services.

To assess the Augmented Representation of Cultural Objects system of the virtual museum exhibition, Karoulis et al. [48] used the combination of empirical and expert-based methods. The questionnaire for museum curators and cognitive walkthrough for visitors were conducted. Karoulis et al. [48] stated that the combination of an expert-based method with the real user-based method, which is less resource consuming and more reliable is promising to valuate interfaces of a complex cultural heritage virtual interfaces.

From another point of view, the audiences’ websites are from across regional, languages, and boundaries, and they require the websites designed influentially by their local cultural perspectives. Meanwhile, cross-cultural usability is making a website an effective “bridge” between the global web owner and the local user. Consideration about the (H2) Nielsen’s heuristics, “match between system and the real world” and the intercultural usability heuristics, Díaz et al. [49] confirmed that the independent of the objective culture, the website should “speak” the language of its users with words, phrases, and concepts familiar to them. This viewpoint makes the possible actions easy to comprehend, rather than using concepts related to the system or jargons. Therefore, the measure of cross-cultural usability has become contentious in the area of HCI [39].

Harms and Schweibenz [50] conducted a study to evaluate the usability of Saarland’s museum website. Their first aim was to assess some evaluation methods, and the second was to improve the usability of the Saarland Museum website. They adopted a combination of HE and user testing as suggested in their research literature. The HE detected numerous usability problems in comparison with the laboratory test.

An expert review checkpoint, essential for enhancing the usability of the platform, was developed by Travis [51]. While developing functional systems requires much more than simply applying guidelines, by encouraging continuity and good practice, guidelines can also make a significant contribution to usability. The overview of the results is clearly displayed by expert review checkpoints that use the spider web according to the specified criteria.

For virtual museums, Tehrani et al. [43] stated that this kind of museum could present UX in visualizing the real museum. The usability problems for the Virtual Museum Negara prototype and calculate the experts’ results via the Content Validity Index in improving the interface design were identified via some factual statements of 10 usability heuristics of Nielsen.

Ten usability heuristics of Nielsen are the foundation for HE because they apply to multiple contexts and audiences. However, due to the innovative traits of new applications, new sets of usability heuristics, modifying Nielsen’s heuristics and or adding new heuristics to evaluate interactive applications and websites in the domain of CH are needed. Although the nature of these two environments of interactive applications and websites are different, to measure the usability of interactive technologies in general, the same synthesized checklist with the most common characteristics about them should be used to make the evaluation process more convenient.

3. Study Design

Using these mixed methods of HE and user surveys satisfies a pragmatic category with five attributes, including usefulness, reliability, ease of use, efficiency, and accessibility [19,20]. Nielsen [24] concluded that usability specialists evaluate much better than those without usability expertise at finding usability problems. Therefore, based on these reasons, collecting the results from both expert-based evaluation (phase 1) and user-based survey (phase 2) is a good and reliable approach to detect usability errors of CH interactive technologies.

3.1. Heuristic Evaluation in Phase 1

In phase 1, the HE was carried out to identify the usability issues about the designs of the interactive applications and websites. After careful analysis, some issues were raised by the evaluators regarding the usability aspects. The problems identified by the evaluators through the HE were listed together with their suggestions.

Nielsen proposed a four-point-rating scale to evaluate the severity of a problem: 1 = a ‘cosmetic problem’, the user will be mildly frustrated, and it would be nice to fix; 2 = a ‘minor problem’, users will be frustrated/have difficulty continuing to their goal, could be fixed; 3 = a ‘major problem’, users will be very frustrated/having difficulty continuing to their goal, should be fixe; and 4 = a ‘catastrophic problem’, users will not be able to continue to their goal, must be fixed [33]. The results will be presented in Section 4.1.

3.1.1. Method

Our heuristic study for audio guide and website evaluations applied a customized checklist to identify usability issues from the three national museums. We recognized that the last three heuristics from Gómez’s et al. [23] are not suitable for our evaluation, such as personal data (privacy); and data entry (pleasurable and respectful interaction), and input devices, or novice and expert users (skills). Moreover, the physical interaction and ergonomics from Inostroza [41], which shows the characteristic of the interactive app (such as physical buttons, audio guides’ size), need to be considered in the evaluation. Therefore, our checklist added this sub-heuristic, SH35-Physical interface/buttons, for only audio guides and placed it under (H8) Aesthetic and minimalist design. The physical interface and button designs were likely thought related to H8, H3 (User control/Freedom), or H7 (Flexibility and Efficiency of use). We agreed to place this sub-heuristic, Physical interface/buttons, under H8 (see Table 1). This study consented that the essential criterion is detecting the usability issues of the system rather than paying attention to the position of this sub-heuristic.

Table 1. Checklist for Heuristic Evaluation.

Heuristics & Sub-Heuristics Code		Explanation
(H1) Visibility of system status	SH01	System status feedback
	SH02	Local information
	SH03	Response time
	SH04	Selection/input of data
	SH05	Presentation adaption
(H2) Match between system and the real world	SH06	Metaphors
	SH07	Navigational structure
	SH08	Menus
	SH09	Simplicity
	SH10	Output of numeric information
(H3) User control	SH11	Explorable interface
	SH12	Some level of personalization
	SH13	Process confirmation
	SH14	Undo/cancellation
	SH15	Menu control
(H4) Consistency and standards	SH16	Design consistency: menus/input fields
	SH17	Naming convention consistency
	SH18	Menus/task consistency
	SH19	Functional goal consistency
	SH20	System response consistency
	SH21	Orientation
(H5) Error prevention	SH22	Fat-finger syndrome Accidental activation (lack of back button)
(H6) Recognition rather than recall	SH23	Memory load reduction
	SH24	General visual cues
	SH25	Input/output data
	SH26	Menus
	SH27	Navigation
(H7) Flexibility and efficiency of use	SH28	Search
	SH29	Navigation
(H8) Aesthetic and minimalist design	SH30	Multimedia content
	SH31	Icons
	SH32	Menus
	SH33	Orientation
	SH34	Navigation
	SH35	Physical interface/buttons
(H9) Help user recognize, diagnose & recover from errors	SH36	Error(s) Expression of error messages
(H10) Help and documentation	SH37	Help menu
	SH38	Documentation

In summary, the checklist is available for UX experts, which can be considered a comprehensive checklist mainly adapted from the 10 usability heuristics from Nielsen, with detailed explanations from Gómes et al. [23].

3.1.2. Participants

In this phase, the evaluation was carried out by five evaluators who had UX knowledge. They evaluated the usability aspects of the interactive applications and websites using the customized heuristics checklist. We invited these five participants via the professional network of our group members and the UX Design Group Meetups in Melbourne. The evaluators are Vietnamese and Australian, who are working at a private companies and universities in the area of UX. UX experts all confirmed that they are interaction designers (AR/VR), UX engineers, or UI designers. All UX experts are familiar with Nielsen’s heuristics. The evaluated information is in English, including the videos about audio guides’ operations and websites. The information about their years’ experience in UX has not been sufficiently collected.

3.1.3. Procedures

Five UX experts conducted the heuristic evaluations, for both museum website and interactive application evaluations. The checklist was sent to the UX experts to evaluate for both audio guides and museums websites.

For audio guides, in each museum, videos about its operation (all in English) were filmed and sent to the evaluators. The length of each video is 3 min maximum and were recorded by the camera with at least 7.2 megapixel (the general rule for high-quality sharp prints is 300 pixels per inch) to guarantee the effective visualizations. We had three videos with 3:00, 2:05, and 2:15 min length for VNMH, TL, and NMA, respectively. To evaluate the usability of audio guides and websites, in this research, we consented that the quality of a video would not be paid much attention; instead, we focused on the performance of these applications (Figure 2). Expert-based evaluations were conducted via video of visitors’ interactions that contributed to data collection. Therefore, the experts did not need to attend the museum directly. However, evaluations via videos have their drawbacks, as discussed in the last part of Section 5.

Figure 2. The video snapshots from the videos about audio guides’ operation from VNMH (left), TL, NMA (right).

For website evaluations, the links of websites (including virtual museums) were also sent to these UX experts (Figure 3). The followings are the websites’ URLs.

Figure 3. Screenshot of homepage’s websites of VNMH.

• Vietnam National Museum of History: http://baotanglichsu.vn/en (accessed on 29 October 2021). Its virtual museums: http://disanvanhoaphatgiao.egal.vn/ (accessed: 29 October 2021) and http://denco.egal.vn/ (accessed on 29 October 2021).
• Hanoi Temple of Literature: http://vanmieu.gov.vn/en (accessed on 29 October 2021).
• National Museum of Australia: https://www.nma.gov.au/ (accessed on 29 October 2021).

3.1.4. The Heuristic Checklist

As mentioned in phase 2′s method (Section 3.2.1), a heuristic document was prepared with a set of 10 heuristics and 38 sub-heuristics and a brief explanation of each sub-heuristic. Table 1 provides the list of heuristics.

In terms of the task list for audio guides, evaluators watched and evaluated one video per museum about the audio guide’s operation. Regarding the tasks for the websites, the evaluators assessed the museum’s homepages and the first children.

3.1.5. Conducting a Heuristic Evaluation

The evaluators were introduced to the evaluation process with a brief introduction provided by email. Before starting the usability test, evaluators were briefed on purpose, and the relevance of the testing, and concerns or questions were cleared. All necessary information to evaluate audio guides and websites is available in English. The Nielsen heuristics list was sent to them together with the consent form, the evaluation checklist, and the description of our project before the evaluation. The evaluators were asked to familiarize themselves with the HE checklist. While evaluating, they were asked to assign the rating and propose suggestions for the problems based on the heuristics provided. For each evaluator, a total of six checklists were expected to be returned (three museums and one report for app and one report for website). It took an evaluator approximately 2 to 3 h to evaluate both audio guides and websites independently. All documents were collected and filed.

3.2. Usability Survey in Phase 2

In this phase, we conducted field-usability surveys at three national museums for museum audio guides and websites via two questionnaires distributed to the visitors after they experienced using audio guides and museums’ websites. The surveyed museums include two national museums in Vietnam (VNMH and TL) and one national museum in Australia (NMA). The survey was chosen because this method is simple and stimulating, used prevalently in CH field to collect both quantitative and qualitative, and goes well with the usefulness attribute [20].

3.2.1. Procedures

When visitors entered the museums, they were approached and asked whether they would like to participate (two survey languages were Vietnamese or English). If they were willing to join, and did not borrow the audio guide, it was delivered to visitors. The consent forms and two questionnaires were handed over to the visitors. After experiencing the exhibition(s) freely with audio guides and museums’ websites, the visitors came back to our survey area and answered the questionnaires, which took approximately 25 min to complete. One laptop with an internet connection was available to participants to search or verify information from the museum’s website. During the survey, the researcher encouraged participants to discuss if there were some things unclear or vague. After completing the survey, two questionnaires were collected on the spot.

3.2.2. Measures

The questionnaire for audio guides we based on the combination of the scales (measuring the usability of using multimedia guide [52], constructed on the variety of prestigious studies and professional organizations from the UK; and the criteria for evaluating the effectiveness and the usability of the interactive device [25]). The questionnaire for CH websites was designed to collect participants’ comments about information presentation, layout, navigation, and interface. Both questionnaires used five-point Likert scale, ranging from 1 (“strongly disagree”) to 5 (“strongly agree”). The results will be presented in Section 4.2.

3.2.3. Participants

There were 205 visitors, including 117 females and 88 males, participating in the study in both countries (110 for Vietnam and 95 for Australia). The respondents came from an array of demographic backgrounds (different countries, educations, or work backgrounds). A significant proportion of participants was in the age group 25–39 (41.46%). The dominant age group in Vietnamese museums was in18–24 (46.36%), while the major age group in the Australian museum was 25–39 (45.53%). At Vietnamese museums, 80.9% Vietnamese answered our survey. In contrast, at the Australian museum, the number of participants who were native English speakers was 52.63%, near-native speakers were 13.68%, good English speakers were 20%, and average English speakers were 13.68%.

4. Results

Our usability evaluations presented in this section are obtained from phase 1, the HE, and phase 2, the usability survey. Our findings from HE will be presented in Section 4.1, and the usability survey will be presented in Section 4.2.

4.1. Heuristic Evaluations from UX Experts

In this session, the heuristic evaluation results and some comparisons between the analyzed data are presented.

The outputs of the HE were two schemes on measurement of severity levels and the agreement among evaluators for audio guides (Section 4.1.1) and websites (Section 4.1.2). These schemes are presented by the heat maps, the visual representations showing the comparative views of measurement of severity levels for audio guides and websites, and agreements of 5 evaluators on the severity levels of the three national museums.

The measurement of severity levels is a weighted scheme where levels are taken into account. We calculate the weighted values by multiplying value of the severity level by its level and then add them together. For example, for VNMH’s SH01, three evaluators ranked its severity at 1, 2, 3 levels. Thus, the overall calculation would be: (1 × 1) + (1 × 2) + (1 × 3) = 6.

Regarding the color scheme of the measurement of severity levels, the dark red represents the highest value data points (the most severe error), and the green represents the lowest value data points (no error). Regarding the color scheme of the agreement of five evaluators on the severity scale, the dark red represents the lowest agreement, and the green represents the highest agreement.

The agreement among evaluators is computed by the sum of five severity levels (with level 0 is no error) divided by the count of those five severity levels provided by all evaluators.

In other words, the sub-heuristics with the highest measurement of severity level and the minor agreement of evaluators are highlighted in red. (See Table 2 and Table 3).

Table 2. Measurement of severity levels for audio guides (left heat map with red is the highest and green is the lowest values); and the agreement of five evaluators on severity scales (right heat map with red is the lowest and green is the highest values).

Heuristics & Sub-Heuristics Code		Explanation	Measurement of Severity Levels for Audio Guides			Agreement of 5 Evaluators on Severity Scales
			VNMH	TL	NMA	VNMH	TL	NMA
H1	SH01	System status feedback	6	4	0	1.25	1.67	5
	SH02	Local information	0	0	0	5	5	5
	SH03	Response time	3	0	0	1.67	5	5
	SH04	Selection/input of data	5	0	0	5	5	5
	SH05	Presentation adaptation	0	0	0	5	5	5
H2	SH06	Metaphors	7	0	1	1.67	5	2.5
	SH07	Navigational structure	3	0	4	2.5	5	1.67
	SH08	Menus	3	3	3	2.5	2.5	2.5
	SH09	Simplicity	7	0	0	2.5	2.5	2.5
	SH10	Output of numeric information	0	0	0	5	5	5
H3	SH11	Explorable interface	3	0	0	2.5	5	5
	SH12	Some level of personalization	0	0	0	5	5	5
	SH13	Process confirmation	3	1	2	2.5	2.5	2.5
	SH14	Undo/cancellation	7	4	0	1.67	2.5	5
	SH15	Menu control	7	0	0	1.67	5	5
H4	SH16	Design consistency: menus/input fields	0	0	3	5	5	2.5
	SH17	Naming convention consistency	0	0	0	5	5	5
	SH18	Menus/task consistency	0	0	0	5	5	5
	SH19	Functional goal consistency	0	2	0	5	2.5	5
	SH20	System response consistency	1	0	0	2.5	5	5
	SH21	Orientation	0	0	0	5	5	5
H5	SH22	Fat-finger syndrome. Accidental activation (lack of back button)	10	5	0	1.67	1.67	5
	SH23	Memory load reduction	0	0	0	5	5	5
H6	SH24	General visual cues	2	0	0	2.5	5	5
	SH25	Input/output data	0	0	0	5	5	5
	SH26	Menus	3	0	0	2.5	5	5
	SH27	Navigation	3	0	5	2.5	5	1.67
H7	SH28	Search	0	0	0	5	5	5
	SH29	Navigation	0	0	3	5	5	2.5
H8	SH30	Multimedia content	5	3	0	1.67	1.67	5
	SH31	Icons	1	2	0	2.5	2.5	5
	SH32	Menus	0	0	0	5	5	5
	SH33	Orientation	0	0	0	5	5	5
	SH34	Navigation	2	0	0	2.5	5	5
	SH35	Physical buttons	6	5	0	2.5	2.5	5
H9	SH36	Error(s); Expression of error messages	5	1	3	1.67	2.5	1.67
H10	SH37	Help menu	10	5	2	1.67	1.67	2.5
	SH38	Documentation	3	3	0	2.5	2.5	5

Table 3. Measurement of severity levels for websites (left heat map with red is the highest and green is the lowest values); and the agreement of five evaluators on severity scales (right heat map with red is the lowest and green is the highest values).

Heuristics & Sub-Heuristics Code		Explanation	Measurement of Severity Levels for Websites			Agreement of 5 Evaluators on Severity Scales
			VNMH	TL	NMA	VNMH	TL	NMA
H1	W_SH01	System status feedback	10	4	0	1.00	1.67	5.00
	W_SH02	Local information	0	6	0	5.00	1.67	5.00
	W_SH03	Response time	3	10	0	1.67	1.67	5.00
	W_SH04	Selection/input of data	0	0	2	5.00	5.00	2.50
	W_SH05	Presentation adaption	0	0	0	5.00	5.00	5.00
H2	W_SH06	Metaphors	6	0	0	1.67	5.00	5.00
	W_SH07	Navigational structure	6	3	5	2.50	2.50	1.67
	W_SH08	Menus	3	1	0	1.67	2.50	5.00
	W_SH09	Simplicity	4	4	0	2.50	2.50	5.00
	W_SH10	Output of numeric information	0	0	0	5.00	5.00	5.00
H3	W_SH11	Explorable interface	4	4	4	2.50	2.50	1.67
	W_SH12	Some level of personalization	0	0	0	5.00	5.00	5.00
	W_SH13	Process confirmation	0	0	0	5.00	5.00	5.00
	W_SH14	Undo/cancellation	4	0	0	2.50	5.00	5.00
	W_SH15	Menu control	7	3	2	1.67	2.50	2.50
H4	W_SH16	Design consistency: menus/input fields	4	0	2	1.67	5.00	2.50
	W_SH17	Naming convention consistency	0	0	0	5.00	5.00	5.00
	W_SH18	Menus/task consistency	1	0	0	2.50	5.00	5.00
	W_SH19	Functional goal consistency	0	0	2	5.00	5.00	2.50
	W_SH20	System response consistency	0	0	0	5.00	5.00	5.00
	W_SH21	Orientation	0	0	0	5.00	5.00	5.00
H5	W_SH22	Fat - finger syndrome + Accidental activation (lack of back button);	3	0	0	2.50	5.00	5.00
	W_SH23	Memory load reduction	0	0	0	5.00	5.00	5.00
H6	W_SH24	General visual cues	5	0	0	1.67	5.00	5.00
	W_SH25	Input/output data	0	0	0	5.00	5.00	5.00
	W_SH26	Menus	7	0	0	1.67	5.00	5.00
	W_SH27	Navigation	0	4	0	5.00	1.67	5.00
H7	W_SH28	Search	10	9	0	1.00	1.25	5.00
	W_SH29	Navigation	5	7	2	1.67	1.67	2.50
H8	W_SH30	Multimedia content	1	3	4	2.50	1.67	2.50
	W_SH31	Icons	2	4	0	2.50	2.50	5.00
	W_SH32	Menus	3	4	0	1.67	2.50	5.00
	W_SH33	Orientation	0	4	0	5.00	2.50	5.00
	W_SH34	Navigation	0	7	0	5.00	1.67	5.00
H9	W_SH36	Error(s); Expression of error messages	8	0	0	1.67	5.00	5.00
H10	W_SH37	Help menu	4	5	2	1.67	1.67	2.50
	W_SH38	Documentation	1	9	2	2.50	1.25	2.50

A total of 163 usability issues were identified across five levels of severity. Out of the total, 72 and 91 issues were identified from the audio guides, and websites respectively.

4.1.1. Evaluator Ratings of Three CH’s Audio Guides

Out of 72 usability issues identified by 5 UX experts, Help menu and Physical interface/buttons were evaluated as having more issues. In detail, at the Physical interface/buttons, VNMH and TL were recorded with three and four issues each. For the Help menu, NMA audio guides had 1 error, while TL and VNMH’s audio guides had three errors. Audio guide’ heuristic Undo/cancellation, menu control, Accidental activation and Help menu of VNMH; Undo/cancellation, Accidental activation, Navigation of information, and Sequences of instructions of TL were classified as catastrophic issues and needed immediate action. Twenty-seven issues were categorized as major issues and needed to be rectified, including five issues from MNA (Navigational structure, Menus—H2, Design consistency: menus/input fields—H4, Navigation—H6), four issues from TL (System status feedback—H1, Menus—H2, Help menu, Documentation—H10), and 18 issues from VNMH (mostly at H1, H2, H3, and H10).

Via the left heat map, we can see the green color dominated the measurement of severity levels, in which five evaluators marked these data points as errorless (values = 0). The bigger values were recorded for the red, orange, and yellow colors (from 10 down to 3, especially at VNMH. In particular, accidental activation (lack of back button) and help menu were recorded as the most severe issues for the audio guide at this museum with 10 scores.

The more green color in the NMA shows that the Australian museum’s audio guide has more preeminent functions than Vietnamese museum ones. In detail, there are 29/38 (76%) sub-heuristics of Australian museums’ audio guide recorded no error, while this number is 41/76 (54%) for Vietnamese museums’ audio guides. For the right side of the heat map, the highest agreement among evaluators was seen at H4 and H7; and at NMA (see the Table 2).

Among 72 usability issues, major problems make up 37.5% of the issues (27 issues), cosmetic and minor problems make up nearly 25% of the issues (18 and 19 issues respectively), and the percentage of catastrophic problems is the smallest amount making up 11.11% of the issues (8 issues).

Figure 4 shows the total number of audio guides’ issues of the three national museums regardless of the severity rating scales. VNMH’s audio guides are recorded having more problems at SH01-System status feedback, SH09-Simplicity, SH22-Fat finger syndrome + Accidental activation, SH35-Physical interface/buttons, and SH37-Help menu. The total numbers of audio guides’ issues of VNMH, TL, and NMA are 41, 19 and, 12, respectively. Of the data gathered, Vietnamese audio guides noted more usability problems than Australia with the ratios of 3.4:1 (VNMH and NMA), and 1.6:1 (TL and NMA). The Vietnamese audio guides’ usability problems were mainly recorded as the outdated model, only text appears on the screen, lacking the related image(s) to the background and illustration or videos (VNMH), and bulky and old-styled device (TL).

Figure 4. Usability issues of audio guides of 3 museums surveyed.

4.1.2. Evaluator Ratings of 3 CH’s Websites

Among the 91 usability issues identified by five UX experts, Search was evaluated as having more issues with seven errors each, in which TL and VNMH have recorded three and four issues, respectively. System status feedback and Navigation were noted at six errors, with two issues at VNMH, two issues at TL, and one issue at NMA (in bold). Six teen websites’ heuristics were classified as catastrophic issues and needed immediate action. They are System status feedback, Metaphors, Simplicity, Undo/cancellation, and Search of VNMH; Local information, Simplicity, Explorable interface, Search, Navigation, Icons, Menus, Orientation, Navigation, and Documentation of TL; and Multimedia content of NMA. Twenty-four issues were categorized as major issues and needed to be rectified. They are the issues from VNMH (System status feedback, Navigational structure, Menu control, Accidental activation, General visual cues, Menus, Search, Navigation, Error(s), and Help menu); issues from TL (System status feedback, Response time, Navigational structure, Menu control, Navigation, Search, Navigation, Help menu, Documentation); and issue from MNA (Navigational structure).

In terms of measurement of severity levels, the weighted scheme shows that the system status feedback (H1) and Search (H7) of VNMH; and Response time (H1) of TL were scored highest. More green in the NMA websites indicates that the Australian museum website is also better than those in Vietnamese museums. In particular, Australian museums’ website has 29/38 (76%) sub-heuristics with no errors, but Vietnamese museums’ websites have 34/76 (45%) sub-heuristics with no errors.

Regarding the agreement on evaluation, H4 and H6 had the highest unanimity (Table 3).

Among 91 usability issues, minor problems make up 35.16% of the issues (32 issues), percentage of major problems is 26.37% (24 issues), cosmetic problems and catastrophic problems are relatively small with 20.88% (19 issues) and 17.58% of the issues (16 issues) respectively. Figure 5 shows the total number of websites’ issues of the three national museums regardless of the severity rating scales. The number of websites’ issues of VNMH, TL, and NMA are 46, 32, and 13, respectively. The ratios of usability problems detected for Vietnamese and Australian museum websites are 3.5:1 (VNMH and NMA) and 2.5:1 (TL and NMA). Vietnamese museums’ websites’ usability errors mainly are about response time (slow when displaying the details of images from the gallery), aesthetic and minimalist design (the broken site layouts causing all of the issues of menus and orientation), and search (inactive advanced search). The Vietnamese museums’ websites’ usability errors are recorded at navigation (main navigation arrows don’t follow convention) and menu (many menu controls are not intuitive).

Figure 5. Usability issues of websites of 3 museums surveyed.

4.2. Usability Survey

Usability issues of museums’ audio guides and websites are categorized primarily based on Nielsen heuristics, explanations from Gómez [14], and one sub-heuristic from Inostroza et al. [17]. As stated in the introduction, this usability survey is a part of a more extensive study about how to design good UX for interactive technologies in developed and developing countries.

4.2.1. Usability Evaluation for Audio Guides

We used 15 close-ended questions and two open-ended questions to measure the usability of audio guides (see Table A1-English and Table A2-Vietnamese-Appendix A). To analyze the data, we pay attention to the number of participants who ranked strongly disagree and disagree to see how visitors ranked statements negatively. The results are presented in Table 4.

Table 4. Usability evaluation of audio guides (bold values represent the highest number of Strongly Disagree or Disagree in each museum.).

Subheuristic (Nielsen Heuristic)	Audio Guide Usability	Number of Strongly Disagree or Disagree		AM (N = 95)
		VNMH (N = 58)	TL (N = 52)
US01 (H2)	The information given by the audio guide is reasonable and understandable	9	0	0
US02 (H1)	It is easy to determine where I am in the visit with the audio guide	24	4	10
US03 (H4)	Learning to operate the audio guide is easy	10	1	2
US04 (H2/H10)	I am clear with all instructions/help of audio guide	9	1	3
US05 (H4)	Using the audio guide does not require much training	8	1	0
US06 (H6)	It is not difficult to select the option I want with the audio guide	21	2	8
US07 (H3)	I feel that I am in control of audio guide	19	2	8
US08 (H2/H4/H7/H8)	I find it easy to read the text on the screen of the audio guide	9	11	4
US09 (H2/H4/H7/H8)	I find it easy to hear the material on the audio guide	5	1	4
US10 (H8)	The audio guide presents information in an understandable manner	5	0	4
US11 (H1)	The audio guide clearly provides feedback about my actions	16	6	15
US12 (H2)	My experience in the virtual environment is consistent with my real-world experience	7	4	14
US13 (H1/H7)	I am able to anticipate what would happen next in response to the actions that I perform	20	4	4
US14 (H1/H7)	I experience the delay between my actions and expected outcomes	36	3	10
US15 (H8)	The visual display quality interferes or distracts me from performing required activities	16	1	3

The number of people who either strongly disagreed or disagreed with these statements is presented in Table 4. The most significant number of the VNMH was “delay between my actions and expected outcomes” (62.07%), TL was “easy to read the text on the screen” (21.15%), and AM was also “easy to read the text on the screen” (15.79%). VNMH’s audio guide was recorded as the device with the most significant number in negative ranking. Most of the museum audio guide’s metrics had a negative rank. Only the statement “The information given by the audio guide is reasonable and understandable” was not ranked negatively at two museums (TL and AM). Values of audio guide factors that have the highest negative ranking are highlighted in Table 4.

The thematic analysis from participants showed more issues from the audio guides via the question about the worst aspect(s) of using the device and what the visitors would change about the worst aspect(s). Firstly, visitors of Vietnamese museums commented that the biggest limitation of the automatic VNMH audio guide was the delay or miss-presentation between the content and the physical relics, resulting in the visitors hearing the wrong audio (33% of VNMH responses), which links to H2 (Match between system and the real world). Comments such as “hard to use” (14% of VNMH responses), “inflexibleness” (H7, flexibility, and efficiency of use), “hard to concentrate”, and “less interaction” were recorded. Besides, the content of audio guides should be concise and rich, which was expected to be presented in more languages rather than Vietnamese and English (H4, Consistency). No option for choosing the language in the device (museum installed the language when visitors borrowed this device at the reception desk) was the worst function of TL audio guide (12% of TL responses) (H2, Match between system and the real world). A location-aware audio guide was expected. Otherwise, a map about the exhibits’ stops should be included in the app (10% of TL responses) (H6, Recognition rather than recall). Due to the redundancy of buttons (buttons have no function), VNMH’s audio guide looks cumbersome. The overall size of the TL audio guide needs to decrease, while the screen/button size should be larger. Besides, the content was relatively short with a slow reading speed, and adding pictures would make the content dissemination more effective (H4, Consistency).

Regarding the issues of the NMA audio guide, the limited languages (only English) and too long audios/lots of reading on a small screen are the biggest negative sides of the app (10% of NMA responses) (H2, Match between system and the real world). Challenging to plan where to go/app need to know where the visitor is in the museum/unclear what path to follow is the audio guide’s drawbacks also related to the function of location awareness (H6, Recognition rather than recall).

The visitors’ suggestions are simple and easy to make, such as the suggestions for the issue “not a 2-way interaction” is “add visual technologies” or “need a 2-way interaction”. Some of them are valuable, for example, “the map should be included in the app”, “speed of audio guide should be faster” for TL audio guide; “instead of typing the section you are in, the app should use GPS then automatically start to display the art”, “The app could ask about general interests and then plan a visit for you.”, “The app could ask about general interests and then plan a visit for you.”, and “I would add foreign languages” for NMA’s audio guide.

To sum up, we can see the negative rankings on these 15 statements with the usability survey. The most significant numbers were almost seen at VNMH (214 times for strongly disagree and disagree), followed by AM (89 times for strongly disagree and disagree). On the other hand, the details of errors and user suggestions/solutions were presented via open-ended questions.

4.2.2. Usability Evaluation for Websites

We used 13 open-ended questions, and two open-ended questions to examine the usability of museums’ websites (see Table A3-English and Table A4-Vietnamese, Appendix B). The number of participants who ranked strongly disagree and disagree was analyzed. Table 5 presents the results.

Table 5. Usability evaluation of websites (bold values represent the highest number of Strongly Disagree or Disagree in each museum).

Subheuristic (Nielsen Heuristic)	Website Usability	Number of Strongly Disagree or Disagree		AM (N = 95)
		VNMH (N = 58)	TL (N = 52)
WUS01 (H2/H8)	I find that there is a reasonable number of words on each website page	1	1	13
WUS02 (H4)	The contents of the website are useful to me	1	0	1
WUS03 (H4)	The information about the museum exhibits on the website is clear	1	2	2
WUS04 (H7)	I often perform the tasks without getting any error	6	7	5
WUS05 (H2)	I find that there is terminology jargon on the website	6	12	15
WUS06 (H3)	I find it convenient to choose the languages displayed	1	3	51
WUS07 (H7)	I find it is easy to navigate the website	3	1	3
WUS08 (H4)	I am able to discover/learn individually about the objects/exhibits from the website	3	4	3
WUS09 (H10)	I am able to perform the tasks easily with the help of the website	3	2	4
WUS10 (H1)	I find it there is ambiguity of link labels	19	28	9
WUS11 (H3/H8)	I enjoy the layout of the website	4	6	4
WUS12 (H4)	I find that the page layout is consistent throughout the website	1	2	3
WUS13 (H6/H8)	I find the graphics on the website are attractive	3	9	1

Approximately 53.68% of people strongly disagreed or disagreed with the statement “I find it convenient to choose the languages displayed” was the largest number at AM. “The ambiguity of link label” was recorded as the statement with the most significant negative rankings at TL (53.85%) and VNMH (32.76%). The values in bold show the website factors that have the highest negative ranking in each museum.

Thematic analysis was conducted following Braun and Clark [53] by Nvivo 12. The first author read through the response and created codes by counting the number of occurrences to identify major themes. The codes were reviewed by the second author to finalize. The thematic analysis of open-ended questions recorded the answers for the worst aspects of using Vietnamese museum websites. Too much information on one page (H8—Aesthetic and minimalist design) was the most significant limitation of Vietnamese museums’ websites, mainly TL (13% of TL responses). While VNMH virtual museums received lots of positive feedback from visitors on the audio and visual aspects, their content needs to be updated and richer (H8). At TL, more clips, pictures, animations, virtual exhibitions, eye-catching layout, and good interaction need to build to enhance the visitors’ attractiveness (H8). Vietnamese and English are not enough for visitors, and visitors need more languages such as German, Japanese (H2—Match between system and the real world). There is some Vietnamese content in the English version as the inconsistent error in presentation (H4, Consistency).

Like the significant defect of Vietnamese museums websites, information (H8) overwhelmed was commented most by NMA’s participants (11% of NMA responses). The second drawback of the NMA website is language (H2) (no Italian language, more language expectation, certain words written in American rather than English, could not find text in other languages, using jargon). Content limitation (H8), all exhibitions look the same (H6), and not much visual display (H8) are other error detections. Long roll and cannot see the whole screen were noticed as the aesthetic and minimalist design drawbacks. Lastly, the menu button had a problem (slightly unresponsive and moved a lot).

Visitors proposed suggestions/solutions, such as “should have more languages rather than just Vietnamese, English”, and “should mark the places with numbers and have the guide to use” for VNMH’s websites. They also suggested “Improve the quality of content, and sound” for TL’s website; and “support more languages, even if it is only simple how to get or how to park in different languages”, and “more interactive, more videos such as virtual museums” for NMA’s website.

In closing, via user survey, we see the most negative rankings at the “ambiguity of link labels” and “convenient to choose the languages displayed”. Those biggest numbers were almost seen at AM (114 times for strongly disagree and disagree) followed by TL (77 times for strongly disagree and disagree). The open-ended questions showed the details of errors and user suggestions/solutions for the errors.

4.2.3. Comparisons of Usability Evaluations for Audio Guides and Websites

To compare the data of usability evaluations for museums’ audio guides and websites of developing and developed countries, we grouped the 2 Vietnamese national museums’ mean scores. They are both national CH museums in the Vietnamese capital city. We use Kruskal-Wallis H Test to determine statistically significant differences between CH museums in two developing and developed countries.

As stated in Section 4.2.1, we used 15 close-ended questions (and 2 open-ended questions) to measure the usability of audio guides (see Appendix A). All p-values of the Shapiro Wilk Test are smaller than 0.05. We do not assume these normal distributions. Hence, the overall level of visitor experience was investigated for audio guides by conducting an analysis of the Kruskal–Wallis test to check the significant differences in the audio guide usability factors. The results are presented in Table 6.

Table 6. Usability evaluation of audio guides: Mean, SD, and p-value.

Audio Guide Usability	Mean Scores (±SD)		Asymp. Sig.
	Vietnamese Museums (N = 110)	Australian Museum (N = 95)
US01	3.84 ± 0.761	4.52 ± 0.543	0.000
US02	3.41 ±1.034	4.03 ± 0.973	0.000
US03	3.81 ± 0.851	4.31 ± 0.716	0.000
US04	3.82 ± 0.826	4.18 ± 0.743	0.001
US05	3.85 ± 0.826	4.45 ± 0.541	0.000
US06	3.33 ± 0.910	3.89 ± 0.805	0.000
US07	3.39 ± 0.987	3.96 ± 0.967	0.000
US08	3.45 ± 0.925	4.16 ± 0.854	0.000
US09	4.01 ± 0.760	4.45 ± 0.755	0.000
US10	3.90 ± 0.729	4.33 ± 0.736	0.000
US11	3.25 ± 0.893	3.19 ± 0.829	0.495
US12	3.46 ± 0.725	3.56 ± 0.931	0.324
US13	3.15 ± 0.859	3.63 ± 0.718	0.000
US14	3.09 ± 0.963	3.56 ± 0.837	0.000
US15	3.39 ± 0.858	3.83 ± 0.698	0.000

Many of the p values (Asymp. Sig.) of the samples from Vietnamese and Australian museums were less than 0.05, showing that the differences in weight gain are statistically significant. However, for the two statements US11 (Provide clear feedback—H1) and US12 (Consistent experience in the virtual and real world—H2), their p-values were 0.495 and 0.324, respectively, hence the differences in the two of them are not statistically significant. Most of the mean scores of Australian museums are significantly larger than that of Vietnamese museums. The largest Kruskal-Wallis H is 44.844 at US01 (Reasonable and understandable information—H4), indicating the most significant difference between groups we are comparing.

To compare the usability evaluations of museums’ websites, we used the 13 open-ended questions and two open-ended questions were used to examine the usability of museums’ websites (see Appendix B). Because the data distributions are not normal, we used Kruskal–Wallis test to check the significant differences in website usability factors. Table 7 presents the results.

Table 7. Usability evaluation of websites: Mean, SD, and p-value.

Website Usability	Mean Scores (±SD)		Asymp. Sig.
	Vietnamese Museums (N = 110)	Australian Museum (N = 95)
WUS01	3.93 ± 0.550	3.87 ± 0.948	0.560
WUS02	4.12 ± 0.532	4.11 ± 0.610	0.947
WUS03	3.97 ± 0.588	4.17 ± 0.647	0.014
WUS04	3.37 ± 0.783	3.92 ± 0.753	0.000
WUS05	3.41 ± 0.837	3.51 ± 0.849	0.424
WUS06	4.04 ± 0.716	2.66 ± 1.017	0.000
WUS07	3.91 ± 0.705	4.14 ± 0.662	0.015
WUS08	3.96 ± 0.702	4.04 ± 0.683	0.446
WUS09	3.88 ± 0.694	3.76 ± 0.782	0.181
WUS10	2.69 ± 0.832	3.60 ± 0.804	0.000
WUS11	3.65 ± 0.840	3.92 ± 0.663	0.007
WUS12	3.77 ± 0.595	4.05 ± 0.658	0.001
WUS13	3.71 ± 0.874	4.12 ± 0.650	0.001

The differences in WUS03, WUS04, WUS06, WUS07, and WUS10 to WUS13 are statistically significant as their p-values are less than 0.05, while the other website’ usability are not. The greatest Kruskal–Wallis H is recorded at WUS06 (Convenient in language choosing, H3) at 76.982, showing the largest difference between Vietnamese and Australian national museum websites.

4.3. Suggestions to Fix the Usability Problems from UX Experts

The detailed problems and their suggestions are presented in Appendix C, Appendix D and Appendix E. Here, we emphasize two main aspects (1) the sub-heuristic receiving more suggestions from five experts and (2) suggestions for the catastrophic problems (severity level = 4) needing immediate actions. Interestingly, there was no fault marked as a catastrophic problem for NMA’s audio guide and website. Therefore, the following suggestions are for Vietnamese museums: VNMH and TL.

4.3.1. Suggestions for Audio Guides

The UX experts’ suggestions for three museums’ audio guides are presented in Appendix C (Table A5, VNMH), Appendix D (Table A7, TL), and Appendix E (Table A9, NMA).

(1)

Four out of five UX experts gave suggestions for sub-heuristic SH35, physical buttons of VNMH’s and TL’s audio guides, in which mainly the useless buttons should be removed. Sub-heuristics of VNMH’s audio guide, including system status feedback; simplicity; accidental activation; and help menu, are received three suggestions from five UX experts. Help menu from TL’s audio guide also had three suggestions from the experts.

(2)

Regarding VNMH’s audio guide, Metaphors need to be updated immediately because the current metaphors are somewhat dated and obsolete. The VNMH’s audio guide’s screen size needs to be redesigned urgently (analog screen supports well for only audio, not for pictures). In this vein, for Undo/cancellation (SH14) and Accidental activation (SH22), undo and redo buttons were expected to fix soon. Menu control needs signifiers for menu control, for example, pressing and holding the “L” button to change the language. For TL’s audio guide, a delete/backspace button was needed urgently to solve SH14 and SH22. See Appendix C (Table A5) and Appendix D (Table A7) for the details.

4.3.2. Suggestions for Websites

The UX experts’ suggestions for three museums’ websites are presented in Appendix C (Table A6, VNMH), Appendix D (Table A8, TL), and Appendix E (Table A10, NMA).

(1)

Four out of five UX experts detected the issues and gave suggestions for VNMH’s websites on System status feedback, and Search; for TL’s website on Response time, and Multimedia control. Those sub-heuristics, including Design consistency: menus/input fields, Navigation, and Error(s); Expression of error messages at VNMH; and Search at TL received 3 suggestions for each.

(2)

For VNMH’s website, System status feedback, Metaphors, Simplicity, and Search needed improvement immediately, while Simplicity from TL needed urgent actions. See Appendix C (Table A6) and Appendix D (Table A8) for the details.

5. Discussion

There are many ways to identify usability problems [20,43,48,52]. The combination of HE and usability survey (the open-ended questions) can help detect the usability problems of the systems, and they may be appropriate for identifying some specific issues [18]. In this vein, HE or usability inspection identified many issues and proposed suggestions to fix those [37].

Regarding the research method, both techniques, including HE and usability surveys, help our findings meet five out of eight attributes, including usefulness, ease of use, efficiency, accessibility, and identification [19,20]. To investigate the agreement of real users of a system with the problems identified by HE, HE should be conducted first, and then a real users’ survey [18] to verify the result. In phase 1, the checklist used for HE was experimentally evaluated as a tool for design, relevant to the tested product in the CH domain, and overcome the irrelevant problems mentioned in Mazlan et al. [54]. We used the Nielsen’s severity four-levels, including ‘cosmetic problem’, ‘minor problem’, ‘major problem’, and ‘catastrophic problem’ to pay attention to all the issues, which is different from using the two final levels only of Khajouei et al. [18].

In this study, the usability surveys, a part of the bigger UX surveys, support and consolidate for HE results which was mentioned in Section 1, Introduction. Similar to our study’s purpose, Karoulis et al. [48] conducted the usability evaluations of the museum’s system, in which the evaluators’ opinions (from two groups: usability experts and museums’ curators) were concluded differently.

We find that HE is sensitive to identify. Usability surveys contribute the users’ opinions to support for experts’ evaluations through their rankings or replies to the open-ended questions. For example, in the audio guide’s H1—Visibility of system status, experts found the problems at SH01 (at VNMH and TL), SH03 (at VNMH), and SH04 (at VNMH). Real users also ranked negatively on US02, US11, and US13 at the three museums (see Table 4). Another example from H8—Aesthetic and minimalist design, UX experts detected the issues on SH30, SH31. Usability surveys showed the strongly disagree and disagree rankings on SH08, SH09, and US15. Mainly, usability issues on the new SH35 (Physical interface/buttons) were found by the UX experts to be modified by the users’ replies from the open-ended questions, in which both buttons’ size (at TL) and redundancy button (at VNMH) inhibit the user’s use. In terms of the usability errors from the website’s H7—Flexibility and efficiency of use, experts showed the issues on W_SH28 (Search), and W_SH29 (Navigation), the real visitors complemented via the negative rankings on WUS04 (perform the tasks without getting any error) and WUS07 (easy to navigate the website) (see Table 5). In general, Vietnamese museum audio guides and websites were recorded as having more usability problems than Australian’s. Vietnamese CH museums’ audio guides are outdated and cumbersome with analog screens, low resolution, and ten useless number buttons (VNMH’s). Vietnamese museums’ website problems relate to information presentation, bandwidth limit especially for virtual museums, and language (in the English version).

Some data need to have more explanations and discussions. For Vietnamese museums’ audio guides, the delay or miss-presentation between the content in the device and physical relics was the greatest weakness of the VNMH automatic audio guide, causing visitors to hear the wrong audio. The error in locational tracking may be due to the smaller than prescribed distance of the arrangement of objects (3 m < 5 m).

Audio guides in the museums have essential limitations, including small screen size, limited input capability, different input methods, bandwidth or network access, different display resolutions, and minimal or different processing capacities. Hence, the capability to attract visitors’ attention was limited [55]. According to Bredin [56], the good audio guide’s ingredients are free of charge, good content (specialist knowledge), multilingual support, and support for disabilities (for example, visually impaired). Similarly, undersized screen, low resolution, bulky appearance, and cost of use are the drawbacks of VNMH and TL’s audio guides. NMA’s audio guide, on the other hand, does not support multilingualism. Regarding CH websites, good examples are good interaction, flexible accessibility, and intuitive navigation [57]. The three museums’ website limitations are too much information on one page. The navigation and accessibility errors are not recorded. The VNMH and TL’s website participants still encounter dead link(s), less interaction, and unattractive interfaces. The NMA’s website users dissatisfy with the excessive scrolling, unresponsive menu button, and not much interaction.

The agreements of users and the experts were at a medium level in Khajouei et al. [18]; however, usability surveys strengthen the HE results in this research.

Much of the studies illustrate very little agreement on a usability problem and how severe it is [48,58], and this study found the similarities. The agreement is mostly seen at the “no error” data points. Catastrophic problems for both audio guides and websites were recorded at the smallest amount at 12.5% and 16.67%, respectively. However, the number of cosmetic problems is not the highest compared to minor, major and catastrophic problems as shown in Karat et al.’ findings [59].

Having confirmed the disparity of the evaluators, the next point of discussion is conducting the HE via videos has certain drawbacks because some issues might only be detected by physically visiting the museums to experience the device. Some outstanding examples are “the delay between the actions and expected outcomes” or “cannot match the content of physical object/place” which are not recognized by experts via video examination. However, Karoulis et al. [48] showed that usability experts are closer to the goal (even than the museum curators) because they understand thoroughly what and how has to be examined. That may explain why experts (4 out of five) detected those useless buttons on VNMH and LT’s audio guides, while the visitors could not state the issue clearly than experts.

More usability issues were detected and examined by 5 UX experts, and the usability surveys strengthened the HE results from the real visitors’ viewpoints.

The issues identified by UX experts are allocated in 10 heuristics and usability surveys to help the museums’ curators, and interactive designers thoroughly understand their products. One of the suggestions from our UX experts for VNMH’s website is the recommendation for Multimedia content (SH30), in which Flash used for virtual museums cannot be browsed in some videos. Therefore, VNMH should use another platform replacing Flash [23,60].

From this 2-phase study, to have good “ingredients” for interactive app/website, some technical considerations should be paid attention to:

• Help menu and documentation should be provided effectively to users.
• Multilingual support is required due to exposure to international visitors.
• The two-way interaction, flexible accessibility and trouble-free navigation are needed.
• The quality of the device should be sufficient. For example, the quality of audio guides’ screens and screens for web browsing are crucial.
• Considering technology for information presentation for accessibility and cross-platform compatibility should be paid attention; for example, Flash is replaced with HTML5 or WebGL.

Other design recommendations need to be considered, such as the interactive app/website should be free of charge, and the content should be concise, rich, and current.

6. Conclusions

The paper shows the usability issues of museums’ interactive apps and websites from different angles, experiences, and approaches. Because of these differences, the number of UX experts’ suggestions is more than that of real visitors (77 and 91 suggestions from UX experts, and 10 and 12 suggestions from visitors for audio guides and for websites, respectively). In common with the experts, real visitors pointed out some similar usability issues; however, there are some different viewpoints on showing the issues that they experienced. HE is more sensitive to detect problems for both audio guides and websites in all usability heuristics, especially at H10.

In contrast, usability surveys quickly detected the issues relating to language-changing options. Additionally, HE ranks the severity scales of the problems so that the museum curators can respond appropriately. Having said that, conducting the HE brings to light more significant results and usability surveys complementing HE. Besides, there are some valuable suggestions from real visitors that the UX experts were unable to detect. The examples are the evaluations of the audio guide via videos, such as the suggestion “the presentation of content of audio guide should be synchronized with the location of the visitors” to solve the issue “cannot match the content of physical object/place with the audio guide”.

In the context of Vietnamese and Australian museums, this paper is likely to be the first usability review to better understand the strengths and weaknesses of interactive technologies in CH areas, especially for developing countries such as Vietnam. The findings are expected to help the museum director and the museum’s prospective interactive service providers by reflecting on the usability of websites and on-site interaction installations. The HE complements the reliability of usability surveys. Additionally, experts’ suggestions can help CH organizations and interactive service providers improve their adoptions of these technologies to better satisfy user needs. The limitations of this study are that two Vietnamese national museums and one Australian national museum used for the survey are a small sample of national museums; hence the number of audio guides and websites surveyed is small.

Further, UX experts were recruited via emails at different periods, so there was no discussion among the evaluators about the issues and the severity levels. Finally, Nielsen’s 10 heuristics interpretation is subjective; therefore, there is likely a different classification of opportunity for sub-heuristics of usability surveys. The findings are the audio guides and websites’ usability issues detected and suggestions proposed by UX experts and visitors. Australian museum interactive technologies have fewer usability errors than those in Vietnam. What Vietnamese national museums should do to mitigate the limitations and improve the user experience, especially during and after the COVID-19 pandemic, will be presented in future work.