*3.2. Recommendation Algorithm*

The ACUX-R algorithm consists of three stages. In the first stage, the Classification stage (Section 3.2.1), the visitor is classified under one or more ACUX profiles according to their visiting preferences. In the second stage, the Adjustment stage (Section 3.2.2), the user is allowed to manually adjust their assigned profile(s), if desired, overriding the outcome of the Classification stage. In the third and final stage, the Recommendation stage (Section 3.2.3), the set of recommended POIs is calculated according to the user's final visiting profile, and the recommended POIs are presented as pins on a map and/or in the form of a list. Table 1 presents an overview of the ACUX-R algorithm.


**Table 1.** The ACUX-R recommendation algorithm.

#### 3.2.1. Classification Stage

The Classification stage is the initial stage of the recommendation algorithm, where the user's visiting profile is determined based on their visiting preferences. The user selects and provides as input information a set of icons (five or more) representing their visiting preferences. For example, the icons depicted in Figure 2 (from left to right: sculptures, galleries, arts and crafts, concert halls, and graffiti) represent the Art Seeker profile.

**Figure 2.** Icons visually representing the Art Seeker profile.

According to the specification of the ACUX typology [22], the various visiting preferences that form the ACUX profiles are not necessarily matched with a single profile. As such, the icons created in ACUX-R to represent those visiting preferences (forty in total) may correspond to multiple ACUX profiles. For example, the icon galleries, which represents the preference of visitors to visit galleries, is assigned to both the Archaeologist and the Art Seeker.

Next, based on the user's selected icons, a score is calculated for each ACUX profile as follows:

$$\begin{array}{c} \text{foreach i} \\ \text{SCORE}\_{\text{i}} = \text{s}\_{\text{i}} \text{s} \times \mathbf{100} \end{array} \tag{1}$$

where i is the ACUX profile identifier, **si** is the number of selected icons per ACUX profile, **s** is the total number of selected icons (**s** ≥ 5), and **SCOREi** is the score per ACUX profile, which is a number between 1 and 100.

For example, let's assume that Visitor1 selects the icons: museums, theatres, graffiti, lakes, distilleries, farms, and temples. According to the ACUX typology [22], museums are assigned both to the Archaeologist and the Art Seeker profiles, theatres and graffiti to the Art Seeker profile, lakes and farms to the Naturalist profile, distilleries to the Gourmand profile, and temples to both the Religious Seeker and the Archaeologist profile. As a result, Visitor1 is classified under the following profiles: Archaeologist with a score of 29 (2 out of 7 icons), Naturalist also with a score of 29 (2 out of 7 icons), Art Seeker with a score of 43 (3 out of 7 icons), Gourmand with a score of 14 (1 out of 7 icons), and Religious Seeker also with a score of 14 (1 out of 7 icons).

#### 3.2.2. Adjustment Stage

At the Adjustment stage, the visitor can override the results of the Classification stage and adjust their generated ACUX-R profile manually, given that they are not completely satisfied with the profiling outcome. This is a non-obligatory stage and has been implemented by providing in the GUI a set of slider controls, which enable the user to increase or decrease the generated score for each ACUX profile.

Following the same example, let us assume that Visitor1 is satisfied with the score of the Religious Seeker, Art Seeker, and Naturalist profiles but wishes to adjust the score for the Gourmand and Archaeologist profiles in order to receive more recommendations for restaurants and breweries and fewer for archaeological destinations. Consequently, Visitor1 sets the Archaeologist score to 15 and the Gourmand score to 30.

#### 3.2.3. Recommendation Stage

The Recommendation stage is the final stage of the ACUX algorithm, where the recommended POIs are specified based on the scores assigned in the previous stages. For each ACUX profile assigned to the user (i.e., for each ACUX profile with a non-zero score), one or more POIs are recommended as follows:

$$\begin{array}{c}\text{foreach i whose } \mathbf{SCORE\_i} > 0\\\mathbf{DISP\_i} = \mathbf{ROUNDUP(SCCORE\_i/100} \times \mathbf{p\_{i\nu}} \text{ 0)}\end{array}\tag{2}$$

where **pi** is the total number of POIs per ACUX profile, i is the ACUX profile identifier, **DISPi** is the number of recommended POIs per ACUX profile, and **SCOREi** is the generated score per ACUX profile. The total number of recommended POIs is the sum of **DISPi**.

Finally, drawing on the same example, the recommended POIs are presented to the user as pins on a map and also in the form of a list sorted according to the score of the Archaeologist, Art Seeker, Religious Seeker, Naturalist, and Gourmand profiles.

#### **4. Evaluation**

To assess the usefulness of ACUX-R in practice, we conducted a user study and an online questionnaire survey. Fifty participants of various ages, educational backgrounds, and current professions were chosen to participate in the user study, including academic staff and students from Aegean University, Android, and iOS developers, and also members of the local community (Mytilene, Lesvos). In general, the participants were regular smartphone users who enjoyed traveling and who had already visited the city of Athens or were planning to do so in the near future. Their ages ranged between 20 and 55 years.

First, the participants were asked to submit their background information, including demographic data and level of familiarity with cultural-tourism MRSs and mobile applications in general. As a next step, participants were briefly informed about ACUX-R and instructed to download and install it on their mobile devices, following online instructions (http://ii.ct.aegean.gr/acux-evaluation/, accessed on 5 September 2022). Participants were advised to work in groups or individually. Finally, a discussion was held based on the following topics:


As a next step, we conducted an online questionnaire survey. Thirty-five participants installed and used ACUX-R and then filled in a questionnaire (https://tinyurl.com/yayw3 853, accessed on 9 September 2022) (Appendix A), which is a part of the User Experience Questionnaire (UEQ) data analysis tool [37,38]. Both classical usability aspects (efficiency, perspicuity, dependability) and user experience aspects (originality, stimulation) were measured. Each item of the UEQ consisted of a pair of terms with opposite meanings (e.g., not understandable to understandable, inefficient to efficient). Each item was rated on a 7-point Likert scale. Thus, answers ranged from −3 (fully agree with a negative term) to

+3 (fully agree with a positive term). This analysis yielded the final questionnaire with 26 items, arranged in six scales:


Then, the mean values per scale for ACUX-R are compared with the existing mean values per scale for other products from a dataset provided by the UEQ data analysis tool, which contains data from 21,175 people from 468 studies concerning different products (business software, web pages, mobile apps, social networks). The overall results of the ACUX-R mean values per scale compared to the UEQ data set are depicted in Table 2 and Figure 3.

**Table 2.** ACUX-R mean values per scale compared to the UEQ dataset.


**Figure 3.** ACUX-R mean values per scale compared to the UEQ dataset.

In Table 3, the confidence interval per scale for the precision of the estimation of the scale mean is presented. The smaller the confidence interval, the higher the precision of the estimation and the reliability of the results.



In the user study conducted, ACUX-R was rated high in inspiration, excitement, interest, and enthusiasm. Most of the participants stated that they would like to get a diversified set of recommendations at the beginning when no further specific preferences are known to the system. Most of them stated that they chose recommendations based on the profile determined by ACUX-R, whereas only a few adjusted their profiles. Another issue discussed was that a future add-on of uploading images for profile classification could be overwhelming for most of the participants or lead to the absence of cultural features for a few of them. Moreover, most of the participants stated that even the best advice couldn't keep unexpected things from happening to cultural visitors while visiting a destination. For example, attractions may be temporarily closed due to inclement weather, and outdoor performances may be canceled. Finally, most participants appreciated the detailed information describing the ACUX profiles, which helped them understand why certain POIs had been recommended.

Regarding the online survey, the Stimulation and Novelty scales scored above average, indicating that, in general, found ACUX-R motivating and creative. Moreover, positive ratings on the Perspicuity and Efficiency scales indicate that ACUX-R usability features met or exceeded high criteria. The Dependability rating was below average, indicating that several participants experienced security and confidence difficulties. Finally, the Attractiveness scale was rated excellent, implying that the icon-based approach results in a pleasant experience.

#### **5. Conclusions and Future Research**

In this paper, we proposed the ACUX Recommender (ACUX-R), an MRS for personalized recommendations of cultural POIs to visitors based on their visiting preferences. The ACUX-R experimentally employs the ACUX typology for assigning profiles to cultural visitors. To assess the usefulness of ACUX-R in practice, a user study and an online questionnaire survey were conducted.

The evaluation showed that ACUX-R satisfied cultural visitors as it successfully captured their nonverbal visiting preferences and needs. Most of the participants stated that they agreed with the recommended POIs provided, whereas some adjusted their profiles.

In the future, we are planning to enhance the icon-based representation of visiting preferences with more multimedia elements, such as audio or video. Another interesting feature would be to reutilize past recommendations by recording them in order to feed them to future visitors with similar profiles. Finally, social media, gamification, and AR tools will be utilized in order to boost the visitor's motivation to visit a cultural destination, further improving the usability and effectiveness of ACUX-R.

**Author Contributions:** Conceptualization, M.K.; methodology, M.K. and Y.C.; software, J.A.; validation, M.K. and J.A.; formal analysis, M.K. and J.A.; investigation, M.K.; resources, M.K. and J.A.; data curation, M.K., Y.C. and J.A.; writing—original draft preparation, M.K.; writing—review and editing, M.K., Y.C. and G.C.; visualisation, M.K. and J.A.; supervision, G.C.; project administration, M.K.; funding acquisition, M.K. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was co-financed by the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship, and Innovation, under the call RESEARCH CREATE INNOVATE (project name: CAnTi, project code: MIS—5056234).

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A. UEQ Questionnaire**

Please fill out the following questionnaire for the assessment of the ACUX-R. The questionnaire consists of pairs of contrasting attributes that may apply to the product. The circles between the attributes represent gradations between the opposites. You can express your agreement with the attributes by ticking the circle that most closely reflects your impression.

For example:


This response would mean that you rate the application as more attractive than unattractive.

Please decide spontaneously. Don't think too long about your decision to make sure that you convey your original impression. Sometimes you may not be completely sure about your agreement with a particular attribute, or you may find that the attribute does not apply completely to the particular product. Nevertheless, please tick a circle in every line.

It is your personal opinion that counts. Please remember: there is no wrong or right answer!

Please assess the product now by ticking one circle per line.


#### **References**

1. Alexandridis, G.; Chrysanthi, A.; Tsekouras, G.E.; Caridakis, G. Personalized and Content Adaptive Cultural Heritage Path Recommendation: An Application to the Gournia and Çatalhöyük Archaeological Sites. *User Model. User Adapt. Interact.* **2019**, *29*, 201–238. [CrossRef]

