Next Article in Journal
Multi-Criteria Energy Management with Preference Induced Load Scheduling Using Grey Wolf Optimizer
Previous Article in Journal
Simplified Model Predictive Current Control of Four-Level Nested Neutral Point Clamped Converter
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sustainability
Volume 15
Issue 2
10.3390/su15020956
sustainability-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Studying Users’ Perceptions of COVID-19 Mobile Applications in Saudi Arabia

by
Mashael Alghareeb
1,
Abdulmohsen Saud Albesher
1,* and
Amna Asif
2,*
1
The Department of Information Systems, College of Computer Sciences and Information Technology, King Faisal University, Hofuf 31982, Saudi Arabia
2
School of Computing and Communications, Lancaster University Leipzig, 04109 Leipzig, Germany
*
Authors to whom correspondence should be addressed.
Sustainability 2023, 15(2), 956; https://doi.org/10.3390/su15020956
Submission received: 1 December 2022 / Revised: 29 December 2022 / Accepted: 30 December 2022 / Published: 4 January 2023
(This article belongs to the Special Issue Service Operations During COVID-19)
Browse Figures
Versions Notes

Abstract

:
In Saudi Arabia, several mitigating measures were implemented in response to the COVID-19 pandemic, including the creation of COVID-19 mobile applications (apps) for public use. The Saudi government has made the use of these apps mandatory for its citizens and residents. However, it is essential to explore the perception that common users have regarding using these apps in terms of usability and user experience. Therefore, this paper assesses user experience in terms of effectiveness, efficiency, and user satisfaction with the usability of the Saudi COVID-19 apps. The reviews of five mobile apps launched by the Saudi Data and AI Authority (SDAIA) and the Ministry of Health in the Apple Store were extracted using an online tool and analyzed using the content analysis method. The number of collected reviews was 29 for Sehha, 406 for Sehhaty, 442 for Mawid, 107 for Tabaud, and 1338 for Tawakkalna. The results of the study showed that Mawid (82%) and Tabaud (81%) had the highest usability of all the apps studied. Sehha ( − 138%) and Sehhaty (−107%) received the lowest usability scores, followed by Tawakkalna ( − 22%). Based on these results, we identified several usability issues with each app. Some of the main problems reported by users were increased battery drain, lack of privacy, and technical issues.

1. Introduction

The Saudi government has expanded the country’s telemedicine services by incorporating innovative technology into its health care. There are many applications that facilitate citizens’ access to health services [1,2]. Specifically, in 2018, the Sehha e-health app was released, allowing users to have visual medical consultations with their doctors on their smartphones [3]. The Saudi government also introduced a number of medical applications before COVID-19, such as Mawid and Sehhaty [4]. Later, during the spread of the novel coronavirus disease (COVID-19), the government released more applications, such as Tawakkalna and Tabaud, to support and connect with citizens (Saudi nationality holders) and residents (people allowed to live in the country on visa status) [4]. The use of these mobile applications was made mandatory for citizens and residents during the COVID-19 pandemic. The government actively updated these apps to make them more usable and adaptable to diverse users.
Previously published research has investigated different aspects of telemedicine. One recent study [5] summarized the digital technologies used during the COVID-19 pandemic, while others [6,7] focused on the barriers related to telemedicine technologies in Saudi Arabia. Some studies investigated the usage of mobile apps for certain diseases in Saudi Arabia [8], especially during the pandemic [9]. In particular, the focus has been on the COVID-19 apps used by all people in Saudi Arabia [5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]. Alghamdi et al. conducted a literature review to examine the user experience of using these apps by analyzing published data, press briefings, and announcements made by the Ministry of Health in Saudi Arabia [10]. Binkheder et al. looked at the public perception of using these apps by applying sentiment analysis to samples from Twitter [4]. Almufarij and Alharbi evaluated users’ awareness, use, and perceptions of these apps through online questionnaires [11]. Alharbi et al. used the mobile application rating scale (MARS) to evaluate the overall quality, satisfaction, engagement, functionality, aesthetics, and information of six COVID-19 apps being used in Saudi Arabia named Aseafni, Mawid, Sehha, Sehhaty, Tatmman, and Tawakkalna [12]. Some studies evaluated the patient experience with only one app (i.e., Maid) [13,14]. The usability of various COVID-19 apps used in Saudi Arabia was also measured through an expert review [15,16]. Many research papers [17,18,19,20] investigated the acceptance of such apps by end users. Although user experience is a significant factor in users’ satisfaction with any application [21], users’ experiences and the apps’ usability were not investigated explicitly using data from standard users of these apps. Mobile apps that have problems with user satisfaction fail or lose popularity among people [22]. By contrast, mobile apps that provide satisfactory experiences have positive effects on their continuous use [23]. Yu and Huang indicated that improving user experience satisfaction with mobile health (m-health) applications increases people’s desire to use them [24].
In the ecosystem of mobile apps, user reviews contain a wealth of information about user experiences and expectations [25]. Additionally, app stores allow users to search for, purchase, and install mobile apps and provide feedback through reviews and ratings. A review may contain users’ application experiences, opinions, feature requests, and bug reports [26]. In addition, user reviews represent an essential factor in accepting the app [27]. Reviews are also valuable for users who want to know what others think and for developers and mobile app development companies who can use this information to better understand their audience [22]. Positive comments add value to the developers, while negative comments add more value to the user community. Although negative reviews are detrimental to the adoption of the app, they may help developers refocus their subsequent efforts and priorities [28].
In this study, we aimed to identify and explore the usability of and user experiences with the Tawakkalna, Tabaud, Sehhaty, Sehha, and Mawid apps, all of which were launched by the Saudi government. Henceforth, we will refer to these apps as the “Saudi COVID-19 apps.” In our approach, we first extracted the application users’ online reviews [29,30] and stored them in an Excel file. The users’ reviews are posted in both English and Arabic on the app store. Then, we exported them to the qualitative data management software MAXQDA for analysis [31,32,33] (Springer: Basel, Switzerland). We used the content analysis method [34,35,36] to explore the collected reviews. We preprocessed the data and excluded reviews that were not related to the app user’s experience or the app usability. Next, we analyzed the collected qualitative data [37] by applying quantitative measures of frequency of occurrence and usability scores to answer the following question:
What are the perceptions of standard users (citizens and residents) concerning their experiences with and the usability of the Saudi COVID-19 apps?
To answer this question, we empirically evaluated the Saudi COVID-19 apps using real data and proposed a series of recommendations to improve usability, with the following objectives:
  • To collect real-life data from sources such as reviews related to users’ experiences using the Saudi COVID-19 apps instead of surveys or questionnaires, as reviews are a useful source to obtain feedback from users of mobile apps.
  • To process qualitative data from user reviews to further investigate users’ experiences regarding application usage and to quantitatively analyze the data to provide objective information about usability.
  • To analyze the qualitative data based on various factors of user experience and usability to explore both positive and negative issues. The results could be helpful for the provider to consider while developing similar applications.
  • To present the user experience and provide usability recommendations for improving these mobile apps based on our findings, observations, and analysis.
The remainder of this paper is organized as follows. Section 1.1 introduces m-health in Saudi Arabia and then presents the tested mobile apps. In Section 1.2, we clarify the differences between usability and user experience. Section 2 describes previous work related to this study. In Section 3, we explain the study’s methodology, while in Section 4, we report the results of our research. A discussion and recommendations are provided in Section 5, and we close this paper with the conclusion in Section 6.

1.1. M-Health in Saudi Arabia

The collection of public services that are delivered to citizens and firms through mobile technology is referred to as the Mobile Government (m-Government) [38]. This allows quick access to government services, information, and assistance for citizens, businesses, and civil servants through wireless devices [39]. The development of mobile communication devices, such as smartphones and tablets, has sparked rapid growth in m-health, as these devices offer the opportunity for dynamic patient and healthcare provider engagement and are seen as a new way to improve health outcomes [40]. The World Health Organization’s (WHO’s) global survey of 114 countries found that many countries have established m-health initiatives. The Global Observatory of Electronic Health (GOe) has defined m-health as a medical and public health practice supported by portable devices, such as mobile phones, patient monitors, personal digital assistants (PDAs), and other wireless devices [41]. In other words, m-health is a new approach to e-health that builds on remarkable advances in mobile technology and apps to help provide health care, such as through the use of voice calls, short message service (SMS), and wireless data transmission [42,43]. Malaysia, Thailand, Singapore, and India have adjusted their healthcare systems to encourage professionals to use technology-based solutions to address health issues [44,45].
Saudi Arabia has been using mobile technologies in the past few years, especially with the encouragement of Vision 2030 and the country’s massive digital transformation. Saudi Arabia’s information and communications technology (ICT) market is the biggest in the Middle East and North African region, with a worth of over USD 32.1 billion. Saudi Arabia plans to increase the size of its ICT and emerging technology markets by 50% and create more than 25,000 jobs, thus raising the contribution of the ICT sector to the country’s GDP by USD 13.3 billion [46]. Therefore, the Saudi government is working toward the idea of a society based on knowledge, speed of response, and interaction, considering the importance of the time factor [47]. Consequently, this has facilitated everything necessary to enable the provision of m-Government services to the target audience from anywhere and at any time via smart devices (mobile phones, laptops, assistive apps, etc.). Saudi Arabia is one of the countries that succeeded in optimizing and maintaining a strategy to mitigate the spread of the COVID-19 virus using m-health technologies [10]. The Saudi government is using the m-Government to implement citizen–government communication on a massive scale. Overall, 60% of the health services provided in Saudi Arabia are distributed by the Ministry of Health, and the remaining 40% are administered by other government and private facilities [48].

COVID-19 Mobile Applications in Saudi Arabia

A mobile app is a software program designed specifically for small wireless computing devices, such as smartphones and tablets, as opposed to desktop or laptop PCs [49]. According to Statista, the number of worldwide smartphone subscriptions exceeds 6 billion [50], and mobile devices account for almost 60% of web traffic [51]. In addition, mobile devices have various uses, such as calling, messaging, web browsing, video watching, entertainment, etc. [52]. Thus, users can download and install mobile apps from online stores, such as the Apple App Store, and update them either manually or automatically. The app store is an interface for exploring and downloading apps [53]. Moreover, the download page includes information about the program file, such as the title, developer, average rating, description, comparable apps, upload history, current user reviews, screenshots, video previews, and a download link. As a result, these stores provide a convenient and efficient way for users to download or purchase apps and allow developers to receive feedback from users about their experiences [54].
The Saudi government relied on five mobile apps during the COVID-19 pandemic: Tawakkalna, Tabaud, Sehhaty, Sehha, and Mawid. Tawakkalna was first launched to show a person’s COVID-19 status (active or inactive), but later, it was expanded to provide more services, such as permits, vaccination status, health passports, and other personal documents. Tabaud mainly uses Bluetooth technology to inform users if they were recently in contact with a COVID-19 patient. The Sehhaty app is useful for making vaccine or medical counseling appointments, obtaining vaccination status reports, and searching for medicine and the nearest pharmacies. Sehha allows users to request lab and test reports and pay for them. Mawid permits users to find, book, cancel, and reschedule research health center and hospital appointments [55]. Figure 1 shows the logos of the tested apps as they appear in the Apple Store.

1.2. Usability and User Experience

The difference between usability and user experience is more complicated than can be covered in a one-paragraph summary. However, some of the related findings are discussed in this section. An important point is that there is no specific definition of user experience [56]. Additionally, measuring user experience is challenging [57], and claiming that one is a subset of the other is debatable. Evaluators measure usability to design a better user experience [58,59]. Some scholars believe that usability is mainly about effectiveness and efficiency [60,61], while others rely on the ISO 9241-11 usability standard, which grants usability a broader meaning by looking at it as “good user performance and user experience” [59] (p. 16). However, user experience can be seen as a separate concept that is attached only to emotional and hedonic aspects [62]. Ardito et al. believe that experience is affected by the characteristics of the examined system, the user’s psychological state, and the interaction’s context [63]. In this context, we are merging the concepts of usability and user experience, since efficiency and effectiveness are more closely related to usability, while satisfaction is more logically related to user experience. Nevertheless, user experience and satisfaction are subjective and abstract constructs [64].
Usability is one of the most important factors that must be considered in m-health apps. Studies conducted on usability found that an SMS alert system was effective for users and made the applications very easy to use. The ability to learn how to use the application, determine its reliability, and satisfaction were all accounted for. The application of usability heuristics of consistency, reversal of actions, error prevention and simple error handling, reducing short-term memory load, design for multiple and dynamic contexts, design for small devices, design for limited and split attention, design for speed and recovery, building a quality mobile application, design for “top-down” interaction, allowing for personalization, design for enjoyment, and various annotation and simplification tools can make it easier for people to use the mobile apps [65].

2. Related Work

2.1. Studies on User Experiences

In the case of maternal and infant health apps, Biviji et al. aimed to assess users’ self-reported experiences, perceived benefits, and general feedback by analyzing publicly available user reviews on two popular app stores: Apple’s App Store and the Google Play Store [27]. A general inductive qualitative content analysis approach was used to code and analyze the reviews. The results could be grouped by three main topics: application functionality, technical aspects, and content. Six secondary topics included usage patterns, social support, application costs, application comparisons, health-care assistance, and customer service support. Based on the results, users tended to rate low-cost and preferably free apps as having high-quality content, superior features, improved technical aspects, and user-friendly interfaces. Users also found the response of the application developer essential, since it granted them the opportunity to participate in the development and delivery process.
The Saudi government has supported improving the new mode of interaction between the government and its residents. Indeed, Vision 2030 encourages government institutions to provide unique interactive and online ways for citizens to interact with them and engage through mobile apps available in the app store [47]. Additionally, in 2021, a platform was launched for the Digital Government Authority of Saudi Arabia, which contributes to improving the beneficiaries’ experience of reaching a proactive digital government through an initiative that achieves the goals of the Kingdom’s ambitious Vision 2030 [66].

2.2. Studies on the Efforts of SDAIA and the Ministry of Health in the Kingdom of Saudi Arabia

Among many key players in the ICT sector of Saudi Arabia, the Saudi Data and AI Authority (SDAIA) is responsible for countries’ national data and AI agendas [46]. Alanzi et al. proposed an evaluation of the Mawid mobile app developed by the Ministry of Health in the Kingdom of Saudi Arabia [14]. This app manages primary care hospital appointments and tracks COVID-19 cases. The authors used an online survey to collect data based on three key factors: ease of use, satisfaction, and the benefits that the app provides its users. There were 2542 participants, 345 of whom completed only part of the questionnaire, and 204 did not use the application. After removing these 549 irrelevant reviews, the data analysis was performed on a final sample of 1993. The results showed that 82.1% of the participants described the app as easy to use, while 79.8% expressed great satisfaction with the app. The t-test results revealed significant differences between males and females and between young and older participants in the ease of use and satisfaction levels associated with the Mawid application.
In addition, AlGothami and Saeed performed a cross-sectional experiment to evaluate and measure user experience when using the Tawakkalna mobile app in a nonexperimental manner [20]. To this end, the authors collected a questionnaire (survey) from targeted users across the Kingdom of Saudi Arabia. An online user experience questionnaire (UEQ) was administered to 87 participants (75 participants had previous experience with Tawakkalna) from Saudi Arabia, with an Arabic and an English version of the UEQ. The results showed that the app was attractive, with a score of 1.51 > 0.8; the pragmatic quality aspects (clarity, efficiency, and reliability) were good, with a score of 1.55 > 0.8; and the quality aspects (novelty and motivation) were also good, with a score of 1.21 > 8. The results of comparing the evaluated product with normative data ranked the quality of the evaluated product as “above average” for attractiveness, outlook, and novelty and “good” for efficiency, reliability, and motivation.
Based on these results, recommendations were made to the mobile app developers to improve the appearance of the app’s interface, as most of the negative responses were related to the design. For example, Booday and Albesher conducted a study to evaluate and measure the usability of the Kingdom’s five COVID-19 apps [15]. The heuristic evaluation method was based on SMART heuristics. The results revealed many usability issues with each application. The problems were discussed in detail, one of them being the trade-off between designing many different, simple apps that focus on providing one service versus designing one comprehensive application that provides many different services. Recommendations were made for application designers to increase usability.
Aldekhyyel et al. aimed to evaluate three telemedicine applications used in the Kingdom of Saudi Arabia during COVID-19 using a heuristic evaluation method that focused on understanding usability issues in the application’s user interface [16]. Assessments were conducted in April 2020 and June 2021 to identify positive application features using Nielsen’s 10 Usability Inferences on a five-point severity rating scale, and redesign recommendations were made. The results revealed 54 user interface usability issues that may have an impact on the app’s overall usability. Accordingly, all three apps need to improve their user interface designs in order to enhance the overall user experience and ensure the continuity of these services after the pandemic.
Previous studies have used various methods to explore the usability of the Saudi COVID-19 health apps. However, none considered using users’ self-reported data. We deem self-reported data, such as user reviews, as a reliable source of information for investigating the usability of such apps.

3. Methodology

We followed a series of steps to answer our research question regarding users’ (citizens’ and residents’) perceptions of the Saudi COVID-19 apps. First, we explored the existing literature on similar studies and data availability. It is a common practice among app users to provide reviews to share their app usage experiences on various app provider platforms or social media. It is also possible to prompt users to rate an application on Google Play [67] or the Apple App Store. The reviews directly reported by app users are considered more reliable than those from other sources, such as social media, which are a source of rumors or disinformation [68]. The Saudi COVID-19 apps were specifically launched to manage the pandemic on a nonprofit basis. Therefore, we selected the application review data for analysis purposes. We considered choosing reviews in both English and Arabic. To extract the app reviews automatically from the app store, we used the Heedzy website tool.
As the extracted reviews were available in Arabic and English, we applied the content analysis method [36]. Content analysis was chosen because of its strengths in methodically categorizing and summarizing vast amounts of text-based data and its capacity to assist in identifying patterns in a text, with particular emphasis paid to the context from which sample data are collected [69]. Qualitative content analysis is described as a research strategy for interpreting the content of text data subjectively using a systematic classification procedure of coding and detecting themes or patterns [35]. Furthermore, qualitative content analysis provides the flexibility of using the inductive approach in data analysis and allows the extraction of the meaning of apparent and latent content. In general, the process of data analysis includes the following basic steps: selecting the unit of analysis, creating categories, and establishing themes [70]. We found that the qualitative data management software MAXQDA is a powerful and flexible tool for analyzing textual data in various formats [71]. Since our data consisted of app user reviews, MAXQDA was found to be suitable for handling such data. In the final step, we analyzed our data based on various user experiences and usability factors to explore users’ perceptions of the Saudi COVID-19 apps.
According to the International Organization for Standardization (ISO 9241-11), usability is the extent to which a product is effective and efficient to use and also depends on the degree of user satisfaction after its use [72]. Therefore, three polarity lexicons were constructed based on the selected usability factors: satisfaction, effectiveness, and efficiency. The reviews for each app were labeled based on usability factors. A polarity score was given for each factor, and we used a quantitative measure for how positive or negative each comment was. The average polarity score ranged from − 1 to + 1, with negative values indicating negative opinions, values of zero indicating neutral opinions, and positive values indicating positive opinions. The total usability score was calculated for each review based on the sum of each review’s satisfaction, effectiveness, and efficiency scores [73]. The primary predefined (usability measurement) factors were identified as follows:
  • Satisfaction, where users talked about their satisfaction with the application and about the positive or negative aspects of the app;
  • Effectiveness, where users discussed app features and functions and the positive and negative aspects of the application;
  • Efficiency, where users talked about the efficiency or performance of the application and about the positive or negative aspects of the app.

Data Collection and Analysis

In this study, we considered reviews as the primary source of data analysis. A qualitative analysis [69] of user reviews was conducted for five publicly available health apps in the app store from February 2021 to December 2021. The apps were identified by searching the Apple App Store using the terms “SDAIA” and “the Ministry of Health in Saudi Arabia.”
The user reviews for the Tawakkalna, Tabaud, Sehhaty, Sehha, and Mawid apps were extracted from the app store using the Heedzy website [74]. All the reviews were downloaded in Arabic and English, including the review text and application details. We extracted 1338 reviews of the Tawakkalna app, 107 reviews of the Tabaud app, 406 reviews of the Sehhaty app, 29 reviews of the Sehha app, and 442 reviews of the Mawid app. The user reviews were subsequently exported to program members and then imported into the qualitative data [75,76] management software MAXQDA for analysis [31,33]. Furthermore, the reviews that did not focus on usability were excluded. Since we wanted to measure the apps’ usability, we focused on positive and negative reviews that highlighted usability-related issues, as shown in Table 1. Overall, the Tawakkalna app received the most reviews; however, the negative reviews (510) outnumbered the positive reviews (466). Mawid was ranked second in terms of the total number of reviews, with 369 positive reviews and only 37 negative reviews; Sehhaty stands third in terms of the total number of reviews, with 174 negative and 84 positive reviews. Finally, Tabaud had 83 positive and 4 negative reviews, and Sehha had 6 positive and 18 negative reviews.
Figure 2 presents statistics related to the users’ reviews of Tawakkalna, Tabaud, Sehhaty, Sehha, and Mawid. Panel (a) summarizes the total reviews collected from all the apps and shows that the highest number of reviews was received by the Tawakkalna app. The Mawid app received the highest percentage of positive reviews, as shown in panel (b), while the Sehhaty app received the highest percentage of negative reviews, as shown in panel (c).

4. Results

Table 2 presents the frequency of positive and negative reviews received by the Saudi COVID-19 apps. From the data, we found that there were more negative than positive reviews related to usability factors.
Table 3 shows the five applications rated based on the three factors—satisfaction, effectiveness, and efficiency—and their respective percentages of total usability. We summed up all the positive and negative percentages of usability to calculate the total usability of all of the Saudi COVID-19 apps.
Figure 3 shows the total usability of the five m-health apps. The chart shows that Tabaud and Mawid received positive usability scores, and Tawakkalna is in third place, as it has significantly more reviews than the other apps (see Table 2).
An Excel spreadsheet was used to determine the sum of the positive and negative usability of the five apps based on the predefined factors of usability. The percentage of positive reviews and the percentage of negative reviews were summed. Then, all of the resulting positive and negative usability values were summed, yielding the total usability score for each application. For example, the total usability of the Tawakkalna application was found to be − 22%. Figure 3 shows that Mawid had the highest usability percentage (82%), followed by Tabaud (81%). Sehha ( − 138%) and Sehhaty ( − 107%) had the lowest usability scores, followed by Tawakkalna ( − 22%).
First, regarding the satisfaction factor, it was found that Mawid (85%) and Tabaud (84%) had the most positive scores, with Tawakkalna (43%) in third place, while Sehha and Sehhaty both had the lowest positive score (23%). Users gave Sehha the most negative score ( − 69%), followed by Sehhaty ( − 48%) and Tawakkalna ( − 47%). Second, concerning effectiveness, Tabaud had 10% usability and Mawid had 2% usability, whereas the other applications scored 0%. The highest negative effectiveness scores were for Sehhaty ( − 68%) and Sehha ( − 54%), followed by Tawakkalna ( − 14%), Tabaud ( − 9%), and Mawid ( − 6%). Third, the efficiency percentage in terms of positive usability was highest for Mawid (6%) and Tawakkalna (1%), with the other applications scoring 0%. The highest negative percentage was attributed to Sehha ( − 83%), followed by Sehhaty ( − 14%). Tawakkalna ( − 5%) and Mawid ( − 2%) received fewer negative ratings for this category. The total usability score for all of the Saudi COVID-19 apps was − 104%, as determined based on 2032 reviews.
Figure 4 presents the word clouds of the user reviews for each selected app separately and together. The original reviews were a mixture of Arabic and English; the Arabic reviews were translated into English by an Arabic native speaker. The word cloud was generated based on the number of occurrences of specific words in the text. In a word cloud, the most frequent word is represented in the largest size; however, the color coding does not mean anything. Figure 3 and Figure 4 help us evaluate the user experience of COVID-19 apps based on the user review data.

5. Discussion and Recommendations

The Saudi COVID-19 apps were launched through SDAIA’s and the Ministry of Health’s efforts during the onset of the pandemic to facilitate the government–citizen/resident interaction and provide the latter with better support. The apps’ users are therefore literate or illiterate, have special needs, and are either Arab or non-Arab. Therefore, it was challenging to develop an application that was accessible to all types of users. Tabaud, Sehhaty, Sehha, and Mawid were used for various purposes, such as doctor visits and vaccinations. However, to make these applications available, usable, and accessible, the developers needed to determine their acceptance and usability from the users’ points of view. Therefore, our study investigated users’ perceptions of the usability and user experience of these apps. We found that Tabaud and Mawid are the most usable applications based on users’ reviews; these apps help users remain aware of the rate of infection and help them book doctors’ appointments. Our quantitative findings revealed the negative user experience with the five SDAIA and Ministry of Health apps, with the exception of Tabaud and Mawid. The largest number of reviews was retrieved for Tawakkalna. Although the user experience for this app was slightly negative, its interface is considered usable, with 43% satisfaction, which is encouraging in the field. Figure 4 presents the word cloud of users’ reviews of the Saudi COVID-19 apps. Each word cloud consists of 50 words in total; the mixture of positive and negative words provides us with an idea of the user experience for the users of these apps. Given the growing number of smartphone users in Saudi Arabia, mobile apps are easily accessible to all types of users. The results indicate that users show a positive attitude toward the growing ICT market in Saudi Arabia.
The word clouds presented in Figure 4 provide us with an overview of the user experience with each COVID-19 app. It is clear that the users preferred the Mawid app among all the selected apps. The main purpose of Mawid is to help users book hospital appointments. Tabaud and Sehha come in second place in terms of user experience, as users liked the apps but complained about technical problems, such as the application stopping and errors. It seems that users were not so happy with Tawakkalna and Sehhaty. They complained that the apps run in the background all the time and consume a lot of their smartphone battery. In addition, Sehhaty has the problem of updating too often. As such problems that discourage users from using these apps cannot be ignored, designers must consider usability factors to increase and maintain application interactivity. For example, the main problems are related to technical issues (technical failures), usability, privacy, or requirements related to the functionalities of apps reported by some users. Booday and Albesher [15] found similar results when they evaluated these applications in order to discover usability problems using a heuristic approach.
Most of the issues with the apps were described in general terms, not specific ones. In the Tawakkalna app, there was a specific issue that generated users’ dissatisfaction: users were forced to keep the app always activated and did not have other options, which led to increased battery consumption. Since the application tracks mobile users periodically throughout the user experience, location services need to be active all the time. Similar results were reported for nine European m-health apps [77]. Another source of dissatisfaction was privacy issues. Privacy concerns were raised regarding the location always being activated, specifically pertaining to Tawakkalna; this is similar to what has been reported for other m-health apps [78]. According to Choudhury [79], other countries have explored and used different data collection methods that are more sensitive to privacy issues. Singapore was the first to develop a well-designed contact-tracing app that was shared with the rest of the world by making the development code open source. This helped other countries roll out similar apps at a faster rate. One example is the Tabaud app, which relies on Bluetooth technology to collect location and interaction data to track the movement patterns of quarantined individuals and send deidentified data to individuals who may have had physical interactions with those with confirmed cases of COVID-19. Based on the results of another study [80], public health officials and app developers recommend that four key lessons be learned when developing mobile apps: they should aid current responses to COVID-19, inform users, protect users’ personal information, and adapt to users’ environments. The primary concerns have been maintaining a balance between the collection and use of personal data and protecting this information. Therefore, health apps should be transparent about their privacy policies, which should be provided in plain language. This assures users that their data are protected and increases their confidence in the application. Regarding privacy issues in the Tabaud and Tawakkalna apps, only one complaint was registered for Tawakkalna, but none for Tabaud. In Tabaud, users did not complain about privacy issues because before using the application for the first time, a pop-up message appeared describing the idea behind it, how to use it, and clarifying the privacy policy.
Several user reviews were related to technical issues with the apps. Some users mentioned that not receiving any responses to their reported problems or questions was very frustrating. This negatively affects user service. For example, all five apps use GPS technologies to provide some of their services. In the case of Sehhaty, users have heavily criticized the app because it does not detect health center locations correctly and effectively. It was also noted that the application did not provide instructions about reaching the chosen health center. In other words, the application does not take the user to Google Maps to provide them with the necessary location and directions when needed. Another example regarding Sehhaty and Sehha was that the users complained about the apps not being compatible with all iPhone versions. One solution for technical assistance and support is a chatbot: a computer program designed to simulate conversation with users, especially over the Internet. According to Ahmed et al. [81], a website integrated with a healthcare chatbot may help solve this problem. As previously reported [82], all apps that made use of chatbots were found to be of high quality and highly popular.
Several suggestions were repeated by multiple users and may be valuable to developers and designers. However, the applications do not provide a space dedicated for user suggestions. One recurrent recommendation was that designers should make use of widgets for a better experience. The widget is a type of user interface object that is used to display specific information that resides in an associated application, and it shortens the time and effort to open the application by directly showing the information the user needs [83]. One example is the health status being shown in Tawakkalna without needing to open the application. Another suggestion was that designers should take advantage of wearable iPhone devices, such as the Apple Watch, which could improve the ease of use. The COVIDSafe application developed by the Australian Government’s Department of Health [84] and the Saudi COVID-19 apps have many similarities and differences. For example, users can book a vaccine appointment using the Tawakkalna and Sehhaty apps, or they can book general appointments in Mawid and Sehhaty. Instead of distributing medical services across multiple applications, designers should implement one unified application that provides all medical care services in an organized and well-categorized manner to facilitate and improve patient care [85].

6. Limitations and Future Research

This study has some limitations. The user reviews of these apps published in Google Play and in the Huawei Store were not analyzed. In future studies, we will consider increasing the number of reviews by including the data from these two app stores. In addition, the data are not balanced, considering the significantly higher number of reviews of Tawakkalna compared to the other apps. However, Tawakkalna was one of the mandatory apps used by citizens and residents. Therefore, the number of users is significantly higher than that of the other apps. Moreover, although the methodology we used proved powerful in evaluating some usability factors, such as efficiency, effectiveness, and satisfaction, other factors, such as learnability, need to be assessed using other methods, such as laboratory usability experiments. Thus, another study is highly needed to measure additional factors using different usability methods.
In a future study, we will explore the meaning of negative and positive user experiences. Using the interview or focus group method, we will try to answer the question: “What are the reasons for negative or positive evaluations among users, whether in terms of satisfaction, efficiency, or effectiveness?” Furthermore, we are planning to contact application providers to provide us with the demographic information of the users, such as age, gender, education, occupation, income, and geographic location. This information can help us explore the effect of users’ profiles on the user experience of these mobile apps and answer questions such as “What are the differences in the user experience of literate and illiterate users or male and female users?”

7. Conclusions

Digitalizing government communication is a topic of significance. The efficient flow of information between the government and citizens/residents has proved to be an important factor in fighting crises and emergencies. During the pandemic, the medical sector was one of the most critical sectors that needed to improve how services were provided to individuals. This study aimed to evaluate the user experience of the SDAIA and Ministry of Health apps in the Kingdom of Saudi Arabia in terms of effectiveness, efficiency, and user satisfaction. Five Saudi COVID-19 apps published in the Apple Store—Sehha, Sehhaty, Mawid, Tabaud, and Tawakkalna—were evaluated using the content analysis method. The results of the study showed that the highest usability percentage was obtained by Mawid (82%) and Tabaud (81%), with Sehha ( − 138%) and Sehhaty ( − 107%) receiving the lowest usability scores, followed by Tawakkalna ( − 22%). Further analysis of users’ reviews revealed that the most frequently used words were positive, which reflects the user experience with the usage of the COVID-19 apps. These findings are promising for identifying positive users’ behavior toward the usage of such applications. This is especially important, as Saudi Arabia plans on developing its ICT market, which is endorsed by the citizens and residents of the country. For these reasons, these applications need to be improved in terms of usability. Finally, valuable recommendations have been reported from users’ reviews data to the app designers to improve usability.
The present work complements a previous study that evaluated all five applications through an expert review. It was necessary to look at the users’ perceptions of these apps. It has become important to investigate the expectations of the developers of these applications so that scientific research can help provide useful recommendations for developing these apps to achieve a better user experience. Our findings should be helpful to the users of the apps, the developers of the apps, and the decision makers at SDAIA and the Ministry of Health in terms of understanding users’ perceptions of the apps’ usability. Furthermore, our results can be helpful for all countries that are considering taking initiatives in the e-government sector, especially in the health sector. Finally, our study design should help researchers test the usability of mobile applications in other sectors and countries.

Author Contributions

Conceptualization, M.A., A.S.A. and A.A.; methodology, M.A., A.S.A. and A.A.; validation, M.A.; formal analysis, M.A. and A.A; investigation, M.A., A.S.A. and A.A.; resources, M.A., A.S.A. and A.A.; data curation, M.A. and A.A; writing––original draft preparation, M.A.; writing––review and editing, A.S.A. and A.A.; visualization, M.A. and A.A.; supervision, A.S.A. and A.A.; project administration, A.S.A.; funding acquisition, A.S.A. All authors have read and agreed to the published version of the manuscript.

Funding

This research was supported by the Deanship of Scientific Research, Vice Presidency for Graduate Studies and Scientific Research, King Faisal University, Saudi Arabia (Grant No. 2453).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Acknowledgments

The authors thank the Deanship of Scientific Research, Vice Presidency for Graduate Studies and Scientific Research, King Faisal University, who kindly sponsored this research.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Saudi Arabia National Portal. Government Mobile Applications. Available online: https://www.my.gov.sa/wps/portal/snp/content/mobileGovernment (accessed on 20 October 2021).
  2. Ministry of Health Saudi Arabia. MOH Apps for Smartphones. Available online: https://www.moh.gov.sa/en/Support/Pages/MobileApp.aspx (accessed on 20 October 2021).
  3. Alharbi, A.; Alzuwaed, J.; Qasem, H. Evaluation of E-Health (Seha) application: A cross-sectional study in Saudi Arabia. BMC Med. Inform. Decis. Mak. 2021, 21, 1–9. [Google Scholar] [CrossRef]
  4. Binkheder, S.; Aldekhyyel, R.N.; AlMogbel, A.; Al-Twairesh, N.; Alhumaid, N.; Aldekhyyel, S.N.; Jamal, A.A. Public perceptions around mhealth applications during COVID-19 pandemic: A network and sentiment analysis of tweets in Saudi Arabia. Int. J. Environ. Res. Public Health 2021, 18, 13388. [Google Scholar] [CrossRef]
  5. Alghamdi, N.S.; Alghamdi, S.M. The role of digital technology in curbing COVID-19. Int. J. Environ. Res. Public Health 2022, 19, 8287. [Google Scholar] [CrossRef]
  6. Al-Hazmi, A.M.; Sheerah, H.A.; Arafa, A. Perspectives on telemedicine during the era of COVID-19; What can Saudi Arabia do? Int. J. Environ. Res. Public Health 2021, 18, 10617. [Google Scholar] [CrossRef]
  7. Alghamdi, S.M.; Aldhahir, A.M.; Alqahtani, J.S.; Siraj, R.A.; Alsulayyim, A.S.; Almojaibel, A.A.; Alhotye, M.; Alanazi, A.M.; Alqarni, A.A. Healthcare providers’ perception and barriers concerning the use of telehealth applications in Saudi Arabia: A cross-sectional study. Healthcare 2022, 10, 1527. [Google Scholar] [CrossRef]
  8. Alzahrani, A.; Gay, V.; Alturki, R. Exploring Saudi individuals’ perspectives and needs to design a hypertension management mobile technology solution: Qualitative study. Int. J. Environ. Res. Public Health 2022, 19, 12956. [Google Scholar] [CrossRef]
  9. Hussein, E.S.E.; Al-Shenqiti, A.M.; Ramadan, R.M.E.-S. Applications of medical digital technologies for noncommunicable diseases for follow-up during the COVID-19 pandemic. Int. J. Environ. Res. Public Health 2022, 19, 12682. [Google Scholar] [CrossRef]
  10. Alghamdi, S.M.; Alsulayyim, A.S.; Alqahtani, J.S.; Aldhahir, A.M. Digital health platforms in Saudi Arabia: Determinants from the COVID-19 pandemic experience. Healthcare 2021, 9, 1517. [Google Scholar] [CrossRef]
  11. Almufarij, A.; Alharbi, A. Perceptions of using mobile health apps (mHealth) during COVID-19 pandemic in Saudi Arabia: A cross-sectional study. J. Health Inform. Dev. Ctries. 2022, 16, 1–16. [Google Scholar]
  12. Alharbi, N.S.; Alsubki, N.; Altamimi, S.R.; Alonazi, W.; Fahlevi, M. COVID-19 mobile apps in Saudi Arabia: Systematic identification, evaluation, and features assessment. Front. Public Health 2022, 10, 803677. [Google Scholar] [CrossRef]
  13. AlAli, E.; AL-Dossary, R.; Al-Rayes, S.; Al-Ansary, N.; Alshawan, D.; Almulla, S.; Alanezi, F.; Alakrawi, Z.; Alnaim, N.; Saraireh, L.; et al. Evaluation of the patient experience with the mawid app during the COVID-19 pandemic in Al Hassa, Saudi Arabia. Healthcare 2022, 10, 1008. [Google Scholar] [CrossRef] [PubMed]
  14. Alanzi, T.M.; Althumairi, A.; Aljaffary, A.; Alfayez, A.; Alsalman, D.; Alanezi, F.; Alhodaib, H.; AlShammari, M.M.; Al-Dossary, R.; Al-Rayes, S. Evaluation of the Mawid mobile healthcare application in delivering services during the COVID-19 pandemic in Saudi Arabia. Int. Health 2022, 14, 142–151. [Google Scholar] [CrossRef] [PubMed]
  15. Booday, M.; Albesher, A. Evaluating the Usability of Mobile Applications: The Case of COVID-19 Apps in Saudi Arabia. In Proceedings of the 22nd International Arab Conference on Information Technology (ACIT 2021), Muscat, Oman, 21–23 December 2021; pp. 1–7. [Google Scholar] [CrossRef]
  16. Aldekhyyel, R.N.; Almulhem, J.A.; Binkheder, S. Usability of telemedicine mobile applications during COVID-19 in Saudi Arabia: A heuristic evaluation of patient user interfaces. Healthcare 2021, 9, 1574. [Google Scholar] [CrossRef] [PubMed]
  17. Alsyouf, A.; Masa’deh, R.; Albugami, M.; Al-Bsheish, M.; Lutfi, A.; Alsubahi, N. Risk of fear and anxiety in utilising health app surveillance due to COVID-19: Gender differences analysis. Risks 2021, 9, 179. [Google Scholar] [CrossRef]
  18. Alharbi, N.S.; AlGhanmi, A.S.; Fahlevi, M. Adoption of health mobile apps during the COVID-19 lockdown: A health belief model approach. Int. J. Environ. Res. Public Health 2022, 19, 4179. [Google Scholar] [CrossRef]
  19. Alsyouf, A.; Lutfi, A.; Al-Bsheish, M.; Jarrar, M.; Al-Mugheed, K.; Almaiah, M.A.; Alhazmi, F.N.; Masa’deh, R.; Anshasi, R.J.; Ashour, A. Exposure detection applications acceptance: The case of COVID-19. Int. J. Environ. Res. Public Health 2022, 19, 7307. [Google Scholar] [CrossRef]
  20. AlGothami, S.S.; Saeed, S. Digital Transformation and Usability: User Acceptance of Tawakkalna Application during COVID-19 in Saudi Arabia. In Pandemic, Lockdown, and Digital Transformation; Bolivar, M.P.R., Thurasamy, R., Eds.; Springer International Publishing: Cham, Switzerland, 2021; pp. 95–109. [Google Scholar] [CrossRef]
  21. Myers, B.A. A brief history of human-computer interaction technology. Interactions 1998, 5, 44–54. [Google Scholar] [CrossRef] [Green Version]
  22. Feng, L.; Wei, W. An empirical study on user experience evaluation and identification of critical UX issues. Sustainability 2019, 11, 2432. [Google Scholar] [CrossRef] [Green Version]
  23. Kwon, J.-Y.; Lee, J.-S.; Park, T.-S. Analysis of strategies to increase user retention of fitness mobile apps during and after the COVID-19 pandemic. Int. J. Environ. Res. Public Health 2022, 19, 10814. [Google Scholar] [CrossRef]
  24. Yu, N.; Huang, Y.-T. Important factors affecting user experience design and satisfaction of a mobile health app—A case study of daily yoga app. Int. J. Environ. Res. Public Health 2020, 17, 6967. [Google Scholar] [CrossRef]
  25. Wang, H.; Li, H.; Guo, Y. Understanding the Evolution of Mobile App Ecosystems: A Longitudinal Measurement Study of Google Play. In Proceedings of the World Wide Web Conference, San Francisco, CA, USA, 13–17 May 2019; pp. 1988–1999. [Google Scholar] [CrossRef]
  26. Genc-Nayebi, N.; Abran, A. A Systematic literature review: Opinion mining studies from mobile app store user reviews. J. Syst. Softw. 2017, 125, 207–219. [Google Scholar] [CrossRef]
  27. Biviji, R.; Williams, K.S.; Vest, J.R.; Dixon, B.E.; Cullen, T.; Harle, C.A. Consumer perspectives on maternal and infant health apps: Qualitative content analysis. J. Med. Internet Res. 2021, 23, e27403. [Google Scholar] [CrossRef] [PubMed]
  28. Hoon, L.; Vasa, R.; Schneider, J.-G.; Mouzakis, K. A Preliminary Analysis of Vocabulary in Mobile App User Reviews. In Proceedings of the 24th Australian Computer-Human Interaction Conference, Melbourne, VI, Australia, 26–30 November 2012; pp. 245–248. [Google Scholar] [CrossRef]
  29. Merdenyan, B.; Petrie, H. User Reviews of Gamepad Controllers: A Source of User Requirements and User Experience. In Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play, London, UK, 3–7 October 2015; pp. 643–648. [Google Scholar] [CrossRef]
  30. Hedegaard, S.; Simonsen, J.G. Extracting Usability and User Experience Information from Online User Reviews. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France, 27 April–2 May 2013; pp. 2089–2098. [Google Scholar] [CrossRef]
  31. All-in-One Qualitative & Mixed Methods Data Analysis Tool. MAXQDA. Available online: https://www.maxqda.com/ (accessed on 20 January 2022).
  32. Marjaei, S.; Yazdi, F.A.; Chandrashekara, M. MAXQDA and its Application to LIS Research. Libr. Philos. Pract. 2019, 1–9. [Google Scholar]
  33. Nicholas, J.; Fogarty, A.S.; Boydell, K.; Christensen, H. The reviews are in: A qualitative content analysis of consumer perspectives on apps for bipolar disorder. J. Med. Internet Res. 2017, 19, e105. [Google Scholar] [CrossRef]
  34. Phillips, C.; Klarkowski, M.; Frommel, J.; Gutwin, C.; Mandryk, R.L. Identifying commercial games with therapeutic potential through a content analysis of steam reviews. Proc. ACM Hum.-Comput. Interact. 2021, 5, 1–21. [Google Scholar] [CrossRef]
  35. Hsieh, H.-F.; Shannon, S.E. Three approaches to qualitative content analysis. Qual. Health Res. 2005, 15, 1277–1288. [Google Scholar] [CrossRef]
  36. Harwood, T.G.; Garry, T. An overview of content analysis. TMR 2003, 3, 479–498. [Google Scholar] [CrossRef]
  37. Hahn, C. Introduction, Coding Terminology, and the Big Picture. In Doing Qualitative Research Using Your Computer; SAGE Publications, Ltd.: Thousand Oaks, CA, USA, 2008; pp. 2–17. [Google Scholar] [CrossRef]
  38. Albesher, A.S.; Stone, R.T. Current state of M-government research: Identifying future research opportunities. Int. J. Electron. Gov. 2016, 8, 119–139. [Google Scholar] [CrossRef]
  39. Lee, M.; Yun, J.J.; Pyka, A.; Won, D.; Kodama, F.; Schiuma, G.; Park, H.; Jeon, J.; Park, K.; Jung, K.; et al. How to respond to the fourth industrial revolution, or the second information technology revolution? Dynamic new combinations between technology, market, and society through open innovation. J. Open Innov. Technol. Mark. Complex. 2018, 4, 21. [Google Scholar] [CrossRef] [Green Version]
  40. Sama, P.R.; Eapen, Z.J.; Weinfurt, K.P.; Shah, B.R.; Schulman, K.A. An evaluation of mobile health application tools. JMIR MHealth UHealth 2014, 2, e3088. [Google Scholar] [CrossRef]
  41. Kay, M.; Santos, J.; Takane, M. mHealth: New Horizons for Health through Mobile Technologies; World Health Organization: Geneva, Switzerland, 2011; pp. 66–71. [Google Scholar]
  42. Babirye, D.; Shete, P.B.; Farr, K.; Nalugwa, T.; Ojok, C.; Nantale, M.; Oyuku, D.; Ayakaka, I.; Katamba, A.; Davis, J.L. Feasibility of a short message service (SMS) intervention to deliver tuberculosis testing results in Peri-Urban and Rural Uganda. J. Clin. Tuberc. Other Mycobact. Dis. 2019, 16, 100110. [Google Scholar] [CrossRef]
  43. Hoehle, H.; Venkatesh, V. Mobile application usability. MIS Q. 2015, 39, 435–472. [Google Scholar] [CrossRef]
  44. Hussain, A.; Mkpojiogu, E.O.C.; Ishak, N.; Mokhtar, N.; Ani, Z.C. An interview report on users’ perception about the usability performance of a mobile e-government application. Int. J. Interact. Mob. Technol. 2019, 13, 169–178. [Google Scholar] [CrossRef]
  45. Hussain, A.; Mkpojiogu, E.O.C.; Jamalsse, A.; Mohammed, R.A. Grab mobile app: A UX assessment on mobile devices. JARDCS 2018, 10, 1233–1238. [Google Scholar]
  46. Saudi Arabia—Information and Communications Technology. Available online: https://www.trade.gov/country-commercial-guides/saudi-arabia-information-and-communications-technology (accessed on 16 December 2022).
  47. National Transformation Program. Vision 2030. Available online: https://www.vision2030.gov.sa/v2030/vrps/ntp/ (accessed on 20 October 2021).
  48. Ministry of Health Saudi Arabia. Statistical Yearbook. Available online: https://www.moh.gov.sa/en/Ministry/Statistics/book/Pages/default.aspx (accessed on 20 October 2021).
  49. Weichbroth, P. Usability of mobile applications: A systematic literature study. IEEE Access 2020, 8, 55563–55577. [Google Scholar] [CrossRef]
  50. Suseno, M.; Hayat, B.; Putra, M.D.K.; Bien, J.K.; Rachmawati, R.; Hartanto, H. A Differential item functioning (DIF) analysis of the mobile phone problem use scale in indonesian schools with and without smartphone banned policy. Cogent Psychol. 2022, 9, 2137306. [Google Scholar] [CrossRef]
  51. Bernritter, S.F.; Okazaki, S.; West, D.C. Mobile technology and advertising: Moving the research agenda forward. J. Advert. 2022, 51, 407–410. [Google Scholar] [CrossRef]
  52. Islam, R.; Islam, R.; Mazumder, T. Mobile application and its global impact. IJEST 2010, 10, 72–78. [Google Scholar]
  53. Apple Saudi Arabia. App Store. Available online: https://www.apple.com/sa-ar/app-store/ (accessed on 20 October 2021).
  54. Johann, T.; Stanik, C.; Alizadeh, A.M.; Maalej, W. SAFE: A Simple Approach for Feature Extraction from App Descriptions and App Reviews. In Proceedings of the 25th International Requirements Engineering Conference (RE), Lisbon, Portugal, 4–8 September 2017; pp. 21–30. [Google Scholar] [CrossRef]
  55. Alqifari, S.; Saleh, S.M.; Habboush, O.; Ibrahim, A.A. Characteristics of electronic health services in Saudi Arabia during the COVID-19 pandemic. Cureus 2022, 14, e28441. [Google Scholar] [CrossRef]
  56. Nascimento, I.; Silva, W.; Gadelha, B.; Conte, T. Userbility: A Technique for the Evaluation of User Experience and Usability on Mobile Applications. In Proceedings of the International Conference on Human-Computer Interaction, Toronto, ON, Canada, 17–22 July 2016; pp. 372–383. [Google Scholar]
  57. Gulliksen, J.; Cajander, Å.; Eriksson, E. Only Figures Matter?—If Measuring Usability and User Experience in Practice is Insanity or a Necessity. In Proceedings of the International Workshop on, Reykjavik, Iceland, 18 June 2008; pp. 91–96. [Google Scholar]
  58. Hassenzahl, M.; Tractinsky, N. User experience—A research agenda. Behav. Inf. Technol. 2006, 25, 91–97. [Google Scholar] [CrossRef]
  59. Bevan, N.; Carter, J.; Harker, S. ISO 9241-11 Revised: What Have We Learnt about Usability since 1998? In International Conference on Human-Computer Interaction; Kurosu, M., Ed.; Springer International Publishing: Cham, Switzerland, 2015; pp. 143–151. [Google Scholar]
  60. Isleifsdottir, J.; Larusdottir, M. Measuring the User Experience of a Task Oriented Software. In Proceedings of the International Workshop on Meaningful Measures: Valid Useful User Experience Measurement, Reykjavik, Iceland, 18 June 2008; pp. 97–101. [Google Scholar]
  61. Sonderegger, A.; Uebelbacher, A.; Sauer, J. The UX Construct–Does the Usage Context Influence the Outcome of User Experience Evaluations? In IFIP Conference on Human-Computer Interaction 2019; Lamas, D., Loizides, F., Nacke, L., Petrie, H., Winckler, M., Zaphiris, P., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 140–157. [Google Scholar]
  62. Nasruddin, Z.A.; Ariffin, N.H.M.; Norwawi, N.M.; Ismail, R.; Wati, A.; Abidin, Z.; Nor, F.N.S.M. Evaluating user experience (UX) factors and emotions of open distance learning (ODL) during the pandemic COVID-19 among secondary school students. Int. J. Inf. Educ. Technol. 2022, 12, 1374–1380. [Google Scholar] [CrossRef]
  63. Ardito, C.; Buono, P.; Costabile, M.F.; De Angeli, A.; Lanzilotti, R. Combining Quantitative and Qualitative Data for Measuring User Experience of an Educational Game. In Proceedings of the International Workshop on Meaningful Measures: Valid Useful User Experience Measurement (VUUM), Reykjavik, Iceland, 18 June 2008; pp. 27–31. [Google Scholar]
  64. Green, W.; Dunn, G.; Hoonhout, J. Developing the Scale Adoption Framework for Evaluation (SAFE). In Proceedings of the International Workshop on Valid Useful Experience Measurement, Reykjavik, Iceland, 18 June 2008; pp. 49–55. [Google Scholar]
  65. De Paula, D.F.; Menezes, B.H.; Araújo, C.C. Building a Quality Mobile Application: A User-Centered Study Focusing on Design Thinking, User Experience and Usability. In Design, User Experience, and Usability. User Experience Design for Diverse Interaction Platforms and Environments; Marcus, A., Ed.; Springer International Publishing: Cham, Switzerland, 2014; pp. 313–322. [Google Scholar]
  66. Digital Government Authority. Available online: https://dga.gov.sa/en/ (accessed on 20 October 2021).
  67. Mcllroy, S.; Ali, N.; Hassan, A.E. Fresh apps: An empirical study of frequently-updated mobile apps in the google play store. Empir. Softw. Eng. 2016, 21, 1346–1370. [Google Scholar] [CrossRef]
  68. Alasmari, A.; Addawood, A.; Nouh, M.; Rayes, W.; Al-Wabil, A. A retrospective analysis of the COVID-19 infodemic in Saudi Arabia. Future Internet 2021, 13, 254. [Google Scholar] [CrossRef]
  69. Tesch, R. Qualitative Research: Analysis Types and Software Tools; Routledge: London, UK, 1990. [Google Scholar]
  70. Cho, J.Y.; Lee, E. Reducing Confusion about Grounded Theory and Qualitative Content Analysis: Similarities and Differences. Qual. Rep. 2014, 19, 1–21. [Google Scholar] [CrossRef]
  71. Kuckartz, U.; Rädiker, S. Analyzing Qualitative Data with MAXQDA; Springer: Basel, Switzerland, 2019. [Google Scholar]
  72. Dix, A.; Finlay, J.; Abowd, G.D.; Beale, R. Human-Computer Interaction; Pearson Education: London, UK, 2003. [Google Scholar]
  73. El-Halees, A. Software usability evaluation using opinion mining. JSW 2014, 9, 343–349. [Google Scholar] [CrossRef] [Green Version]
  74. Download App Reviews From iTunes App Store & Google Play, Heedzy. Available online: https://heedzy.com/ (accessed on 2 March 2020).
  75. Baptista, S.; Wadley, G.; Bird, D.; Oldenburg, B.; Speight, J.; Group, M.D.C.R. User experiences with a type 2 diabetes coaching app: Qualitative study. JMIR Diabetes 2020, 5, e16692. [Google Scholar] [CrossRef]
  76. Bowen, G.A. Naturalistic inquiry and the saturation concept: A research note. Qual. Res. 2008, 8, 137–152. [Google Scholar] [CrossRef] [Green Version]
  77. Garousi, V.; Cutting, D.; Felderer, M. Mining user reviews of COVID contact-tracing apps: An exploratory analysis of nine european apps. J. Syst. Softw. 2022, 184, 111136. [Google Scholar] [CrossRef]
  78. Elkhodr, M.; Mubin, O.; Iftikhar, Z.; Masood, M.; Alsinglawi, B.; Shahid, S.; Alnajjar, F. Technology, privacy, and user opinions of COVID-19 mobile apps for contact tracing: Systematic search and content analysis. J. Med. Internet Res. 2021, 23, e23467. [Google Scholar] [CrossRef]
  79. Choudhury, S.R. Singapore says it will make its contact tracing tech freely available to developers. Available online: https://www.cnbc.com/2020/03/25/coronavirus-singapore-to-make-contact-tracing-tech-open-source.html (accessed on 20 September 2022).
  80. Zhou, S.L.; Jia, X.; Skinner, S.P.; Yang, W.; Claude, I. Lessons on mobile apps for COVID-19 from China. JSSR 2021, 2, 40–49. [Google Scholar] [CrossRef]
  81. Ahmed, A.; Ali, N.; Aziz, S.; Abd-Alrazaq, A.A.; Hassan, A.; Khalifa, M.; Elhusein, B.; Ahmed, M.; Ahmed, M.A.S.; Househ, M. A review of mobile chatbot apps for anxiety and depression and their self-care features. CMPB Update 2021, 1, 100012. [Google Scholar] [CrossRef]
  82. Pandey, A.K.; Janghel, R.R.; Sujatha, R.; Sathish Kumar, S.; Sangeeth Kumar, T.; Chatterjee, J.M. CoronaGo Website Integrated with Chatbot for COVID-19 Tracking. In Proceedings of the CEUR Workshop, Delhi, India, 25–27 February 2021; pp. 521–527. [Google Scholar]
  83. Apple Support. Available online: https://support.apple.com/en-us/HT207122 (accessed on 10 May 2022).
  84. Covidsafe-app. Available online: https://www.health.gov.au/resources/apps-and-tools/covidsafe-app (accessed on 10 May 2022).
  85. Schwamm, L.H. Telehealth: Seven strategies to successfully implement disruptive technology and transform health care. Health Aff. 2014, 33, 200–206. [Google Scholar] [CrossRef] [PubMed]
Sustainability 15 00956 g001 550
Figure 1. The logos of the tested apps.
Figure 1. The logos of the tested apps.
Sustainability 15 00956 g001
Sustainability 15 00956 g002 550
Figure 2. Statistics regarding users’ reviews for the five apps. (a) Pie chart of the total number of reviews for the five apps, with 1338 for Tawakkalna, 107 for Tabaud, 406 for Sehhaty, 29 for Sehha, and 442 for Mawid. (b) Pie chart of the positive reviews of all applications: 466 for Tawakkalna, 83 for Tabaud, 84 for Sehhaty, 6 for Sehha, and 369 for Mawid. (c) Pie chart of the negative reviews of all applications: 510 for Tawakkalna, 4 for Tabaud, 174 for Sehhaty, 18 for Sehha, and 37 for Mawid.
Figure 2. Statistics regarding users’ reviews for the five apps. (a) Pie chart of the total number of reviews for the five apps, with 1338 for Tawakkalna, 107 for Tabaud, 406 for Sehhaty, 29 for Sehha, and 442 for Mawid. (b) Pie chart of the positive reviews of all applications: 466 for Tawakkalna, 83 for Tabaud, 84 for Sehhaty, 6 for Sehha, and 369 for Mawid. (c) Pie chart of the negative reviews of all applications: 510 for Tawakkalna, 4 for Tabaud, 174 for Sehhaty, 18 for Sehha, and 37 for Mawid.
Sustainability 15 00956 g002
Sustainability 15 00956 g003 550
Figure 3. Comparison of the Saudi COVID-19 apps in terms of total usability.
Figure 3. Comparison of the Saudi COVID-19 apps in terms of total usability.
Sustainability 15 00956 g003
Sustainability 15 00956 g004 550
Figure 4. Word clouds of the app reviews. (a) Word cloud analysis of the user reviews for Tawakkalna, (b) Mawid, (c) Tabaud, (d) Sehhaty, (e) Sehha, and (f) all the apps.
Figure 4. Word clouds of the app reviews. (a) Word cloud analysis of the user reviews for Tawakkalna, (b) Mawid, (c) Tabaud, (d) Sehhaty, (e) Sehha, and (f) all the apps.
Sustainability 15 00956 g004
Table
Table 1. List of apps included in the qualitative analysis (from January 2021 to December 2021).
Table 1. List of apps included in the qualitative analysis (from January 2021 to December 2021).
AppNumber of Reviews Collected Number of Reviews
Included in the Analysis
Tawakkalna13381097
Tabaud107102
Sehhaty406366
Sehha2929
Mawid442438
Table
Table 2. Number of positive and negative reviews and frequency of occurrence for each factor.
Table 2. Number of positive and negative reviews and frequency of occurrence for each factor.
AppsSatisfactionEffectivenessEfficiency
Number of Positive ReviewsNumber of Negative ReviewsNumber of Positive ReviewsNumber of Negative ReviewsNumber of Positive ReviewsNumber of Negative Reviews
Tawakkalna466 − 5102 − 1556 − 50
Tabaud83 − 410 − 900
Sehhaty84 − 1741 − 2480 − 52
Sehha6 − 180 − 140 − 10
Mawid369 − 379 − 2624 − 8
Table
Table 3. Total usability score calculated from the users’ reviews of the five applications.
Table 3. Total usability score calculated from the users’ reviews of the five applications.
AppsSatisfactionEffectivenessEfficiencyTotal UsabilityNumber of Reviews
PositiveNegativePositiveNegativePositiveNegative
Tawakkalna43% − 47%0% − 14%1% − 5% − 22%1097
Tabaud84% − 4%10% − 9%0%0%81%102
Sehhaty23% − 48%0% − 68%0% − 14% − 107%366
Sehha23% − 69%0% − 54%0% − 38% − 138%29
Mawid85% − 8%2% − 6%6% − 2%82%438
Total − 104%2032
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Alghareeb, M.; Albesher, A.S.; Asif, A. Studying Users’ Perceptions of COVID-19 Mobile Applications in Saudi Arabia. Sustainability 2023, 15, 956. https://doi.org/10.3390/su15020956

AMA Style

Alghareeb M, Albesher AS, Asif A. Studying Users’ Perceptions of COVID-19 Mobile Applications in Saudi Arabia. Sustainability. 2023; 15(2):956. https://doi.org/10.3390/su15020956

Chicago/Turabian Style

Alghareeb, Mashael, Abdulmohsen Saud Albesher, and Amna Asif. 2023. "Studying Users’ Perceptions of COVID-19 Mobile Applications in Saudi Arabia" Sustainability 15, no. 2: 956. https://doi.org/10.3390/su15020956

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop