COVIDTAS COVID-19 Tracing App Scale—An Evaluation Framework

Abstract

Mobile apps play an important role in COVID-19 tracing and tracking, with different countries taking different approaches. Our study focuses on 17 government owned COVID-19 Contact Tracing Apps (CTAs) and analyze them using a proposed COVIDTAS framework. User satisfaction is not directly related to the COVIDTAS score or the interaction between users and the app developers. To increase adoption of CTAs, government leadership must offer assurance to its citizens that their identify will be concealed and emphasize the benefits of CTAs as it relates to shared public health. While no country has topped the list on all three major factors (COVIDTAS Score, User Reviews, and User Ratings), the CTA from India seems to have above average performance on all three factors.

1. Introduction

The COVID-19 pandemic, through its sheer scale has highlighted and widened the socioeconomic divides that exist today and made vulnerable populations such as those with chronic illnesses, elderly, minorities, people of lower socioeconomic backgrounds, etc., more susceptible [1]. Mobile phones and digital technology has been in the forefront of addressing this crisis in fields varying from education to [2] healthcare [3]. One such approach is contact tracing using mobile applications. The concept of contact tracing in relation to infectious diseases in not new. For years, practitioners of community medicine have isolated those most likely to be infected by a certain infectious disease from any infected individual, isolated these potential candidates, and provided treatment as needed [4]. This approach is extremely effective when done at a local level, with a relatively low number of cases [5]. Contact tracing can be done by local health care workers in person, or by use of technologies ranging from RFID [6] to wearable tracking devices [7].

In recent times, with the advent of smart phones and the high penetration of mobile technology, cell-phone apps have been widely used for contact tracing. Mobile contact tracing applications can be used to identify the proximity of each user to others, including those infected with a contagious disease. Once an infected person is identified, some mobile applications even warn its users when contact is imminent [8]. Clearly, such applications have a potential to minimize person to person transmission of diseases. In addition, these applications allow healthcare workers to identify those at risk of infection, allowing for pre-emptive isolation and treatment to minimize further spread as well as the intensity of infestation.

These applications have become more mainstream in light of the ongoing COVID-19 pandemic. It should be noted that the use of such applications is not without controversy, and it has been argued that extensive data gathering by governments can eventually do more harm than good [9]. However, mobile apps have been employed successfully across the world for years to prevent the spread of contagious diseases. Therefore, a methodology to assess the applications currently employed for contact tracing on criteria such as data security, privacy, data retention, etc., is of significance. Understanding these applications in the context of various discrete criteria can help assess their risk to citizens and governments should the data be misused. In that light, this study presents a novel COVIDTAS framework for assessing COVID-19 tracing applications. The evaluations rely on public data and reviews on various platforms for analysis. However, before analysis it is essential to gain a basic understanding of contact tracing applications and their use in medicine. In that light, the next section explores a few such examples with a focus on how government mechanisms in the past have employed mobile contact tracing apps. This is followed by a discussion on digital apps used for COVID-19 tracing specifically. The study then focuses on COVID-19 applications that are initiated by governments and the concerns associated with a government driven approach. The data used in this study is then presented and analyzed before discussing the results of the analysis. In addition, the perceptions surrounding government controlled COVID-19 applications is also discussed in this paper before presenting the conclusion and future research direction.

2. History of Contact Tracing

Contact tracing becomes essential when the potential of a disease for person to person spread is high. Naturally, the earlier applications of mobile apps for contact tracing has been for diseases such as Ebola, SARS/MERS, TB, etc. Broadly speaking, these applications fall under m-health or mobile health interventions, where mobile technology is used to aid health care. It should be noted that while this paper focuses on contact tracing, m-health encapsulates applications such as remote diagnosis, data sharing, and information dissemination in addition to contact tracing. Contact tracing apps have been employed extensively in managing the spread of Ebola in African countries over the last decade. For example, in the 2014–15 outbreak of Ebola in West Africa, mobile interventions were used for contact tracing in addition to community level manual tracing. In Sierra Leone, a mobile app termed as Ebola Contact Tracing application (ECT app) was employed for contact tracing [10]. Results show that app-based interventions allowed for more concrete, accurate, and timely updating of the centralized database if done correctly. In addition, the manpower and paperwork required for contact tracing was also drastically reduced [10]. A study of 58 Ebola contact tracing apps by [11] identifies three applications, Community Care, Sense Ebola Follow up, and Surveillance and Outbreak Response Management and Analysis System as capable of carrying out surveillance, contact tracing, management of cases, and data management for laboratory data, thereby making them complete data management systems for Ebola and similar contagious diseases. A more generic software Go.data [12] which can be employed for any disease outbreak with peer to peer transmission has been used successfully in Democratic Republic of Congo for Ebola contact tracing successfully. Similarly, mobile apps have been employed for contact tracing in case of TB, SARS, and other respiratory diseases [13,14].

3. Present Use of Contact Tracing Apps for COVID-19

The potential use of contact tracing in the control of COVID-19 has been well established by studies, especially in cases where there was less chance of transmission before the onset of symptoms [15]. The use of mobile apps for contact tracing has come to the mainstream with the recent COVID-19 pandemic. Governments and healthcare agencies across the world are employing mobile apps for identifying potential paths of infection and isolating those at risk. The MIT project on COVID-19 tracking apps lists over 25 apps at this stage, which are employed by governments across the world for contact tracing during this pandemic [16]. The project lists a number of applications ranging from those that are compulsory to those that are optional. Many vary widely in terms of transparency, data destruction (for privacy), and the technology employed. Similarly, FIPRA (2020) lists the applications being used in Europe for contact tracing.

A few examples include the COVIDSafe app by Australian government, which faced resistance from the public in general due to concerns about privacy and was not entirely successful in providing contact tracing due to lack of widespread adoption of the application [17]. India’s Aarogya Setu app, which is the most downloaded contact tracing application, effectively collects personal information and constantly adds to location information in real time for its users [18]. On the other hand, the Corona-Warn-App used in Germany posed no major privacy concerns as it was developed with potential privacy infringements in mind. Data is depersonalized and stored locally to avoid any concerns in this regard [19]. While many countries such as India have made contact-tracing apps practically compulsory for activities such as air travel [20], other countries such as the UK have refrained from employing similar procedures due to public concerns over privacy [17,21,22]. There are clearly diverse approaches taken by governments during this pandemic when it comes to mobile contact tracing. However, in general, contact tracing apps have been employed as a novel tool for curtailing the spread of this disease. The concerns raised by many with regards to privacy and data security in relation to these applications are valid and need to be discussed in detail.

4. Government Interventions in Contact Tracing

The role of governments in contact tracing has been a matter of much discussion. Multiple surveys and studies find varying degrees of public trust in government contact tracing applications. For example, a survey carried out in the Republic of Ireland with over 8000 participants shows a general acceptability for a contact tracing application. In fact, 54% of the respondents were positive about downloading such an app, while another 30% were moderately likely to do the same [23]. This is in sharp contrast with a survey on over 2000 participants from the UK, where the acceptance rate for a government-controlled application was under 55%. However, there was greater acceptability for an app controlled by the NHS [24], clearly highlighting a lack of trust in the government handling sensitive medical information. Citizens of many other countries are effectively left without a choice in the matter. For example, while data collection for public good needs no individual consent in China, it is possible for the government to provide private parties access to the same data, thereby leading to potential privacy concerns [19]. Similarly, a study on a COVID-19 contact tracing app propagated by the government of France highlights how such applications, though immediately beneficial, may be a serious challenge to data privacy in the future. The study also suggests that the government may be downplaying this adverse effect for the immediate purpose of contact tracing [25].

Similarly, the use of contact tracing in Russia has raised serious concerns about it being used for mass movement surveillance. The application suggested in Moscow may require citizens to generate a unique code each time they exit their living space, and share the code with law enforcement, should they be asked to do so [26]. Some countries such as South Africa and the UAE are yet to have concrete data privacy laws, making the deployment of such applications all the more concerning as they do not have to adhere to existing data laws [27]. Amnesty international (2020) has raised privacy concerns related to contact tracing apps and identifies the applications employed in Bahrain, Kuwait, and Norway to be among the least safe for data privacy. There have also been significant differences in how governments across the world handled these concerns related to surveillance and data security. For example, Singapore chose to publicly acknowledge the potential security concerns with its application and asked its citizens to concentrate on the immediate contact tracing requirement [28]. Hong Kong employed geo fencing instead of GPS tracking to provide some degree of privacy to its citizens while South Korea addressed such concerns by stringently restricting access to the database [29]. In other countries such as India and Russia the concerns regarding misuse of data from tracking apps is still significant [26,30,31]. The limited number of studies available on this issue highlights that mass adoption of these applications is necessary for them to be successful. This can either be achieved through legislation (making it compulsory to use the application), or through trust building with the public. In addition, with regards to COVID-19, the disease disproportionately affects the elderly, who, despite having smart phones, are often technology averse and unlikely to install and use mobile applications [32]. In general, at least in democratic systems, such mobile applications are unlikely to succeed without adequate public trust as illustrated by cases in the UK [33] and Australia 17].

5. Challenges and Apprehensions Regarding Government Contact Tracing

The papers discussed so far clearly illustrate the challenges of balancing individual rights regarding data privacy, with the immediate need for contact tracing [34]. From literature, it is also seen that specific policy and technology measures can help alleviate and minimize these concerns. This is best illustrated by the approach taken by Germany, where data privacy was built into the contact tracing application while also providing decentralized storage of data, thereby minimizing the loss or misuse of largescale data [35]. Similarly, by employing geo fencing instead of GPS tracking, Hong Kong has managed to address concerns of governments tracking its citizens to a great extent. Existing privacy and data security laws also play a part in protecting citizens of any country against the misuse of this sensitive information. In general, it is safe to conclude that in democratic systems, it is best that the government spend adequate resources on educating the public about the importance of contact tracing applications while simultaneously being transparent in addressing any concerns regarding privacy and data security.

6. Materials and Methods

Contact Tracing Apps (CTAs) are designed to automate the process of gathering data of contacts of people with infected individuals, so as to identify the risks based on the context, proximity and duration of contact. CTAs estimate proximity through Bluetooth or Global Positioning System (GPS). GPS is not suitable as the accuracy is low in built-up spaces. Bluetooth is available in smartphones and can support proximity estimation, however the distance estimation may vary based on surrounding objects and the orientation of the phone [30], thus leading to false positive and negative alerts.

Contact tracing apps may be designed with a centralized architecture, where all activities revolve around a trusted server that supports storing encrypted information, analyzing risk of contacts and notifying such identified contacts [36,37]. In this case, the identity of the users is known to the trusted server. India’s Aarogya Setu and Singapore’s TraceTogether are examples of centralized architecture (

Table 1. Characteristics of COVID-19 CTAs.

Architecture	Centralized	Decentralized
Pre-register	Yes	No
Store personal identification	Yes	No
Privacy	Generated by server	Generated at client device
Contact risk analysis	Server can run data analytics including risk analysis and notifications	None, as server does not maintain data
Data protection	A malicious attack can compromise the central server to access user information	Data is stored on device, minimal data uploaded to server
Data retention	Long-term	Comparatively lesser
Provides data on disease transmission (epidemiology)	Yes	Yes

).

In contrast, the decentralized architecture minimizes the information exchange and anonymizes user identification with the server [38]. Australia’s COVIDSafe [39] is built with decentralized architecture. There is also a hybrid model, where only the personal ID and data of COVID-19 positive users or volunteers are saved at the server [40].

This study is focused on contact tracing applications used by governments for COVID-19 mitigation. Data from Contact Tracing Apps of 17 countries with at least one country from each of the seven continents were aggregated as part of this study (

Table 2. COVID-19 CTAs explored in the study.

Continent	Country	App Name	URL
Asia	India	Aarogya Setu	nic.goi.aarogyasetu (accessed on 7 March 2021)
Asia	Singapore	TraceTogether	sg.gov.tech.bluetrace (accessed on 7 March 2021)
Asia	Vietnam	Bluezone—Contact Detection	com.mic.bluezone (accessed on 7 March 2021)
Asia	Japan	COCOA—COVID-19	jp.go.mhlw.covid19radar (accessed on 7 March 2021)
Asia	Israel	המגן 2—האפליקציה הלאומית למלחמה בנגיף הקורונה	com.hamagen (accessed on 7 March 2021)
Asia	UAE	TraceCovid	ae.tracecovid.app (accessed on 7 March 2021)
Asia	Qatar	EHTERAZ	com.moi.covid19 (accessed on 7 March 2021)
North America	Canada	COVID Alert	ca.gc.hcsc.canada.stopcovid (accessed on 7 March 2021)
North America	Mexico	COVID-19MX	mx.gob.www (accessed on 7 March 2021)
Australia	Australia	COVIDSafe	au.gov.health.covidsafe (accessed on 7 March 2021)
New Zealand	New Zealand	NZ COVID Tracer	nz.govt.health.covidtracer (accessed on 7 March 2021)
Europe	Germany	Corona-Warn-App	de.rki.coronawarnapp (accessed on 7 March 2021)
Europe	France	StopCovid	fr.gouv.android.stopcovid (accessed on 7 March 2021)
Europe	Spain	Radar COVID	es.gob.radarcovid (accessed on 7 March 2021)
South America	Brazil	Coronavirus—SUS	br.gov.datasus.guardioes (accessed on 7 March 2021)
South America	Columbia	CoronApp	co.gov.ins.guardianes (accessed on 7 March 2021)
Africa	Morocco	وقايتنا	covid.trace.morocco (accessed on 7 March 2021)

). A multistage approach was adopted to collect the data.

The process followed for data selection and analysis is presented below.

• Stage I involved selection of countries were primarily based on two factors (1) countries that had proactive involvement of state-owned bodies in the control, coordination and reporting of their national status and (2) additionally, to ensure good geographical spread, countries were chosen from various continents. This resulted in three countries being chosen within Asia, three countries from Middle East, two countries from N. America, three countries from Europe, two from South America, one country from Africa, Australia, and New Zealand.
• Stage II involved comprehensive research that involved analyzing the design of CTAs, privacy policies and their mapping to the mandatory requirements of the state. Additionally, policies on transparency were verified through open-source availability of CTA software databases and adherence to GDPR requirements. The adherence to various GDPR requirements were assessed for non-European countries from publicly available authentic and reputable sources that provided critical analysis of CTA’s security and privacy features. Cross-validations with news articles and/or research publications were carried out wherever possible.
• Stage III involved evaluation of publicly available reviews inclusive of those shared by user, legal firms and the media.

Certain countries such as USA, Russia, China, S. Africa, etc., were excluded from this study due to lack of adoption of CTAs or reliance on private or multiple entities for contact tracing apps. India, France, Spain, Brazil, Colombia, and Mexico had the highest number of infected people in their respective continents, i.e., Asia, Europe, South America, and North America. Additionally, they were also amongst the Top 10 countries worldwide from the highest number of total cases and total number of deaths perspective. The reason Qatar and Israel were chosen was because of their high density of COVID-19 cases i.e., total cases per million population. The interesting aspect of analyzing data from Singapore was that in spite of having a relatively large number of cases i.e., close to 10,000 per million population, the number of deaths per million is amongst the lowest worldwide i.e., five per million. On the other hand, Vietnam and New Zealand were amongst the countries that had the lowest number of COVID-19 cases per million population and two to three orders of magnitude lower deaths per million in comparison to the highly infected countries. On user ratings and reviews, data was collected primarily from Google app store rather than Apple app store due to the former being a much larger repository of user downloads and usage statistics.

7. Qualitative and Quantitative Parameters in the COVIDTAS Framework

A reliable and sound method for analyzing the various applications selected was essential for this study. The framework should be able to assess the effectiveness of the application on various fronts and provide an overall score for the application for comparison with its peers. This study proposes The Covid Tracing App Scale (COVIDTAS) framework as a platform to compare the various dimensions of these apps. This includes elements of human-centric design from usability, technology and privacy perspectives, their effectiveness in tracking and communicating proximity to COVID-19 patients and ultimately the user perception and acceptance based on their direct feedback. This framework has the potential to serve as an effective tool to key stake holders i.e., the public, medical personnel and federal authorities and assist with enhancing adoption, usage and impact from such apps. COVIDTAS framework was adapted from the sociotechnical framework proposed by [41] that pertained to analyzing the design configurations of the app from technology, governance and user perspectives. Our framework extends this even further by including factors that reflect user experience and sentiment that are critical to its success.

The data collected included both qualitative and quantitative parameters. The qualitative parameters included: the extent of data security, transparency, ease of use, access to information, architecture, and so on. While synthesizing the data, emphasis was placed on the congruence between the qualitative parameters and the policies governing CTAs. Particular attention was paid to good and suboptimal practices that led to the identification of technological, organizational, cultural, and individual factors that were captured in the final empirical framework. The scoring rubric for all qualitative factors ranged between 0 and 2 with 2 representing the highest rating for the most preferred option, 1 representing moderate or partial compliance, and 0 representing the lowest rating for inadequate coverage or poor compliance (

Table 3. Factors and scoring rubric for the COVID-19 CTAs.

Factors	Score 2	Score 1	Score 0
Data collection should be compliant with the General Data Protection Regulation (GDPR) [6] and respect the privacy of the individual. A Data Protection Impact Assessment (DPIA) must be carried out before the deployment of any contact-tracing system. The purpose of the app and the mechanisms to assess its usage need to be clearly defined	Full compliance	Partial implementation of GDPR	Noncompliant or disregard for privacy
Transparency rights: They include the right of users to be notified, to control their own data, transparency regarding which personal data are collected, and of explanation of app-produced output. The app should be auditable	Users are in full control of their data	Partial fields are controlled by users	Users have zero control of their data
Accessibility refers to availability of CTA to everyone without any discrimination	All citizens have equal access to the CTA	N/A	CTAs are only available to certain sections of public
Availability of multimodal educational materials and/or tutorials	At least 10 documents and videos were available	Availability of less than five reference materials	Availability of fewer than two documents that were difficult to find
Type of protocol underlying the design of the CTA	Decentralized protocols	Hybrid protocols	Centralized protocols
Data management with respect to amount of data being collected, retained and saved	Local and temporary retainment of data	Inadequate documentation on data storage	Centralized storage and extended retainment of data
Data security refers to in-built mechanisms to prevent tampering or altering of data while on the device, during access or transfer	Security maintained with pseudorandom identifiers and frequent renewals	Partial compliance to security requirements	No consideration of data security needs
Activation and deactivation process	Swift and user-controlled activation and deactivation process through app	Additional steps involved in deactivation such as calling of authorities	Cumbersome to activate or impossible to deactivate
Nature of CTA’s federating body	State-owned body	N/A	N/A
Mandatory or voluntary download and use of CTA	Voluntary	Partially voluntary	Fully enforced
CTA is open-sourced or not	Open-sourced and allows contributions	Open-sourced and unsupported	Not open-sourced

)

Some of the other factors such as legislative policies, incidental finding or dual-use policy mentioned in Vinuesa et al. (2020) were difficult to compare due to nonavailability of information to make reasonable comparisons between the 17 CTAs. It is also assumed factors such as status of fundamental rights of individuals and any discrimination or stigmatization are low or fairly similar across countries not influential in this analysis.

Several quantitative data points were systematically analyzed from the 17 countries. These included considering their total population, the number of COVID-19 cases in their countries and the number of CTA downloads to derive the percentage of adoptees. Additionally, the total number of cases, total number of deaths and total number of COVID-19 tests taken per million of the population were tabulated. Viewing this data set helped ensure and confirm our objective of selecting countries that captured a wide spectrum of impact from COVID-19 in their respective countries. The quantitative factors coded into the framework included the user review and star ratings of the respective apps. This involved looking at both reviews and star ratings of 17 CTAs between March and September 2020. Analyzing the total number of reviews and responses to the reviews revealed that there was a direct correlation between the number of responses and the percentage of adoption. CTAs from six countries (i.e., Vietnam, Japan, France, Spain, Brazil, and Australia) in this cohort had provided no responses to reviews.

Previous studies [42] have shown that user acceptability is critical in adoption of CTAs resulting in tangible management of the spread of COVID-19. More specifically the privacy concerns were found to be the largest impediment for acceptance and usage in developed countries such as Germany and US. The model used in their work, i.e., health belief model, anchors on the user perception of both i.e., consequences of a health threat and how effective the proposed ways of its management or protection from it are. Thus, direct experiences and opinions of users are important to accommodate in viewing the impact of CTAs holistically. The user ratings utilized in our framework included their review-based ratings and star ratings. A one- or two-star rating was considered inadequate and earned a score of 0 while a four-star or five-star rating was assigned a score of 2. Additionally, Google’s ratings between 0 and 5 were also captured and linearly extrapolated and mapped to score of 0–2. Comparisons between Google’s app ratings and user’s review ratings showed reasonable congruence between them in most cases.

We first provided a characterization of all the CTAs from Google Play Store. Then we analyzed the parameters like number of installs, average star ratings, total reviews, reviews by star ratings from the website applfow.io and androidrank.org. The COVIDTAS score was derived by averaging scores of all 12 factors in the framework. All factors included in the framework were given equal weightage.

8. Results and Discussion

The COVIDTAS framework evaluated whether the 17 state-owned CTAs selected for this study satisfied parameters such as privacy and data protection, transparency rights, data management, etc. as shown in Table 4. Each CTA was thoroughly studied to identify if the apps were compliant with the 12 parameters that were the foundation of the COVIDTAS scale.

Overall, the New Zealand app scored the highest (1.82), followed by the apps from Canada and Australia (1.67), and those from Japan and Germany (1.57). The apps that scored less than 1 were from Qatar and Columbia. Though the EHTERAZ tracing app from Qatar was legally enforced on the public, it was not available in the local language and did not support app-related education and tutorials for the public. The app scored 0 in other aspects as well such as transparency rights, ease of accessibility and security. The South American apps also scored low on the COVIDTAS scale (Brazil 0.67, Columbia 0.58) as they scored 0 in all other parameters. It was also unclear if these state-owned apps can be removed or deactivated easily. All these apps (Middle East and South America) and COVID-19MX from Mexico did not comply with data minimization principles and were found to be a security risk to their users.

The research involved analyzing the average star rating given to the apps by users. To do so, the 1 to 5 scale of Google Play Store were converted into 0, 1, 2, wherein 1–2 was scored 0, 2.01–3.50 was scored 1, and 3.51–5.0 was scored 2. It was determined that the apps from Asia, North America, Columbia, and Africa received the highest rating of 2, whereas apps from Japan, Israel, UAE, New Zealand, Australia, Europe, and Brazil received a one-star rating. When exploring the review ratings for these apps, only six apps scored a rating of 2, namely, India, Vietnam, Canada, Mexico, Columbia, and Morocco. A higher percentage of users (68% to 89%) gave these six apps a four- or five-star rating. The other 11 CTAs scored 1 for this parameter.

Some interesting observations can be made regarding the performance of these COVID-19 apps by employing the COVIDTAS framework. The stress placed on privacy, transparency, and data security seems to be dependent on the state of development of the country involved. Each CTA was examined under the “Privacy and data protection” COVIDTAS parameter to assess if data collection was in agreement with the General Data Protection Regulation (GDPR), user’s privacy was respected, data protection impact assessment was done, and other such variables given in the Table 3. It was seen that CTAs from Europe, Australia, New Zealand, and Canada completely fulfilled these requirements, whereas countries from the Middle East (UAE, Qatar) were not adequate, which, as mentioned above, was reflected in their low COVIDTAS score (1, 0.5, respectively).

To achieve complete protection of users’ data, CTAs should ideally allow interoperability, use Bluetooth technology and follow a complete decentralized protocol. Those that followed a centralized approach were from Singapore (TraceTogether), Qatar (EHTERAZ), Canada (COVID Alert), Mexico (COVID-19MX), Australia (COVIDSafe), France (StopCovid), and South America (Coronavirus—SUS, CoronApp).

Israel, Canada, Australia, New Zealand, and European countries score high in parameters related to privacy, transparency, data management, and security. New Zealand, Germany, Australia, and France score a maximum possible score of 8 (2 in each category) while Israel, Canada, and Spain score 7. It is also interesting to note that these ‘Western’ countries score higher than their Asian counterparts of similar socioeconomic development. Singapore and Japan, the two developed Asian economies considered in this study score 6 points in this regard. Vietnam and India, the two developing Asian economies also score 6 each in this regard. Countries in the Middle East (with the exception of Israel, which is a democracy) and South America (with relatively unstable governments) seem to fare the lowest in this regard, with Brazil, Columbia, UAE, and Qatar scoring zero across these parameters. It is safe to assume that healthy democracies fare better in terms of data integrity and security due to higher levels of accountability to the public. In addition, economic development usually allows for greater data protection and citizens of developing countries face greater challenges in this regard [43]. This trend continues with COVID-19 applications as well.

When factors that determine the effectiveness of the application such as decentralization, accessibility, and awareness raising (through education) are considered, New Zealand and Japan score the highest points, while India, Vietnam, Australia, Germany, and Spain are close behind. Large scale advertisements and interventions regarding COVID-19 and CTAs in general have been shown to improve acceptability of the application [44]. This characteristic does not seem to be dependent on geography, socioeconomic development, or type of government. Ten apps were observed to be educative and user friendly as they provided reading materials and tutorials within the app and external access to websites to improve awareness on the disease as well as information on using the app correctly. However, the apps from Qatar, Mexico, Europe (Germany provided website access), South America, and Africa did not provide any public health awareness or app information. In countries such as India, use of CTAs was mandated in certain locations such as airports, government offices while it was not mandated at other public places such as malls and recreational sites.

The ease of removing the app from the device and whether the installation is forced legally in any way are both measures of the degree of control citizens have over the application. In this regard, much like with data privacy, developed ‘Western’ economies (Australia, New Zealand, Canada, Germany, and France) score the highest values along with Japan, and UAE. It should be noted that Singapore, a developed economy from Asia does not score as highly in this regard, much like with data security. The only serious deviation in the trend between data security and the degree of control citizens have over the application itself is in UAE, where the data security and privacy itself is very low, but citizens are not legally required to install the application and can deactivate the application at will. This may be because the UAE government relied more on ground teams for testing rather than contact tracing for managing the pandemic [38,45].

The significant trends to be observed with regards to privacy is that both the form of government and state of socioeconomic development seems to have a relation to the importance placed on data privacy. Stable democracies with better development indicators seem to fare better in this regard while developing countries and unstable democracies and monarchies are less so. Within developed democracies, Western countries fare better than their eastern and Middle Eastern counterparts. With exceptions, these same countries also give a greater degree of control to its citizens regarding the installation and utilization of the application. The effectiveness of the application in terms accessibility, awareness generation and decentralization does not seem to have any dependence on the developmental status or governance style of the country.

9. Public Perception of CTAs

At the beginning, there was a lot of uncertainty and inexperience in managing the COVID-19 pandemic. Countries around the world tackled it in different ways. Vietnam and Singapore developed strategies to eradicate the virus, Australia tried to arrest and control the transmission, whereas Sweden tried to build herd immunity. Another point to note was the response time taken by the authorities to suppress virus transmission, Italy had a comparatively slower response in contrast to South Korea which had a faster response in preventing the infection. The study reviewed 17 CTAs that targeted the COVID-19 disease from all over the world. The customer ratings of these applications are provided in

Table 5. Google Play Store COVID-19 CTA ratings.

CTA Name	Country	URL	Avg. Star Rating (0 1 2) 1–2, 2.01–3.50; 3.51–5.00	Avg. Ratings	Total Ratings	Avg. Ratings by Reviews	Avg. Rating by Reviews (0 1 2)	Total Reviews	Total Replies
Aarogya Setu	India	nic.goi.aarogyasetu (accessed on 7 March 2021)	2	4.376	1,335,785	4.28	2	54,043	33,385
TraceTogether	Singapore	sg.gov.tech.bluetrace (accessed on 7 March 2021)	2	4.024	7383	3.112	1	2707	893
Bluezone—Contact detection	Vietnam	com.mic.bluezone (accessed on 7 March 2021)	2	4.623	67,422	4.373	2	19,196	2914
COCOA—COVID-19	Japan	jp.go.mhlw.covid19radar (accessed on 7 March 2021)	1	2.989	6295	2.871	1	3157	0
המגן 2 - האפליקציה הלאומית למלחמה בנגיף הקורונה	Israel	com.hamagen (accessed on 7 March 2021)	1	3.418	4629	3.278	1	1739	889
TraceCovid	UAE	ae.tracecovid.app (accessed on 7 March 2021)	1	3.373	314	2.904	1	167	24
EHTERAZ	Qatar	com.moi.covid19 (accessed on 7 March 2021)	2	3.926	26,194	3.053	1	9685	434
COVID Alert	Canada	ca.gc.hcsc.canada.stopcovid (accessed on 7 March 2021)	2	3.872	3033	3.966	2	1383	205
COVID-19MX	Mexico	mx.gob.www (accessed on 7 March 2021)	2	3.602	3116	4.021	2	1810	38
COVIDSafe	Australia	au.gov.health.covidsafe (accessed on 7 March 2021)	1	2.753	12,909	3.147	1	7353	0
NZ COVID Tracer	New Zealand	nz.govt.health.covidtracer (accessed on 7 March 2021)	1	2.478	2816	2.115	1	2200	1038
Corona-Warn-App	Germany	de.rki.coronawarnapp (accessed on 7 March 2021)	1	3.38	72,971	3.438	1	27,247	5679
StopCovid	France	fr.gouv.android.stopcovid (accessed on 7 March 2021)	1	2.88	9121	2.864	1	4741	0
Radar COVID	Spain	es.gob.radarcovid (accessed on 7 March 2021)	1	3.181	1778	2.902	1	942	0
Coronavírus—SUS	Brazil	br.gov.datasus.guardioes (accessed on 7 March 2021)	1	3.238	14,080	2.829	1	6658	0
CoronApp	Columbia	co.gov.ins.guardianes (accessed on 7 March 2021)	2	4.033	64,511	3.601	2	35,968	31,202
wiqaytna)	Morocco	covid.trace.morocco (accessed on 7 March 2021)	2	4.359	12,300	4.566	2	4723	309

.

CTAs that were centrally, federally and government sponsored were included in the study. Based on average customer ratings (>4) on Google Play Store, it was observed that certain country-specific apps had scored the higher ratings, namely Bluezone—Contact detection (Vietnam; 4.62), Aarogya Setu (India, 4.38), Wiqaytna (Morocco; 4.36;), and CoronApp (Columbia; 4.03).

Bluezone—Contact detection app was launched in April and has the second highest ratings on Google Play (total ratings: 67,422) after India. Overall average ratings demonstrated that a high proportion of users were satisfied with the quality of the app for both Vietnamese (84%) and English (61%) versions, with the Vietnamese app scoring higher. In total, 12% gave one or two stars for the Vietnamese version, and for the English version 27%. Total reviews received from users was 19,196 with 1600 reviews/month. In total, 2914 reviews received replies with 243 replies coming every month.

Aarogya Setu app from India is one of the most popular CTAs with more than 100 million downloads and the highest number of ratings from Play Store (1,335,785). The app also received the most reviews (54,043; 4504 reviews/month) and replies to reviews (33,385; 2782 replies/month). A total of 79% of users were satisfied with the performance of this app with 16% giving it a one- or two-star rating.

Compared to all other CTAs, the average star rating (4/5) for Wiqaytna CTA (Morocco) was the highest (89%) indicating that users were satisfied with the quality and performance of this app, whereas only a small percentage (9%) disliked or hated the app. In the Play Store, 12,300 total ratings, and 4723 total reviews with 394 reviews/month was observed. The app received 309 overall replies to reviews and 26 replies/month.

CoronApp from Columbia was launched in March, making it one of the earliest CTA launches. After India, this app received the most replies to reviews. About 64,511 users rated this app, which received 35,968 reviews and 2997 reviews/month. There were 31,202 replies to reviews and 2600 replies/month. Users gave a four- or five-star rating (68%), a few gave one- or two-star ratings (23%).

Tracing apps that received the lowest user ratings (<3) were NZ COVID Tracer (2.48; New Zealand), COVIDSafe (2.75; Australia), StopCovid (2.88; France), and COCOA—COVID-19 (2.99; Japan).

Implementing strict lockdown strategies, improved rapid testing facilities, and case management has enabled New Zealand to contain the COVID-19 outbreak twice. For the purpose of tracing and isolating COVID-19 contacts, New Zealand health authorities developed the NZ COVID Tracer CTA. The app received 2816 ratings on Google. The number of total reviews received was 2200 and 183 reviews/month. In total, this app received 1038 replies to reviews and 87 replies/month. User satisfaction rating (four or five stars) was observed to be 23% in contrast to 68% who gave one- or two-star ratings (hated or disliked the app).

COVIDSafe from Australia received a total of 12,909 ratings, with 7353 reviews in total and 613 reviews/month. This app received no replies to reviews. Half of the users gave a rating of four or five stars and 44% gave it one star or two stars.

In June, the StopCovid app from France was released to the public in English and received 9121 ratings. Out of 4741 reviews, this app received 395 reviews/month. An interesting observation noted with this app was that there were no replies to reviews. In total, 53% of the public gave it four or five stars and 38% gave it a one- or two-star rating.

In Japan, COCOA—COVID-19 was launched in June in the local language. It received 6295 overall ratings. This app received 3157 reviews with 263 reviews/month. Like the StopCovid app from France, this app also received no replies to reviews at all. Users who gave it a four- or five-star rating (38%) were lower than those who rated it one or two stars (46%).

It is evident that the public perception of the application has a correlation to the utility or data privacy measures taken by the application. Countries like India, Vietnam, and Canada which scored above 1.5 also scored high in customer reviews. Morocco, which scored 1.25 in the framework also has the highest overall review score. However, it is evident that the COVIDTAS framework cannot directly measure customer experience. Countries like Australia, New Zealand, Germany, France, and Spain which had COVIDTAS scores above 1.5 have relatively low average ratings, with the application in New Zealand scoring a low 2.12 average score. Surprisingly, the application from Mexico which had a low COVIDTAS score of 0.75 has a remarkably high average customer review score of 4.02. Therefore, it is best to view customer experience and the effectiveness of the application as viewed by the COVIDTAS framework as two separate metrics for assessing the application.

A more detailed analysis of the CTAs from the perspective of customer experience is provided in

Table 6. Google Play Store user reviews for CTAs.

Country	CTA Name	One-Star	%	Two-Star	%	Three-Star	%	Four-Star	%	Five-Star	%	Total Reviews	Total Replies	Four, Five Star Reviews	One, Two Star Reviews
India	Aarogya Setu	5658	13%	1297	3%	1864	4%	3413	8%	30,258	71%	54,043	33,385	1%	0%
Singapore	TraceTogether	746	32%	188	8%	285	12%	216	9%	912	39%	2707	893	0%	0%
Vietnam	Bluezone—Contact Detection (English)	115	21%	32	6%	61	11%	51	9%	281	52%	540	39	1%	0%
Vietnam	Bluezone—Contact Detection (Vietnamese)	1595	9%	470	3%	865	5%	1517	8%	13,736	76%	18,183	2875	1%	0%
Japan	COCOA—COVID-19 Contact App (Japanese)	1000	33%	387	13%	503	16%	348	11%	838	27%	3157	0	0%	0%
Israel	המגן 2 - האפליקציה הלאומית למלחמה בנגיף הקורונה (English)	121	35%	57	16%	40	12%	34	10%	94	27%	346	121	0%	1%
Israel	המגן 2 - האפליקציה הלאומית למלחמה בנגיף הקורונה (Hebrew)	284	27%	76	7%	99	9%	78	7%	517	49%	1054	673	1%	0%
Israel	המגן 2 - האפליקציה הלאומית למלחמה בנגיף הקורונה (Russian)	80	36%	23	10%	30	13%	18	8%	73	33%	224	95	0%	0%
UAE	TraceCovid	40	30%	12	9%	14	10%	10	7%	58	43%	167	24	1%	0%
Qatar	EHTERAZ	3303	41%	530	7%	493	6%	436	5%	3323	41%	9685	434	0%	0%
Canada	COVID Alert	212	17%	64	5%	69	5%	99	8%	836	65%	1383	205	1%	0%
Mexico	COVID-19MX (Spanish)	293	17%	46	3%	94	5%	162	9%	1133	66%	1810	38	1%	0%
Australia	COVIDSafe	2597	36%	568	8%	498	7%	439	6%	3189	44%	7353	0	1%	0%
New Zealand	NZ COVID Tracer	1269	59%	202	9%	165	8%	157	7%	343	16%	2200	1038	0%	1%
Germany	Corona-Warn-App (English)	250	21%	73	6%	82	7%	96	8%	687	58%	1188	402	1%	0%
Germany	Corona-Warn-App (German)	6692	26%	1931	8%	2307	9%	2279	9%	12,153	48%	25,362	5277	1%	0%
France	StopCovid France (English)	62	30%	17	8%	18	9%	23	11%	86	42%	206	0	1%	0%
France	StopCovid France (French)	1795	40%	418	9%	410	9%	367	8%	1475	33%	4465	0	0%	0%
Spain	Radar COVID (English)	19	48%	4	10%	1	3%	5	13%	11	28%	40	0	0%	1%
Spain	Radar COVID (Spanish)	276	35%	82	10%	89	11%	61	8%	284	36%	792		0%	0%
Brazil	Coronavirus—SUS (English)	131	38%	32	9%	20	6%	12	4%	147	43%	342	0	0%	0%
Brazil	Coronavirus—SUS (Portuguese)	2550	41%	592	10%	539	9%	407	7%	2081	34%	6169	0	0%	1%
Columbia	CoronApp—Colombia	220	51%	34	8%	46	11%	43	10%	92	21%	35,698	31,202	68%	23%
Morocco	وقايتنا (Wiqaytna) (English)	39	9%	12	3%	17	4%	30	7%	322	77%	420	19	1%	0%
Morocco	وقايتنا (Wiqaytna) (French)	220	8%	36	1%	51	2%	169	6%	2401	83%	2877	290	1%	0%

.

India, Vietnam, Canada, Mexico, and Morocco have a higher percentage of four- and five-star reviews, indicating greater customer satisfaction. Similarly, the number of times the application has responded to its customers is also a key indicator. When normalized by the number of reviews, it is seen that application from Colombia has responded to 83% of the reviews. Similarly, India and Israel also have high response rates of 61% and 54%, respectively. However, no direct correlation to customer satisfaction could be observed. In fact, highly rated applications from Morocco and Canada have relatively low response rates. This could partly be because customer complaints are more likely to warrant a response from the developer, leading to a larger number of responses to applications. In addition, the approach taken by the nation towards promoting the application could also play a part in the response rate. For example, in a country like India with relatively low technology literacy, the application was made necessary for various travel processes. Therefore, the application had to engage with its users and ensure that any difficulties in using the application be sorted out. Such interventions may not have been necessary in Morocco or Canada, where the application was entirely voluntary.

Some of the challenges in our study were the lack of organized and comprehensive information of CTAs and the inability to download several of them due to lack of accessibility outside of their respective countries. Hence our reliance on collection of data had depended on publicly available information. The rapidity in launch and usage of CTAs was the need of the hour and ability to inculcate the right processes or approaches to CTAs in some countries could have been unintentionally compromised.

Awareness and adoption of CTAs is no silver bullet in the battle against COVID-19 pandemic. In the last few weeks, announcements about COVID-19 vaccines is very encouraging but the citizens should not become complacent about the use of CTAs. In this regard, Govt. leadership must also offer assurance to its citizens that their identify will be concealed and emphasize the benefits of CTAs as it relates to shared public health.

10. Conclusions and Outlook

The current COVID-19 pandemic has shown the importance of expert-based decision making from governments, and highlighted the importance of developing a transparent and inclusive communication campaign. An excellent example of this is the development of COVID-19 smartphone apps, which are aimed at performing contact tracing at a massive scale. As discussed in this study, there are a number of technical solutions to deploy COVID-19 apps which are privacy-preserving, as well as effective when it comes to providing relevant epidemiological information to fight the pandemic. In many cases, the governments have not succeeded in conveying the message that decentralized digital contact tracing is not only able to fulfill all the ethical requirements, but it also provides invaluable information to contain the spread of the infection. This is particularly relevant in countries where less strict measures have been taken. For instance, it is surprising that Sweden has not developed a CTA, which could have significantly helped the lenient approach to the pandemic adopted by their Public Health Agency. Clearly, the current situation confronts the interests of governments, which want to gather as much information as possible to effectively tackle the COVID-19 spread, and individuals, who are concerned about their privacy and other essential ethical issues such as discrimination and stigmatization.

In this work we proposed an evaluation framework, the COVID Tracing App Scale (COVIDTAS), to assess the compliance of CTAs with all these principles. This framework builds on other ethical studies [41], which were based on the 17 Sustainable Development Goals (SDGs) of the United Nations (UN) [46], adding other aspects such as user review and ratings. This study features one of the largest sociotechnical assessments of CTAs, where 17 government-owned technical solutions are thoroughly evaluated. Our analysis indicates that the CTA by New Zealand reached the highest score (1.82), followed by the CTAs by Canada and Australia (1.67), and those from Japan and Germany (1.57). Some important comments regarding the CTAs that scored less than 1 are given next, i.e., the ones from Qatar, Brazil and Colombia. For example, the EHTERAZ tracing app from Qatar was legally enforced to the public, but it was not available in the local language. Interestingly, the CTAs that obtained low scores are generally located in areas with significant data gaps and digital divides [47], which are critical not only terms of handling the pandemic but also regarding a more sustainable global future.

The present framework can be used to analyze newly developed CTAs, so that the public can assess them and make informed decisions based on the compliance with the COVIDTAS methodology. Furthermore, we expect that a wide range of applications will rely on smartphone apps in the near future, ranging from economic issues to measurements of pollution. These apps may constitute an integral part of our lives, and we want to ensure that they will fulfill all the ethical requirements in terms of privacy and equality. Therefore, this framework can form the basis for the study of the future apps that will disrupt our societies. Future research in this area should also aim at gaining a more comprehensive understanding of the role of CTAs in managing this pandemic. In addition, as data stabilizes, more significant statistical analysis on each CTA can be carried out, identifying the best approach for data privacy, user trust, and disease tracking.