Using Citizen Science to Complement IoT Data Collection: A Survey of Motivational and Engagement Factors in Technology-Centric Citizen Science Projects

Abstract

A key aspect of the development of Smart Cities involves the efficient and effective management of resources to improve liveability. Achieving this requires large volumes of sensors strategically deployed across urban areas. In many cases, however, it is not feasible to install devices in remote and inaccessible areas, resulting in incomplete data coverage. In such situations, citizens can often play a crucial role in filling this data collection gap. A popular complimentary science to traditional sensor-based data collection is to design Citizen Science (CS) activities in collaboration with citizens and local communities. Such activities are also designed with a feedback loop where the Citizens benefit from their participation by gaining a greater sense of awareness of their local issues while also influencing how the activities can align best with their local contexts. The participation and engagement of citizens are vital and yet often a real challenge in ensuring the long-term continuity of CS projects. In this paper, we explore engagement factors, factors that help keeping engagement high, in technology-centric CS projects where technology is a key enabler to support CS activities. We outline a literature review of exploring and understanding various motivational and engagement factors that influence the participation of citizens in technology-driven CS activities. Based on this literature, we present a mobile-based flood monitoring citizen science application aimed at supporting data collection activities in a real-world CS project as part of an EU project. We discuss the results of a user evaluation of this app, and finally discuss our findings within the context of citizens’ engagement.

1. Introduction

In recent years, there are hundreds of Smart Cities applications being developed worldwide with themes such as traffic monitoring, parking, air quality monitoring, and transportation, to name a few. These applications mainly monitor and improve existing service-oriented infrastructure to enhance collaboration between different economic and social actors [1]. Different cities have different strategies and frameworks for Smart Cities, for instance, the London Smart City initiative (Smart London, https://www.london.gov.uk/ (accessed on 11 January 2021)) is based on four dimensions: (a) technology and innovation; (b) open data and transparency; (c) collaboration and engagement; and (d) efficiency and resource management [2]. The concept of a Smart City is being used to provide an efficient way to enhance the quality of life for the citizens with the use of big data and intelligent solutions [3]. Extant research suggests that citizen engagement is essential for developing smart solutions as citizens interact with their cities and neighbourhoods as a part of daily life. Citizens can also potentially support data collection efforts—in particular, where installation of sensors or IoT devices is impractical or expensive. This can be observed in the example IES (Internet-Enabled Services) Cities project which involves four European cities: Zaragoza and Majadahonda (Spain), Bristol (UK) and Rovereto (Italy). The project has developed a mobile application platform for citizens that allows accessing open data provided by the city council and other authorities. The application also contributes to collecting heterogeneous real-time data and can therefore enable citizens to be consumers as well as co-producers of data at the same time [4].

CS is often referred to with alternative terminologies such as Technology-Mediated Social Project (TMSP) [5], Public Participation in Scientific Research (PPSR) [6], community-based monitoring [7] and citizen participation and engagement [8]. CS is a scientific activity where citizens, government agencies, industries, academia, community groups and local institutions collaborate to monitor and respond to common issues such as environmental problems, wildlife protection and monitoring and many others [7]. CS has been implemented in a wide variety of themes such as wildlife, ecology, environment, climate change and natural disaster [4]. The most important feature of any CS project is citizens’ participation—often requiring many contributors to make a project successful. Citizens can collect data more efficiently than sensors in many realms of problem-solving such as image and sound detection [9]. Another advantage of citizen participation is citizens’ strong understanding of local context and challenges, which is often difficult to incorporate within sensor systems. Citizens could also offer innovative ideas and diverse perspectives that could help solve robust problems [10,11,12]. A citizen-centric approach is often applied in Smart Cities solutions and can play an active role in real-time data collection. One such example could be developing real-time mapping services, collecting field data from individuals using GPS enabled smartphones [13].

CS projects and methodologies encourage participation across a variety of spectrum ranging from just informing to consultation, involvement, collaboration and empowerment. With the increased digital footprint, participation can offer democratisation of governance on a larger scale than has been possible previously. The Aarhus convention (1998), signed by 40 countries (mostly from Europe), encourages the public to participate in decision-making to empower their citizen [14]. CS benefits the society or community with environmental democracy, social capital, involvement in local issues and scientific literacy.

In a citizen-centric approach, citizens act as sensors and data interpreters with distributed intelligence—where they help in capturing the data in situ as well as interpreting the data by often providing further context. The co-creation and collaboration approaches to CS also involve citizens in defining the problems or designing the experiments and interpreting results [15,16]. For example, in a flood monitoring scenario, deploying sensors at every potential location is not a feasible approach since blockages and water level rise can occur at different places at different times due to a variety of factors (e.g., drains blocked by objects). In such scenarios, citizens can provide data to help to fill any potential data gaps. Meijer and Suzanne [17] analysed 25 CS projects where citizens have generated data that were used for decision-making. Citizens’ inputs have been used as the data source mainly in mobile-based or web-based CS projects [18]. Engaging citizens as data sources provides a considerable variety as each individual can experience events differently [19]. With the enhancement of communication technology, citizens can be connected and communicated for the field assessment of any events [20]. This ability to assess any problem makes citizens a more reliable source of information in CS projects.

In this work, we define CS projects that use technology heavily to support citizens’ participation as “technology-centric”. It has been argued that emerging technologies have the potential to improve data collection, captivate a broader audience, motivate citizens on their participation and improve data quality and automation processes [21,22,23]. The use of technology could allow scaling the efforts to collect data for vital experiments and provide an engaging mechanism for citizens to take part, contribute and learn from the experience. At the same time reliance on technology could also be challenging in terms of keeping up the engagement and contribution going with the technology [24]. One of the central concerns in any technology-centric CS project is how to motivate and engage citizens to participate and support the data collection and interpretation of knowledge. The engagement in CS literature mainly only refers to the initial motivation for citizens to participate in such projects [25,26]. Nevertheless, citizens’ engagement is crucial throughout the CS project lifecycle. Considering these facts, designers of CS-based solutions can improve the chances of project success by thinking about what factors motivate citizen participation and how to use technology to support this. One of the major motivational factors for citizen engagement is the cognitive surplus, which represents the amount of time citizens spends contributing to activities they see beneficial to them and their community. The technology adoption literature shows that citizens adopt technologies that prove useful and not too complex to use [27,28]. For example, Textizen, a mobile and web-based platform currently available in many cities in the USA, allows citizen and local management agencies to interact regarding local issues. Textizen is a replacement for the conventional offline system of civic management such as town hall meetings or mail-based surveys. Technology-centred CS made it easy for them to contribute to a cause of vital importance [29].

This research is in the context of developing a flood monitoring CS application for a large consortium of European cities as part of a European Union-funded project—Smart Cities and Open Data Reuse (SCORE, https://northsearegion.eu/score/ (accessed on 10 January 2021)). The project involves nine cities and three universities to explore the use of IoT and big data technologies for solving challenges facing cities in water, environment and mobility themes. As part of the SCORE project, our research focuses on addressing challenges relevant to the water theme, in particular flood monitoring using a technology-centric CS mobile application. To support the design of an engaging application that motivates citizens in taking part and continue contributing is paramount for the success of our CS project. To guide our design, we carried out an extensive review of the extant literature on the existing approaches and applications on motivational factors to support the engagement with technology in technology-centric CS projects. These applications have been developed with different dedicated services under different projects by focusing on factors that motivate citizens to be engaged in (‘engagement factors’) CS experiments. Exploring different factors, we created a Pareto chart that shows how often these factors have been considered in technology-centric CS projects. The chart helps to identify what are the major factors that have been considered in diverse ways. The chart also guides on what are the major aspects that have been considered in other CS applications. The literature review is also summarised with the “Motivation and Engagement Factors Tree” which shows how different methods have been used. This tree combined similar names and methods under one common class of factors, essentially offering a hierarchy of motivational and engagement factors in techno-centric CS projects. To our knowledge, this is the first work that exhaustively considers factors that motivate citizens to engage in technology-centric CS projects, summarises them to guide other projects and demonstrates its use through a use case.

The rest of the paper is organised as follows. In Section 2, a comprehensive review of technology-centric CS projects and research literature is presented. In Section 3, details of system design and methodology of “Flooding Event Reporting Mobile Application” is presented. In Section 4, we present an analysis of a user evaluation using the System Usability Scale (SUS). We conclude the paper with some discussions, limitations and future work in Section 5.

2. Literature Review

The main objective of this comprehensive review is to elicit motivational and engagement factors from the extant technology-centric CS literature. Factors that motivate citizens to take part in CS projects have been explored by applying an inclusive assessment of CS projects. The review also highlights how the existing works build on such factors to maximise the engagement of citizens in citizen-driven CS projects. In this study, we reviewed citizen science literature from year 2000 to 2020 using a systematic approach. From our analysis, we note a gap in the literature on motivations and engagement factors for citizens in techno-centric CS projects (where technology such as mobile applications or online platforms have been used). New technologies such as the use of wireless sensors, multi-platform mobile applications, user-friendly online web platforms, social media and gaming promise advancement in CS in terms of the use of technologies to motivate citizen participation, improvement in data gathering and quality, and timely decision making [21,24,30]. However, the literature on what works well in engaging citizens in CS projects using such technologies is limited. The past two decades have seen a steady increase of CS projects and peer-reviewed articles—for example, the Zooniverse project has generated more than 50 peer-reviewed articles, while the eBird project is used in at least 90 peer-reviewed articles [31].

We used Arksey and O’Malley’s [32] methodological framework for systematic scoping review, which is considered a good model for a comprehensive systematic review. The research question of this study is: What are the motivational and engagement factors considered by extant literature for citizen participation in Technology-centric CS projects? Our systematic review consists of five stages: (i) identification of the research question; (ii) identification of the relevant studies; (iii) inclusion and exclusion of studies based on the research questions; (iv) screening the data; and (v) paper review and analysis. To perform an extensive and comprehensive literature gathering, four widely used digital libraries were selected: (i) online indexing services (Google Scholar); (ii) publisher database (science direct); and (iii) specific journal related to CS (e.g., peer-reviewed journal Citizen Science: Theory and Practice, and plos|one). Four contexts, as listed in

Table 1. Review keywords.

Context	Keywords
Activity	“Motivation” OR “Participation” OR “Engagement” OR “Citizen Engagement” OR “Motivating Citizen”
Community Context	“Citizen Science” OR “Citizen Engagement” “Crowdsourced” OR ”Community Based“ OR “Volunteer Monitoring”
Environment Context	“Hydrology” OR “Flooding” OR “Citizen Science”
Technology Context	“Smartphone” OR “Mobile” OR “HCI”

, were applied with corresponding keywords in combination for the most relevant literature collection. The literature that covers citizen motivational and engagement factors in any citizen science projects in the context of the environment (water, resource management, natural resources, conservation, species, wildlife, ecology, hydrology and biodiversity, marine) involving smartphone or mobile apps or any online platforms were considered for the review.

Science direct, Google Scholar and specific journal libraries (Plus|one (https://journals.plos.org/plosone/ (accessed on 11 November 2020)) and Citizen Science: Theory and practice (https://theoryandpractice.citizenscienceassociation.org/ (accessed on 15 November 2020)) were used for the exploration of the articles for the review. The selection of articles is summarised with the flow diagram shown in Figure 1. This flow diagram shows how many articles appeared during the search at first, and how the number of 228 articles were selected for the review.

2.1. Motivation and Engagement Factors in Technology-centric CS Projects

To involve more citizens in CS efforts, it is necessary to understand what makes people get sufficiently motivated to participate in such projects and what sustains their motivation to continue contributing. Existing research highlighted factors that motivate community involvement [33]. In addition, communication with the citizens about the impact of their work has been an important motivator for investing time and efforts [22]. In addition to these factors, other motivations also exist—we expand on these motivational and engagement factors in the following subsections. It is important to note that these factors have been examined in the context of both participation in a bigger cause and allow how they are used in the development of the CS tools such as mobile app and online platform.

2.1.1. Theme/Cause

One of the key motivating factors identified in the literature is the ‘theme’ of such projects where certain themes do attract higher engagement and participation than the others [34,35,36,37,38,39,40,41,42]. Research shows that citizens have a high interest in nature-related CS projects than in social or career-related projects [41]. The literature also highlights common motivational themes focused on helping the environment, learning, value and esteem, career, social and outdoor activities [43]. These findings are consistent with other studies as well which identity ‘helping the environment’ theme as the strongest motivator [44], which was also noted as the highest-scoring motivators for environmental volunteers [45]. As such, prior interest on specific thematic areas can influence how to engage citizens in a CS projects [34,35,36,37,46,47,48,49,50,51,52,53]. In other words, the theme of the application is important for the citizen to get engaged. If citizens already have particular goals in mind or concern about supporting any specific causes, they are likely to be motivated as they seek a platform to contribute. Topically relevant themes such as global warming, air quality, water quality, nature local community and contributing to science are the most widely reported themes.

2.1.2. User Interface

One of the most important factors contributing to effective interaction between computer or mobile device and humans is the user interface (UI) [54]. Studies suggest that simple information and easy to use interface are an effective way of encouraging wider public participation [55]. Learnability is an integral attribute of usability—the easier it is to learn, the easier it is to use [51]. Engagement is often initiated by the aesthetic look and the novel presentation of the interface, and it is sustained when the user attention and interest are maintained in an application [56]. In CS projects, user requirements for data presentation are generally different and the application must be accessible to diverse user communities. It is also common understanding that users generally prefer to see meaningful and processed information rather than raw data. They also prefer access to information to be simple and not time-consuming, while the process of registration being easy. Some works also report that no registration required to access an application leads users to participate more [57]. It is possible to improve user contribution and participation by more than 60% by not obliging them to officially sign up [58]. The importance of UI can be better realised by analysg popular social media such as Facebook and WhatsApp where the UI has been one of the main reasons for being adopted by many people as those social media provide easy to use UI for communication and exchange of views, easy to access, user friendly, simple and intuitive functionalities [59].

At the same time, the involvement of citizens in the design of the application is a significant motivating factor [60]. Cultural and environmental attributes, as well as UI, must be tested in broader use case scenarios such as people with different digital abilities. Reducing the chances of user errors by providing drop-down lists, or allowing offline data capturing are also important in CS projects [61]. The importance of UI has been highlighted in many other projects in terms of the interface and wider citizens’ participation [47,48,56,57,62]. These works emphasise that a custom-built application can capture all major aspects of the citizens’ experience while using the application.

2.1.3. Feedback

Feedback is also highlighted as one of the key motivational factors for CS projects [43,46,47,48,49,50,51,52,53,56,57,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76]. Feedback could be provided to users in many forms such as impromptu acknowledgement of the report or data, status or updates on the issues or reports, the outcome of the report or data, how the data are being used, how useful have the data been and how their contribution affected the cause, services or research as part of a CS project [26]. Participants prefer feedback on their contribution and data and adequate feedback mechanisms can help create prolonged engagement for the citizens [46]. Feedback is essential in any means of CS applications such as mobile applications, websites, the publication of results or CS-related communications [77]. Feedback is also reported to be increasing quality, accuracy of contributions and volunteer retention [78], while at the same time it also brings the opportunity for citizens to participate in activities that help with smart innovations. Citizens want to experience the “practical application of ‘smartness’ to their daily lives [79]”.

Feedback plays an important role in keeping the participants active on technology platforms and acts as a way to exchange knowledge, hence also result in the empowerment of citizens [80]. Lack of timely feedback or response has been noted as one of the main reasons for participants getting demotivated and stopping contribution [81]. On the other hand, providing feedback to participants in return for their contribution was reported to result in increased user participation [82]. Continuous feedback to participants about submitted data, data needs, purpose and importance of data is considered vital for long-term participation [83]. Feedback offered to the participant can be in different forms—general feedback, contributory feedback or co-creational feedback. General feedback has been mostly found in applications where advice needs to be given. The two other types of feedback, contributory feedback and co-creational feedback, are always from the citizens who are heavily involved in the citizen science projects and suggest some new features or ask for some improvement in the technology [63].

2.1.4. Reward

Reward is another motivational factor that can be in the form of recognition, acknowledgement, attribution, ratings, points and many other forms. Recognition is the act when an individual’s contribution is singled out and credited for their engagement in data collection or data analysis within an application [26]. Recognition is a crucial motivational factor for all citizens for their contribution as reported in several works [47,52,59,64,65,66,71,75,76,84,85]. For example, in Foldit (https://fold.it/portal/ (accessed on 8 December 2020)) and Eyewire (https://eyewire.org/explore (accessed on 11 April 2021)) applications, citizens pointed out recognition as a motivational factor for sustained engagement. Getting published in any article or journal papers has also been considered as an extrinsic motivation factor for the reward [86]. Another study shows that online public acknowledgement or recognition helps to increase the amount of contribution and motivation [87]. Citizens need to be recognised for their contribution whether it is small or big through some ways of attributions, for example, listing contributors by quality or quantity. Another study suggested that users like to see the rating of other participants to get reliable information about their neighbourhood [88]. A study of Amazon author reviews found reciprocity to be positively related to the quality of the review [89]. It is the personal desire of one to be recognised or identified by their community or neighbourhood, family, friends or colleague that has an impact on continued contributions on tech platforms.

2.1.5. Personal Benefits

A citizen can gain skills, knowledge and improve their network by taking part in CS projects. Such personal benefits are considered leading to broader environmental and social benefits in the shape of increased input [39,51,59,85,90,91,92]. One of the important motivations for the citizen is to report any emergency and risks they might encounter, for example, floods or landslide. In general, citizens want to provide evidence as reports or photographs which allow them to work with city authorities regarding their needs. In a case study presented by Amsterdam University of Applied Science, many citizens participated to be informed about good and bad things happening in their vicinity so that they could carry out activities with their children in safer surroundings [59]. The study reported that some of the participants were reluctant to receive all the notification from local authorities, but they were comfortable getting personally relevant notification from an app such as reports about their streets. This reflects another aspect of citizens’ interest in data contribution where citizens are often focused on their immediate context. One study suggests that personal motivation is the second most important factor for active participation from citizen only after the participants finding any sort of benefits [93]. It is evident that laypeople contribute with data in CS projects’ main agenda to pronounce their concern in a way that can be heard, acknowledged and accepted by the concerned authorities. Many citizens contribute to CS because of their commitment to the city and their wellbeing. Studies also suggest that the majority of the citizens’ motive to participate was to enable their voice to be heard [59].

2.1.6. Social Bonds

Humans have the desire to build social bonds. One of the important aspects of developing any application for citizens is to address social bonds through the application. A CS platform should allow users to interact, share, like and comment on the application as a common platform. Such community-oriented functionality will enhance sustained participation [43,53,56,61,90,91,94,95,96]. A CS platform with the opportunity to form a social bond can increase citizens’ motivation and engagement. Such a community-oriented functionality will enhance sustainable citizens’ participation in the project as the user would like to stay connected to like-minded other citizens in real-time and share observations, concerns and questions [97]. Active communities supported with social activity can engage citizens in CS projects longer, hence increasing the project life [52]. In a survey paper of five different CS projects, one of the most important and common features in all five platforms was social interaction facilitated by forums, chat rooms and other social network tools. These platforms were also successful in serving as a community-building tool to allow collaborations, exchange of ideas and strategies, learning opportunities, social relationship and enjoyment [67].

2.1.7. Gamification

One common definition of gamification is the use of elements of game design in a non-game context [96]. Several studies recommend that the use of awards, points, badges or leader boards are effective to incentivise citizens and keep them engaged [81,88,98]. It is one of the key factors in CS projects that have been used to motivate citizens to increase their contribution and participation. Amusement, play and entertainment as a form of social interaction motivate and encourage citizens’ participation. The game factors in an application need to be applied to make activities fun, socially engaging and to encourage competitive and cooperative participation. One of the CS projects suggests point-based reporting where the citizens gain points on completing a task, such as providing a report on an issue from their area [99]. At the same time, other users vote for or against the report, hence gaining more points for the act of reporting. A different example of a Bio-tracker mobile application shows that citizens who were motivated and attracted by the gaming element have the motivation to participate in conventional CS campaigns such as contributing to science or helping the community [100]. Another example of gamification is the converse website, which allows users to create messages from photos of galaxies that exist on the platform that resembles letters [101]. However, many researchers suggest that gamification is not always successful in attracting a new audience but is more helpful to sustain the existing audience in the CS project over time [102]. In another example based on EyeWire, citizens appreciate entertaining and competitive interfaces to contribute to a worthy cause where complex tasks are divided into small tasks using gamification [103].

2.1.8. Sense of Community

‘Sense of community’ characterises relationship between an individual and a social structure. It plays a catalytic role in influencing individual participation in any intended neighbourhood project. One of the strongest motivations for environmental helpers is the desire to give back to their community [104]. Citizens have myriad reasons to participate, but the social aspect of being part of a community always applies [33]. Benefiting the community is a common practice in CS projects [105]. Sense of community is mostly practised when a community is facing some serious issue, to bring citizen together or to empower the community members. The community collaborative rain, hail and show (CoCoRaHS) was a small community-based project started in Colorado after heavy rainfall in 1998. The accidental network originated is still operational in the United States and Canada with tens of thousands of logs generated daily [37]. Community-based motivational factors were used for counting malaria cells, which had the implication that the high number of participants increased the quality of the task [106]. For citizens’ engagement, being part of a community is very important and the project platform must aim to make the participant feel integral to a community [55]. It plays a vital role in driving motivation in any intended local issues and causes [28,29,30,46,57,61,68,69].

Sense of community is a core element for many CS projects; for example, PathFinder, a project to create online collaboration for citizen scientist, has grown into an online community focused on education [75]. Despite the nature of a CS project, whether it is local or global, to build a community, platforms must enable the user to share their findings, ask questions, have discussions and facilitate them [37]. With the advancement of social networking, a citizen scientist does not need to look locally for a sense of community, rather they can look farther to the global community. The fact that building communities leads to increased participation and engagement is one of the key reasons to have online communities in many CS projects. A survey by Wilson and Musick [107] showed that citizens are more willing to volunteer if they are more social.

2.1.9. Trust

Trust in terms of privacy in the use of the data provided by citizens is one of the potential barriers in engaging citizens to use CS applications. For achieving a broad and inclusive engagement, trust is the most important factor [68]. One of the studies showed that around 76% of participants rated the trust as one of the main participatory factors [59]. In addition, generating the belief and understanding the value of their contribution is also a key factor [48,50,53,59,61,68,74,108,109]. “Why engage in a CS project when citizens start believing that nothing is going to change?” The question of trust in any CS project is real. It is the responsibility of any CS project organiser to instil trust by a clear declaration of how, when and why citizens’ data are used [110]. From the scientist or government authorities’ perspective, trust in the data provided is extremely important (similar to how it is important from the citizens’ point of view that data are used for its intended purpose). Questions such as “who controls the data?”, “who owns it?”, or “who is the beneficiary?” are among the key questions that need to be addressed to establish trust. Updates and presentation of open transparent data will motivate citizens to participate as they witness the system is trustworthy and they are contributing to a common cause [76]. Another study suggests that the citizens of the community should feel that they have benefited from the application even if they do not contribute data [53].

2.1.10. Altruism

Altruism is one of the primary motivations for citizens who like volunteering. Altruism is defined as ‘helping others, working for a cause, serving the community or a selfless motive’ [111]. Often one of the important motivations to take part in CS projects is the welfare of others. It is suggested that personal circumstances are directly linked to their social context and play an important role in affecting participation and motivation [112]. Research also suggests that citizens contribute because it enhances their reputation and they enjoy helping others [113]. A study of Amazon consumers’ reviews suggests that altruism and reciprocity are directly related to the quality of contribution [69]. A study involving Tomnod, a CS platform, showed that altruistism motivation such as helping people and the environment is important to all demographic groups [112].

To help any individual or group of people is a positive contribution, but it is only practical when one thinks the problem being solved is important and interesting. For example, when a scientist, Jim Gray, was missing during a sailing trip to the Farallon Islands near San Francisco in 2007, thousands of volunteers visually inspected 560,000 satellite images hoping to find his location [114]. Although the efforts were unsuccessful, it demonstrated that volunteers do spend their time and efforts for the right cause. Selfless actions to help others, the community or any cause is one of the main motivations to volunteer [18,28,65].

2.1.11. Curiosity Drive

Curiosity is intrinsic motivation that motivates people to explore and contribute for their own sake [47,56,73,86,98,115]. Curiosity is an important aspect in the development of knowledge and competence. The curiosity drive assured that the application has features that help citizens to explore their surrounding and environment [116]. Curiosity is considered an internal motivation for seeking information. According to Malone’s work [98], curiosity is one of the three major factors for user engagement. In their study, the authors of interviewed 28 participants to find the reason for joining a citizen science project; curiosity was reported to be one of the motivations. Any CS platform should address the user’s knowledge gap when they realise that their knowledge structure is inconsistent or incomplete [66]. Another study presented a thematic map of motivation where curiosity drive was again considered as one of the three major factors for initial participation [117]. Curiosity drive improves the citizen’s performance and learning throughout the CS project life [118]. In a survey relating to motivation to participate in the Foldit project, many participants tried the platform after their curiosity was aroused by the media [85]. In another study, it is argued that citizens’ engagement is composed of three elements: attention focus, intrinsic interest and curiosity [119]. In separate study [98], it has been argued that publicity can induce curiosity drive. The more the project is discussed or advertised, the greater is the chance of citizens getting curious about it, hence enabling increased participation.

2.1.12. Psychological Empowerment

Psychological Empowerment (PE) could be described as the connection of the sense of personal competence, the desire for and the willingness to take action in the public domain. An empowered citizen is active in community organisation and activities. It is the process when individuals perceive control and mastery over certain issues and believe their actions or decisions will have a definite change or impact on an issue [14,29,74,75]. It is thought to be a process by which individuals gain mastery or control over their own lives and democratic participation in the life of their community [120]. PE requires a contextual analysis to be fully understood. For example, an empowered person may have no real power in the political sense but may have an understanding of what choices can be made in different situations [121]. Another study postulates empowered individuals are likely to be active in community organisations and activities [111]. PE can be observed in the tool WeLive [122], which provides a framework for the co-creation of the application. In this framework, citizens, companies and public administrations share the same platform where the citizens can present their ideas.

The impact of three different principles of PE was evaluated in one of the studies on a public transport system. Sixty-five participants were involved where each participant was allocated a specific principle of PE in terms of self-efficacy, sense of community, casual importance and the control condition. Mobile text messages were used as a mode of communication and every category had its motivational text. Out of 65 participants, 30% of the participants were categorised into ‘self-efficacy’, 25% of the participants into the sense of community, 27% of the participants into casual importance and 18% of the participants into control condition groups [123]. According to this work, the change of the motivational approach had significant effects. First, those who received motivational messages were more likely to send suggestions rather than complaints. Second, those participants provided more reports and participated frequently. Allowing citizens to participate, suggest, design and create their solutions helped the resulting services with improved and faster acceptance with end-users as well as gaining a greater sense of empowerment and ownership [124]. PE is a vital element and a fundamental tenet to direct citizen participation, as they can be empowered in their role as a consumer, customer or volunteer, which requires direct involvement in problem-solving and decision making about citizens’ problems [125].

2.1.13. Publicity

Publicity is among the factors that indirectly affect citizens’ engagement towards a project. With the publicity of a project in any newspaper or social media helps in increasing participation [126] For example, in the Zooniverse project, when the project was publicised through television, radio and social media, citizens’ motivation and engagement to the project increased significantly [127]. While publicising a project to attract citizens, it is important to highlight the benefits and diversity of the application [128]. The benefits of the application motivate citizens to contribute over the long run. However, there is no one-fit publicity for all—different means of publicity are required to motivate target audience [129].

2.1.14. Situatedness

Another motivational factor is to attract people in-situ. In many scenarios, it is relatively easier to gather information, opinion or data on issues directly co-located. To evaluate the impact of situatedness in collecting feedback and engagement, a case study was carried out in a busy town centre where renovation of pavement was underway which affected businesses and pedestrian flow [106]. Twelve public displays were used to gather public feedback on co-located issues. Among them, five displays were placed near the construction site and seven displays were far away from the location. The public displays right next to the construction site elicited higher contribution. In another study, it was suggested that some CS projects failed to motivate as the information delivered was not related to the everyday environment or experience [130].

2.1.15. Extrinsic Motivation

Extrinsic motivation is another factor that could be set whenever a CS activity is performed to achieve distinguishable outcomes. It explains the degree of autonomy a citizen can experience in CS projects while being engaged in the project. The concept of ’extrinsic motivation’ is also known as “self-determination theory”, which states that citizens are being personally recognised in some way [85]. It is significant when citizens feel the connection in a wider sense with family, community or governing body. This factor has shown to increase citizens participation in combination with gaming [131,132] which often combines rewards. However monetary rewards alone have been found to have a negative impact on participation, often refereed as ‘crowd-out effect’ [87]. Rewards can be advantageous in enhancing extrinsic motivations, but they should be combined with bringing positive change for citizens.

2.1.16. Contextual Cues

Research suggests that the use of contextual cues or digital cues play a role in motivation as our memory is a reconstructive process mediated by day-to-day events [133]. Contextual cues help to relive our past specific experiences and improve past experiences and trigger episodic memories [134]. Goncalves et al. [135] described contextual cues in their study where participants were provided with different conditions: control conditions with no contextual cues and with contextual cues. Their task was to take pictures and report inaccessible locations such as streets and places in the city. Participants contributed significantly more reports with contextual cues as compared to no contextual cues. Contextual cues can work as catalyst and participants sometimes through such cues relate themselves to something that is reported and go through the process of recollection of past experiences.

2.1.17. Crowd Effect

In the literature, few researchers have mentioned the crowd as a source of motivation to participate. It is an important factor and is reported to have high impact on initial participation [53,108]. This is also linked with trust as a general practice; when citizens download any mobile application from Google Play or any other platform, they often check the number of downloads for that app and the reviews. In the context of CS, if a user senses being a part of a larger community and their participation contributes to a better community or society, the sense of ownership of the initiative triggered by the “crowd” can evoke social responsibility and encourage them to continue to participate in the project.

Key motivation and engagement factors of citizens’ participation in CS projects are reviewed and the corresponding CS projects and applications are identified, as listed in

Table 2. Summary of CS projects and Apps.

Motivational and Engagement Factors	Papers	CS Projects	Number of Projects/Papers
Social Bonds	[37,39,62,68,69,71,75,76,90,94,95,96,108,109]	Great Pollinator Project, Feederwatch, iSpot, iNaturalist, Zooniverse, Luminous Blanket, Natural Fuse, Open Burble, Marling, Thingful, Pachube, Air Quality Egg, Safecast, Smart Citizen, Radiation watch, AoT, WeSenseIt, Onnocentive.com, ATIZO.com, IdeaStrom.com, CoCoRaHS eBird, Foldit Galaxy Zoo OPAL, Stardust, Stardust@homei, CWOP, BOINC, Budburst, Nestwatch, Monarch larva, Cocorahs.org and Gcmonitor	32 projects, 14 papers
Feedback	[43,46,47,48,49,50,51,52,53,56,57,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76]	Prourbe, iSpot, iNaturalist, Zooniverse, Sense the Air, Digital heritage, Natureblick, iSPEX, nQuire-it, Smell Pittsburgh, FWC Reporter, Hydrology monitoring Kenya, Onnocentive.com, ATIZO.com, IdeaStrom.com, CoCoRaHS eBird, Foldit, Galaxy Zoo OPAL, Gamified Mobile, CleanUp App, AirQuality app, Love your city, Biotracker DoGood and Täsä	24 projects, 27 papers
Gamification	[62,68,71,73,81,84,88,91,96,98,108]	iSpot, iNaturalist, Zooniverse, Ctd Bristol, Luminous Blanket, Natural Fuse, Open Burble, Marling, Thingful, Pachube, Onnocentive.com, ATIZO.com, IdeaStrom.com, gamified mobile participation prototype, Stardust@homei, CWOP, BOINC, Love your city, Biotracker, DoGood, Täsä, Budburst and Kijk Mobile app	23 projects, 11 papers
Sense of Community	[39,46,49,52,53,61,63,69,90,96,109]	Nestwatch, Monarch larva, Cocorahs.org, Gcmonitor, Budburst, CoCoRaHS eBird, Foldit Galaxy Zoo OPAL, WeSenseIt, Air Quality Egg, Safecast, Smart Citizen, Radiation watch, AoT, nQuire-it, Digital heritage, Wildlife Trackers, Pygmy, Hunter-gatherers forest monitoring in Brazil, Sense the Air, Creekwatch and Prourbe	21 projects, 11 papers
Trust	[50,61,62,65,74,106,108]	Prourbe, iSpot, iNaturalist, Zooniverse, Luminous Blanket, Natural Fuse, Open Burble, Marling, Thingful, Pachube, Smell Pittsburgh, FWC Reporter, iNaturalist.org, iSpotnature.org, Floracaching, NatureNet, Welive and 2Loud	18 projects, 7 papers
Theme	[34,35,36,37,38,39,40,42,48,49,50,51,52,53,65]	Great Pollinator, Creekwatch, Wildlife Trackers, Pygmy, hunter-gatherers forest monitoring in Brazil, iSPEX, nQuire-it, Smell Pittsburgh, FWC App, Crowd Hydrology, Stardust, Nestwatch, Monarch larva, Cocorahs.org, Gcmonitor, WeLive and Zooniverse	17 projects, 15 papers
User Interface	[46,49,50,57,61,62,63,64,66,69,84,109]	Prourbe, iSpot, iNaturalist, Zooniverse, Sense the Air, Digital heritage, nQuire-it, Smell Pittsburgh, Air Quality Egg, Safecast, Smart Citizen, Radiation Watch, AoT, Hydrology monitoring Kenya, CoCoRaHS eBird, Foldit Galaxy Zoo OPAL and WeLive	17 projects, 12 papers
Personal Benefits	[39,50,53,61,66,75,90,91]	Great Pollinator Project, Wildlife Trackers, Pygmy, hunter-gatherers forest monitoring in Brazil, Prourbe, Smell Pittsburgh, Hydrology monitoring Kenya, WeSenseIt, Foldit, iNaturalist.org, iSpotnature.org, Floracaching, NatureNet, Nestwatch, Monarch larva, Cocorahs.org, Welive and Gcmonitor	17 projects, 8 papers
Reward	[37,39,49,52,66,68,71,76,84,85]	Prourbe, Creekwatch, nQuire-it, Hydrology Monitoring Kenya, Onnocentive.com, ATIZO.com, IdeaStrom.com, Stardust, Stardust@homei, CWOP, BOINC and Kijk Mobile app	12 projects, 10 papers
Crowd	[53,108]	Luminous Blanket, Natural Fuse, Open Burble, Marling, Thingful and Pachube	7 projects, 2 paper
Altruism	[48,53,75]	Wildlife Trackers, Pygmy, Hunter-gatherer forest monitoring in Brazil, Great Pollinator and iSPEX	5 projects, 3 papers
Psychological Empowerment	[35,41,49,71]	nQuire-it, Stardust@homei, CWOP and OINC	4 projects, 4 papers
Contextual Cues	[41,63]	iSpot, iNaturalist and Zooniverse	3 projects, 2 papers
Curiosity Drive	[33,49]	nQuire-it	1 project, 2 papers
Situatedness	[41,63]	Sense the Air	1 project, 2 papers

. Analysing these CS projects and application, as shown in Table 2, it can also be found that some of the factors such as “Feedback”, “Theme” and “Reward” are considered in many CS projects, whereas factors such as “Crowd” and “Curiosity” are considered only in a few CS projects. This reflects suitability of these factors in the CS projects where they have been considered. Table also serves as a guideline for any new technology-centric project to cross-reference factors they are considering with projects and applications where they have been applied.

2.2. Motivational and Engagement Factors Tree

Following a rigorous analysis of different CS projects and related literature, we identified and presented 17 motivation and engagement factors in Section 2.1. These factors do not always appear with the same name in different CS projects—based on a variety of factors (domains, applications and personal preferences), different terminologies are used to represent similar concepts. To simplify these naming variations, based on the literature, we present our motivation and engagement factors tree in Figure 2.

The figure shows what are the different names and methods that have commonly been used for the same factors. From the tree, we observe that there has been a wide range of variation in element or method naming for the factors “User Interface”, “Feedback” and “Theme”, whereas factors such as “Crowd” and “Curiosity Drive” have been used without any variation. The variation also depends on how often it has been considered while implementing CS projects. For example, many CS projects, e.g. Foldit, iSpot, iNaturalist and Zooniverse, have their specific area of application: science, nature and animals. We categorise the area of the application for CS projects under the branch “Theme”. Citizens’ response to CS activities is referred with different alternatives such as ‘contributory feedback’, ‘co-creational feedback’, ‘reviews’, ‘acknowledgement’ and ‘updates’. We categorise these element or method names under the branch “Feedback”. In the tree, each small circle represents one motivation and engagement factor, whereas the elements in the rectangle boxes represent different element or method names that have been used in different CS projects.

2.3. Motivational and Engagement Factors Pareto Chart

In analysing how important the motivation and engagement factors are for CS projects, we found that very few projects emphasised their network to establish citizen’s engagement from the early stage of the project covering the technical aspects. Very few studies from the outset discuss motivational and engagement factors and dedicated strategy to address these factors to encourage sustained citizen participation. Fifty-eight per cent of the research work surveyed in our review discuss CS projects and the motivational factors or the barriers for user participation and engagement, methodology, outcome and the design guides in a real CS project. Thirty-four per cent of papers discuss multiple CS apps or projects and 52% cover and discuss motivational factors directly, but it was still not comprehensive to specific projects or a specific theme.

Based on each factor being considered in the CS projects, we plotted a Pareto chart, as shown in Figure 3, that shows how each factor combined to give a cumulative combination on covering all the factors. From the plot, it can be observed that 50% of the research papers mentioned “Theme”, “User Interface” and “Feedback” explicitly as the motivational factors in their work. We describe all the important factors which can work as a checklist for designing a CS platform or app regardless of the project theme. Out of the 17 discussed factors, the 10 most common factors (Theme, UI, Feedback, Reward, Personal Benefits, Social Bonds, Gamification, Sense of Community, Trust and Altruism) cover almost 93% of the research papers reviewed. This plot also summarises the most common and rare factors that have been considered in CS projects.

3. Flooding Event Reporting Mobile Application: System Design and Methodology Using Motivational and Engagement Factors

We have utilised the findings of our literature review to develop a mobile application for flood reporting. This work was carried out as a part of the SCORE project where the main objective of the project is to develop Smart City solutions in the themes of mobility, water and the environment with a focus on CS, IoT and Open data. Reporting flooding issues through the application can help support decision-making processes within local authorities. Additionally, the mobile app also provides an alternative sensor network across the city to collect data, indicating where data is missing due to low coverage. The main purpose of this application is to enable flood-affected communities to report any issues or potential problems in the short or longer term (such as ‘blocked drainage or gullies’, ‘rising river level’, ‘water level rise in house cellars or basements’ and ‘river overflow’). Our design of the application aims to utilise these factors with the broader aim of improving the collaboration between citizens and local councils. Citizens, affected by flooding in the past, from two communities from Bradford Metropolitan District Council and Aberdeen City Council were consulted for the design and prototyping stages of the application.

3.1. Citizens Consultation and Application Conceptualisation

The work we present focuses more on community involvement and new inputs to motivate, engage and accommodate all the demographic factors, and it is underpinned by Human Computer Interaction (HCI) research. Citizens are the main users of this application to report any flooding incidents from their local areas. In a technology-centric CS project such as ours, one of the goals is to develop a piece of technology that allows involving communities in environmental challenges for data collection including capturing context, and at the same time raising awareness of issues. As a part of this, the most challenging task was to come up with a mobile app to generate intrigue, engagement among communities. For the design of this application, three workshops and meetings with around 20 citizens from flood-affected regions from the two councils were organised. Based on the initial analysis, we designed three prototypes of our mobile app, called “PhenoTracker” (based on the concept of ‘tracking phenomenon’), and demonstrated them to citizens. The feedback and suggestions from citizens on the initial prototypes were collected, analysed and used to develop the mobile app. We observed that citizens were very enthusiastic during design workshops and offered constructive feedback. Many of the participants found the app prototypes intriguing and fulfilling their curiosity drive. The following sample notes demonstrate some of the feedback we received for various aspects of the application. The complete notes are presented in Appendix A.

• “How the management agencies respond to our reports and how the data is used”?
• “We want to know what’s happening to our data”
• “We like to know what’s happening in the surrounding and the map view fulfil our knowledge gap”

3.2. Mobile App Design and Development

From our survey of the literature, we observe a significantly growing number of citizen science-based applications. However, research on how the factors affecting engagement of citizens can be incorporated into technology-centric CS projects from the initial design stages is limited. To overcome this barrier, in this work, citizens as potentioal user of the app were engaged at each stage of development. The application features and UI designs were developed based on citizen’s input grounded in motivational and engagement factors from the literature presented in this work. From our literature survey, we note that all the features that we have presented and encorportaed in our app have not been collectively applied in a single technological solution. The Pareto chart presented in Figure 3 illustrates that the inclusion of as many features can cover a wider dimension of the application. In developing our application, we incorporated the following factors to enhance usability of the app.

Theme: In our CS project, ‘flooding reporting’ is defined as the primary theme of the application. This theme, as a solution domain, is defined by two local councils, Bradford Metropolitan District Council and Aberdeen City Council, collaborating as a partner organisation in the SCORE project. Flooding event reporting is set as the theme of the app in our project (Figure 4a). In the context of the work and consulting outcomes from the local communities, we find that flooding is a very strong motivating theme, particularly given the susceptibility to flooding for some of Bradford and Aberdeen geographical areas. This theme includes some of motives required for community involvement, e.g., altruism, collectivism, etc. The flooding theme is also a strong theme according to the statistical information on UK flooding. Around 2.4 million UK properties are at risk from fluvial and coastal flooding each year, while 2.8 million are susceptible to pluvial flooding. One in six properties in the UK is located in areas where there is a quite significant risk of flooding. As many as 40% of businesses never reopened after suffering from flooding [20].

User Interface: The design and layout of each component of the application are equally important. Ease of use of the application was the key aim while finalising the interface of every feature of the application. The application was designed with wider age groups starting from young to senior citizens. Considering this, minimal user input from the keyboard is expected throughout the interaction with the app. Features such as: drop-down input (Figure 5b), speech-to text-functionality to reduce typing (Figure 5a) and auto-location assignment (Figure 4b), are implemented in the app.

Feedback: Feedback plays a significant role in citizens’ continued engagement with technology. Feedback could be in many forms: impromptu acknowledgement of the report or data; general feedback; status or updates on the issues or reports; the outcome of the report or provided data; and how the data are used, how useful are the data provided and how their contribution affected the cause or research [26]. We address feedback factors in some possible areas such as the status of the submitted report (Figure 7c) and acknowledgement on submission of the report (Figure 5c).

Personal Benefit: The application has been designed and developed in coordination with the citizens from the communities who are already affected by flooding in the past and see flooding as a threat. Users of the application can feel the personal benefit in terms of reporting any potential flooding incidents from their surroundings. In addition, the report list view and alert notification further bring the personal benefits, where they can stay updated about evolving flooding situation in their neighbourhoods (Figure 6b,c).

Social Bonds: One of the goals of this application is to create social bond among citizens who suffer from flooding. In this application, we designed the app by organising workshops with communities where citizens were asked if they do like to share their reports on social media, and would be interested in viewing reports and comments by others. Citizens were interested to have such features in the applications. While these features are available in a limited sense in our current version, they will be made operational in a future release (Figure 6b,c).

Sense of Community: From our workshops and interactions with citizens, it was noted that having a sense of community mattered the most to the citizens. Most of the participants mentioned the importance of awareness in the community about flooding incidents and they were interested to see reports coming from their area. To support sense of community through our app, we applied default map loading with user’s current location (Figure 4b) and list the reports based on the nearest report first (Figure 7a). Having these features in the application helps users to visualise reports from the their community.

Trust: We implemented trust in terms of privacy. A full ethical approval with the University Ethics panel was carried out that examined all aspects of the application – including data, and interaction. No personal data are being stored through this application and citizens can voluntarily delete their user account without any obligation (Figure 4c). In addition, we applied face blur features on images before the submission of reports in case of accidental capture of faces (Figure 7c). This feature ensures no personal identification of individuals is captured. To ensure trust when a citizen submits any report, the application is linked with the real-time report processing web application of the council to allow the council to see the reports as they come in real-time. Any change in report status initiated by the council will appear instantly (Figure 7c). Besides these, all the details used in this application are in compliance with GDPR. The users of the app are informed about this when they register for the app and when they provide informed consent.

Altruism: Selfless actions to help others, the community or any other cause is one of the main motivating factors to volunteer. Any citizen using our app is free to report any flooding incident which could help their community. Making their community aware allows citizens to demonstrate altruistism behaviour in a community. Reporting an issue to avoid any hazardous circumstances allows citizens to make a selfless contribution to their community. (Figure 4a–c and Figure 5a–c).

Curiosity Drive: To bring citizens’ curiosity towards using this application frequently, a map view of the reports/issues (Figure 6b) and alerts from the local council (Figure 6c) are provided. The alert coming from the local council with brief details on any event in term of notification to the citizens helps increasing curiosity drive.

Psychological Empowerment: With this application, citizens can report any flooding incident from their area whenever it comes to their notice. The timely reporting of any flooding incident which has the potential to damage or affects the community gives the feeling of physiological empowerment to the citizens. The contribution by posting the report will not only help their own local community but also the local council (Figure 4a–c, Figure 5a–c and Figure 6a–c). Furthermore, citizen-generated reports can also support councils in taking critical decisions in planning future maintenance or design interventions.

Publicity: Advertising, marketing and awareness on certain issues over media, social media or meetings helps or drives citizens to participate, and it can induce curiosity. Considering this, reports are made visible by anyone from anywhere. This gives a wider range of publicity in terms of citizen contribution and issues.

Figure 4a–c, Figure 5a–c, Figure 6a–c and Figure 7a–c describe the major features that were implemented in the mobile application. These features were identified, from the workshops, as the citizens concerned to be incorporated in the application. This reflects the techno-centric design in development of our CS application. In addition, these figures show different factors supported. While designing and developing the app, citizens’ input and optimum use of motivational and engagement factors identified in this study were considered.

Situatedness: Another approach is to attract people in situ. Considering this, developing an application dedicated to the particular problem domain, in this case, flooding, attracts citizens from at risk locations. Any user using this application will see the nearest reports to their current locations which can encourage affected citizens to report or comment in or against the report (Figure 4a and Figure 7a). Citizens can provide further context to their reports by taking a photograph and uploading as part of their report (Figure 6a).

Contextual Cues: This application stores all valid reports from the past, which can help citizens analyse in what context the other reports have been made and if those reports are sufficient to describe the issues local communities are struggling with. Citizens can post as many reports as they want to describe the condition based on their experience to describe the flooding incidents (Figure 5a,b).

Crowd: When an app is made available from the Google PlayStore (Android, (https://play.google.com/store/apps/details?id=com.phen.phonotracker, (accessed on 19 April 2021))), the number of downloads, feedback and reviews are made available as an indicator of authenticity and trust. While the app is limited to two communities from two participant cities, it has potential to be useful to the wider communities affected by flooding. Over a period of time, we believe continued improvements and increased engagement can help build traction and generate further interest.

4. User Study

A user study was conducted to evaluate the design, user interface and other features of the application to analyse motivational and engagement factors that were implemented in the application. To avoid any bias (due to learnability or shared ownership of the design of the application) during the trial of the app, volunteers who were not a part of any of the previous stages of development were invited to participate. The users were only provided a (Google Playstore) link to the application, without any further installation instructions or user guides. This was done to to assess the user experience in term of usability, learnability, user experience and motivational and engagement factors. The trial involved 10 participants who have good IT skill and are active social mobile app users. The feedback was collected online (Google form).

4.1. System Usability Scale (SUS)

In systems engineering, SUS is a standard ten-item Likert scale questionnaire that aims to determine a user’s subjective assessments of the usability of an application. Each question represents a dimension of usability such as frequency of use, complexity, ease of use, learnability and so on. SUS was developed by John Brooke at Digital Equipment Corporation in the UK in 1986 as a tool to be used in usability engineering. This proved to be an extremely simple and reliable tool for use when doing usability evaluations of an application [136]. SUS is used as a tool to measure the usability of any application based on how the user perceives the application when they use it. To achieve this, 10 SUS statements (

Table 3. Modified SUS statements and statistical score for each statement in terms of: “Absolute Mean”, “Standard Deviation” and “Absolute Range”. All measures (between 1 and 5) are based on the degree of agreement with the statement. 1 indicates strong disagreement while 5 indicates strong agreement with the statement.

Modified SUS Statements	Absolute Mean	Standard Deviation	Absolute Range
1. I think that I would like to use this App frequently.	3.8	1.30	2–5
2. I found this App unnecessarily complex.	1.8	0.83	1–3
3. I thought this App was easy to use.	4.0	0.71	3–5
4. I think that I would need assistance to be able to use this App.	2.2	1.30	1–3
5. I found the various functions in this App were well-integrated	4.2	0.83	3–5
6. I thought there was too much inconsistency in this App.	2.2	1.30	1–4
7. I would imagine that most people would learn to use this App very quickly.	4.4	0.90	3–5
8. I found this App very cumbersome/awkward to use	2.6	1.52	1–4
9. I felt very confident using this App	4.0	0.71	3–5
10. I needed to learn a lot of things before I could get going with this App.	2.6	1.14	1–4

, column: “Modified SUS Statements”) were prepared as a questionnaire in Google Forms to analyse citizens’ experience after using the application.

4.2. SUS Feedback Analysis

Table 3 presents our user assessments scores on individual SUS questions. The table shows the mean score, standard deviation and absolute range for each question. Out of 10 dimensions, the scores for the statements “I think that I would like to use this App frequently”, “I thought there was too much inconsistency in this App”, “I found this App very cumbersome/awkward to use” and “I needed to learn a lot of things before I could get going with this App.” vary from 1 to 4 (1 - strong disagreement; 5 - strong agreement). This wider range suggested that there was a varied level of acceptance for the application among our users. We observed that there was a narrower range for the rest of the statements, indicating a greater favourable agreement among users (confident using the app, learnability, functions well integrated, easy to use, not unnecessarily complex). As a next step, as per SUS methodology, these scores were adjusted as for SUS score calculation and then the sum of the adjusted score was multiplied by 2.5 (referred to as “SUS Score”)) [136].

As a rule-of-thumb [137], an application with SUS scores less than 50 has significant concern and is deemed unacceptable; an application with SUS score of 50−70 is marginally acceptable; and an application with SUS score greater than 70 is categorised as acceptable. SUS score in the high 70s to upper 80s are considered to be better than acceptable while highly acceptable applications score over 90. The SUS scores of our application vary from 50 to 97.5 with a mean of 72.5 and a standard deviation of 20.23. This indicates that our application falls under the category of “user acceptable application”, while individually, users scored (SUS score) the application from an acceptable range to a highly acceptable range. While it is encouraging that some users found the application highly acceptable, we recognise the need for further improvements to ensure that the next iterations of the application is more consistent high acceptability from users. In the future, a revised version of this application will be further tested with more users in the two communities of Bradford and Aberdeen. Feedback will be collected to improve the applications’ features in the next iteration of the application development.

5. Discussions and Limitations

A thorough literature review helped us identify 17 motivational and engagement factors that can be used for CS projects. While some of these factors often emerge in the development of different CS technologies, we observed a gap in the literature where they have been considered from the onset of the development. It is important to note that this is an observation based on a review of the literature. In reality, many of these factors may have been considered as a part of the design of the project/application, but not explicitly reported in academic literature. We acknowledge the critical role citizens and communities have played in the co-development of many CS projects. The many nuanced discussions, thoughtful feedback and citizen contributions may often be difficult to structure as a part of scientific literature, particularly when underlying motivations of citizens might be complex, multi-dimensional and often difficult to articulate. Our study, by no means, attempts to simplify the complex justifications and motivations citizens and communities may have in being involved. Instead, we believe that, by making these motivational factors explicit, new forms of engagement and citizen collaborations may emerge in the future.

The development of the application is an exercise in exploring how the different motivations can be encapsulated within a tangible application. While we demonstrate how the different factors have been considered while developing the mobile application, we acknowledge that there are other ways some factors could be implemented. For example, another way to incorporate a “Trust” dimension would be to introduce points for individuals who have contributed to the platform. We are currently exploring ideas as a part of our future work on how these points could help incentivise participation by citizens (by e.g. gamification). At the same time, careful attention needs to be paid to ensure citizens do not put themselves at risk to collect points. For a sensitive topic such as flood monitoring, it is also important to ensure that data submitted by citizens are accurate and appropriate. Future work in this area will involve developing automated and semi-automated approaches to detect inaccuracies and replication in the data being submitted. This is an area of research that is beyond the scope of this study, but initial ideas involve the use of human moderators, experts in local authorities and local community leaders, in conjunction with the use of metadata (e.g., EXIF data, GPS coordinates and text fields).

While we mention the need to develop techniques that can assess the trustworthiness of data and users, it is important to note that trust is multidimensional and reciprocal. While manual and automated mechanisms can be put in place to ensure the information used by decision-makers is trustworthy, citizens also need to trust the decisions that are taken. While generic feedback on how the citizen-generated data have been used to make decisions is important, we believe it is also important to inform users how each report is considered by the local authority. In our implementation, we believe the reflection of live updates and status to reports is an important first step. However, much is to be done in explaining how every submitted report is exactly used for decision-making. Trust is also built from ensuring appropriate policies are in place to protect the privacy of users. In a CS project such as ours, this is critical, while, at the same time, local authorities need sufficient information to ensure they have accurate data to act upon. This is often a sensitive topic that needs negotiation between all stakeholders (e.g. citizens, application developers and local authorities). While out of scope for the context of this paper, we believe this is a complex area that is often difficult to navigate, compounded by varying interpretations of legal and ethical policies and often conflicting interests.

In discussing the social bonds dimension, we mentioned the users expressing an interest in sharing reports on social media. The role of social media in generating citizen interest is much recognised and encouraging. At the same time, social media can often be exploited for misinformation and disinformation campaigns. Given the sensitivity of flooding and its direct impact on citizens and communities, it is important to consider how social media can be effectively incorporated as a part of our application.

We also note that the number of users involved in our usability evaluation is limited. The application developed as a part of our research is at an early stage and, as mentioned above, with each iteration of the design, new users are involved as a part of the process. As such, we believe the ten users involved at this stage is appropriate and sufficient to highlight important issues in the application. Additionally, limitations in using the System Usability Score also exist, as it only highlights how users perceive the usability of the application from a variety of perspectives. In the future, we plan to open our application to a wider set of users for deeper insights into the user interface. We also plan a long-term evaluation where we could assess how well different features of our application directly align with the motivation and engagement factors. In the longer-term evaluation, we plan to use other mechanisms of evaluating the application. Specifically, we plan to provide decision-makers and citizens with scenario-based tasks to explore how well the application can be used in specific applied contexts.

6. Conclusions

Data collection reliability and efficiency are crucial to monitor and manage events in Smart Cities. Employing ‘citizens as sensors’ can serve as an alternative way to collecting data in a variety of circumstances such as flooding where the deployment of IoT sensors are expensive and often practically not feasible. In other words, citizens can play a vital role in collecting data, especially in technology-centric CS applications where IoT sensors can have limited accessibility. However, the participation and continued engagement of citizens in the project is critical to the long-term sustainability of such CS applications. In our study, we conducted an exhaustive review of CS projects and research papers to explore different motivational and engagement factors. Based on our review, 17 motivational and engagement factors were identified that have been applied in some ways in different CS projects. We simplify the presentation of these factors for other aspiring technology-centric CS projects and presented a motivational and engagement factor tree (Figure 2). The tree presents the different classes as factors and corresponding branches as names (synonyms or closely aligned concepts) and methods being used for the same purpose, in the form of a hierarchy. Our observations note that not all CS projects have implemented all the factors, although there have been factors that are more commonly applied. At the same time, choices of these factors often vary between CS projects. Considering the different factors used in different CS projects, we presented a Motivational and Engagement Factors Pareto chart (Figure 3). We believe that our contributions with the motivational and engagement factor tree and the Pareto chart can offer some insights to other CS projects on consideration of factors to be incorporated for new technologies being developed.

We demonstrated the use of these factors using a top-down approach with a flood event reporting mobile application in the context of an EU project involving two flood-affected communities from Bradford and Aberdeen Councils. In the design and development of the application, we integrated motivational and engagement factors that were identified in our review. An initial user evaluation was conducted using a standard questionnaire based approach primarily to evaluate SUS dimensions. Based on the evaluation results, we note that while the application has positive user acceptance, there is scope for considerable improvements. In the future, a refined version of the application will be tested with citizens from two flood-affected communities from two councils over a longer term. We will also explore other dimensions of engagement and develop strategies for analysing trust, quality and acceptability of citizen generated data. Our plans also involve further iterations of design and explore how social media can contribute towards the collection of citizen generated data.