A Practical Approach to Assessing the Impact of Citizen Science towards the Sustainable Development Goals

Abstract

The United Nations’ Sustainable Development Goals (SDGs) are an important global framework which provides a shared vision for a more sustainable future for all people and the planet. In the last five years, citizen science as a discipline has paid increasing attention to the SDGs and the contributions that citizen science could make towards their achievement. This paper presents a collection of 51 questions and a corresponding set of answers which can be used by citizen science projects to self-assess their impact towards the SDGs. The questions and answers were originally derived from the official UN description of the SDG targets and indicators and were refined through a workshop and series of nine interviews with citizen science project coordinators. The outcomes of the workshop and interviews reveal the challenges of assessing impacts towards the SDGs in a way which is relevant to the majority of citizen science projects. In many cases, the wording of the SDGs had to be altered to make sense in the context of citizen science. The final set of questions and answers are structured to reflect two pathways of impact: citizen science contributing to the (official) monitoring of the SDGs, and citizen science contributing to the direct achievement of the SDGs.

1. Introduction

The UN Sustainable Development Goals (SDGs) were set up in 2015 and, since the addition of targets and indicators in 2017, monitoring and assessment have been a central part of activities working towards the goals [1]. As a broad, global framework, many organisations and projects have attempted to connect their work to the SDGs and report on their contribution towards the achievement of the goals, albeit often in a superficial manner [2]. In response, several tools have been developed to help organisations assess their impact with regard to the SDGs [3,4]. The citizen science community has also demonstrated interest in the SDGs. Recent examples include the 2020 Citizen Science SDG Conference [5] and previous Special Issues in scientific journals [6]. Particular attention has been given to how citizen science could contribute data to the (official) monitoring of the 231 unique SDG indicators [7]. An effort to map the contribution of citizen science to the SDGs concluded that citizen science data are already used to monitor 5 SDG indicators, and could be used for the reporting of a further 76 indicators [8].

The input of citizen science to the SDGs is not limited to monitoring indicators; it also has the potential to directly contribute to the achievement of the goals and targets [9], as well as supporting broader societal changes (multi-stakeholder partnerships, education, and citizen involvement in policy), which are prerequisites to many of the changes envisaged in the SDGs [10]. The balance of these two impacts (monitoring and achievement) varies from project to project, with some projects focusing more on one than the other. For example, FreshWater Watch, a water quality citizen science project, focuses on monitoring the parameters of indicator 6.3.2.; however, through the involvement and education of citizens, the project additionally contributes to the achievement of Target 6.b: “Support and strengthen the participation of local communities in improving water and sanitation management” [11]. In other cases, such as food waste (SDG target 12.3), citizen science has the potential to contribute more by influencing action through behaviour change and policy influence, rather than only collecting data for the corresponding indicator [12]. In many cases, a project may not even have the ambition to contribute to the monitoring of an SDG indicator but may still carry out actions relevant to the achievement of the SDGs. For example, education is commonly included as an element of citizen science projects, and the impact on participant’s learning and scientific literacy has the potential to help achieve several SDGs, most obviously SDG 4: “Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all” [13].

In a previous study, a majority of citizen science projects (27 of 30) were found to have a superficial or nonexistent involvement with the SDGs; the only exceptions were projects which fed data directly into reporting for SDG indicators [14]. This shallow or nonexistent involvement with the SDGs may seem surprising given the increasing prevalence of SDGs in citizen science projects. As the paper concludes, some of this difference may be due to technical difficulties and other barriers in reporting on SDG indicators. However, by focusing on citizen science data and the monitoring of SDG indicators, the authors might also miss other contributions of projects towards the achievement of the SDG targets. One reason for this omission could be that there is not currently a clear or consistent way to assess the contribution of citizen science projects towards the implementation of the SDGs, so connections are labelled as “superficial”. For example, a recent attempt to create a self-assessment for European citizen science projects on their contribution to the SDGs asked projects to assess their impact at the level of the 17 goals [15]. Given the lack of detailed knowledge of the SDGs among citizen science projects, and the fact they often have a local scope, it is perhaps unsurprising that assessments at the broad level of the SDGs lead to statements of projects’ superficial impact [14].

Even if, to date, more has been carried out to consider the contribution of citizen science to the monitoring of the SDGs rather than to their implementation, future assessments of citizen science should consider both impacts on the SDGs, not least because a recent study found more potential for projects to make progress towards indicators rather than monitoring them [16]. Furthermore, the SDGs that citizen science has the greatest potential to monitor could potentially be significantly different from those it has the greatest potential to contribute to the achievement of. The four SDGs identified as having the greatest potential for monitoring by citizen science are the following: SDG 15—“Life on Land”; SDG 11—“Sustainable Cities and Communities”; SDG 3—“Good Health and Wellbeing”; SDG 6—“Clean Water and Sanitation” [8]. Meanwhile, another study found the goals most used by citizen science projects in addition to these four were the following: SDG 4—“Quality Education”; SDG 13—“Climate Action”; SDG 17—“Partnership for the Goals” [10]. Whilst these three goals might be less suitable for citizen science monitoring, they are closely aligned with common citizen science impacts (on education, awareness raising, behaviour change, and multi-stakeholder collaborations). A holistic assessment of the impact of citizen science across all the SDGs must therefore consider project contributions to both the monitoring of indicators and the achievement of targets and goals.

Given the predictable increasing prevalence of SDGs in the public discourse until 2030, and the corresponding pressure for projects to incorporate them, there is a need for clear and consistent methods to monitor the impact of citizen science on the SDGs. As a related advantage for citizen science, having a stronger link to the SDGs can help projects secure more funding [14]. Indeed, the SDGs are frequently mentioned in funding calls and the European Commission often specifically requests that proposals consider and address the SDGs, including in the Horizon 2020 SwafS-15-2018-2019 call, under which the MICS project was funded [17]. MICS, “Measuring the Impact of Citizen Science” (https://mics.tools, accessed on 28 February 2022), is a research project of which this study forms a part, and which is developing a platform and toolbox to help citizen science projects to measure their impacts across five domains: society, the economy, governance, the environment, and science and technology. MICS builds on the current state of the art and past learnings by carefully considering and adopting approaches from previous impact assessment methods and combining them in a new framework [18]. The primary expected users of the MICS platform will be citizen science project coordinators. A project coordinator is responsible for leading the research and managing a citizen science project, and is typically a professional employed to carry out this role and coordinate the actions of participating citizen scientists.

This paper presents an attempt to create a practical approach to assess the impact of citizen science projects towards the SDGs. The developed method considers the contribution of citizen science towards both the monitoring and achievement of the SDGs, reflecting the range of connections between citizen science and the SDGs identified in the literature. By applying the SDGs in an impact assessment framework, this research demonstrates some strengths and challenges of using the SDGs as the basis of citizen science impact assessment.

2. Materials and Methods

2.1. Experimental Design

This study was designed to critically assess the suitability of the Sustainable Development Goals for the impact assessment of citizen science projects. It forms part of a larger study within the MICS project, which aims to develop a set of characteristics (captured through a set of questions and answers) that form the basis of an impact assessment method. The experimental design was based on standard methods in survey development and pretesting. The questions and answers were assessed and refined through an iterative process, with data gathering split over three rounds in order to allow for experimental testing, whereby changes made to the questions based on feedback in the initial rounds could be validated in future rounds [19]. Firstly, a workshop was held with three citizen science project coordinators using a co-participation method where the project coordinators gave feedback simultaneously [20]. This was followed by two rounds of five and four cognitive interviews, respectively. Cognitive interviews collect user feedback to identify issues in questionnaires, helping to reduce response errors when the survey is deployed [21]. Both the workshop and the cognitive interviews used “think aloud” protocols, which allow interviewers to better understand participants’ thoughts and reactions in comparison with typical question prompts [22]. The workshop and interviews aimed to test and refine the structure and content of a set of questions and answers used to assess the impact of citizen science projects. The aims, methods, and handling of data in the study were approved by Earthwatch Europe’s ethics process (application code: MICS003).

2.2. Questionnaire Design

The initial set of questions was extracted from a literature review of citizen science impact assessment methods [18]. Some additional sources were also considered, including the ECSA characteristics of citizen science [23] and the SDG descriptions. An initial set of 152 questions was either directly extracted or adapted from these sources. Each question was assigned a unique three-digit number for identification, for example, “question 007”. A collection of answers was also defined for each question. The questions then underwent an informal expert review among this paper’s authors. The questions and corresponding answers were edited based on this review, resulting in a revised set of 159 questions. These questions were grouped into six categories: “general” questions comprising the first category, and five categories of questions related to five domains of impact (society, the economy, governance, science and technology, and the economy). The questions were transferred into a series of online questionnaires, one for each of the categories. Examples of these questionnaires are available in Supplementary Materials (Document S1). This study focuses on the 41 questions (of the total 159) related to the SDGs.

2.3. Workshop and Interview Structure

The workshop and interviews followed the same overall structure. The sessions started with a short welcome and introduction, lasting approximately five minutes. In this time, the interviewers explained the overall objectives, the context for this study, the purpose of the workshop or interview, and details of how the data collected would be handled and used. The introduction also included an explanation of practicalities for the session, for example, how interviewees could access the questionnaires, and approximate timings for the workshop or interview.

The main structure of the workshop and interview was based on the six questionnaires. During the sessions, the project coordinators worked their way through the questionnaires answering all the questions which were applicable to their project. During the workshop, the questionnaires were answered in the following order: 1—general; 2—society; 3—governance; 4—economy; 5—environment; 6—science and technology. For the nine interviews, the order of the questionnaires was manipulated to reduce bias from participant fatigue for the later questions or from the order of related questions in different questionnaires. The order of the questionnaires answered is summarised in a table available in Supplementary Materials (Document S2). The interviewers introduced each questionnaire and then read out each question, allowing time for the interviewee to answer the question and explain their reaction to the question. A set of discussion prompts were also used to help elicit feedback from the project coordinators. For example, interviewees might be asked the following: “Does the question feel relevant to your project?” or “Do you need further information to answer the question?”.

After the project coordinators had answered all the questions in each questionnaire, there was a short, semi-structured discussion designed to encourage feedback on the questionnaire as a whole, and allowing project coordinators further opportunity to give feedback on specific questions. For the interviews, an additional set of discussion prompts were added to gather more details on the project coordinators’ experience with the Sustainable Development Goals. Once all questionnaires had been answered, there was an additional semi-structured discussion to allow the interviewees to reflect on the process of answering all the questionnaires. The workshop and interviews both ended with a short closing section where the interviewers debriefed the project coordinators on the next steps, including how the results from the study would be used. Full details of the session structure, including the question prompts used, are set out in Document S3 (see Supplementary Materials).

2.4. Workshop Procedure

The workshop took place in April 2021 and was held in a meeting room at the Earthwatch office in Oxford, UK. Two workshop facilitators and the three project coordinators invited to participate were present. One facilitator was responsible for moderating the session and leading discussion, whilst the other facilitator took notes from the workshop. The workshop followed the structure described in the section above. Each of the project coordinators used their own laptops during the workshop to access the questionnaires and submit their responses. For each questionnaire, a workshop facilitator guided participants to answer the questions at the same time as each other by reading out each question in turn. Time was allowed after each question had been answered for the project coordinators to discuss and give feedback on the question. A series of prompt questions were written on a whiteboard in view of all the workshop participants. These prompts were read out and explained at the start of the workshop, and participants were invited to consider them throughout. At the end of each questionnaire, following a short discussion on the questions, the participants were offered a break and reminded they were able to request a break at any time during the session. In total, the workshop lasted just over four hours.

Notes were taken throughout the workshop, capturing the participants’ feedback and any additional observations from the facilitators. Notes were recorded in a matrix so that each comment could be associated with both the corresponding question and the project coordinator or facilitator who made the comment. After the workshop, the notes taken were reviewed for spelling or grammar errors and, if necessary, edited to make sure the original intention of the workshop participant had been sufficiently captured. This also gave the workshop facilitators an additional opportunity to add any comments about the workshop. In the days after the workshop, the notes collected were analysed and discussed by the two workshop facilitators and, where possible, changes were made to the questions and answers to resolve issues identified in the workshop. If a change could not be agreed by the two workshop facilitators, then the feedback on that question was discussed by a panel of four researchers (including the original two facilitators) who then agreed whether and how the question and answers should be changed.

2.5. Interview Procedure

The nine interviews took place between April and June 2021. The interviewees were invited via email to take part and sent a consent form, which they signed and returned prior to their interview. At least two interviewers were present at every interview: one to facilitate the interview and one to take notes. Eight of the interviews were held online using the GoToMeeting video conferencing software, and one took place in person at the Earthwatch office. The interviewees used their own devices to access the questionnaires which were emailed to them at the start of the interview. The interviews followed the same structure as the workshop with the addition of a series of discussion prompts added to obtain more details from the interviewees about the relevance of the SDGs to their project (available in Document S3 of Supplementary Materials). The questionnaires were answered in a predetermined order, which was different for each interview. After interviewees had answered and given feedback on all the questions in each questionnaire, there was a short, semi-structured discussion. Participants were offered a break after the end of each questionnaire and were reminded that they were able to request a break at any time during the interview. The interviews lasted between 4 h 56 min and 2 h 30 min, inclusive of breaks (mean 3 h 59 min; median 3 h 54 min). Data were collected in the same manner as the workshop, with interviewee comments and interviewer observations captured in a matrix with the questions. After each interview, the notes taken were cleaned, correcting spelling mistakes or sentences with an unclear meaning. Once the first set of five interviews had been completed, the notes from each of the interviews were combined with the notes from the workshop and were analysed by the interviewers. Changes were made to the questions and answers in the same way as after the workshop. The same process of analysis and revisions was undertaken after the final four interviews had been carried out.

2.6. Participants and Projects

In total, twelve project coordinators were invited to take part in the study. A previous study within the MICS project had interviewed eleven project coordinators, all of whom were invited to participate in this study, and six of whom agreed to take part. Details of the previous study and the recruitment of participants is reported in Sprinks et al. [24]. A further six project coordinators were recruited who had all responded to an open invitation to help test the MICS impact assessment approach, which was advertised at a number of citizen-science-themed conferences and events. This approach allowed participants to “opt-in”, and the authors only contacted those who had already expressed an interest in participating.

The twelve project coordinators each represented a different citizen science project. One of the projects operated only at a local level (within a single city), six of the projects operated at a national level, two of the projects operated across multiple countries, and three of the projects operate worldwide. The projects studied a variety of topics, including the following: water quality, astronomy, biodiversity, plastic pollution, accessibility, nature-based solutions, air quality, and algorithm bias. At the time of participating in this study, seven of the projects were active, four of the projects were no longer possible to participate in, and one project was yet to start. The project coordinators worked in a variety of organisations whilst managing citizen science projects. Five of the coordinators worked at private research institutes, three of the coordinators worked at universities, two of the coordinators were representing NGOs, one coordinator worked at a think tank, and one coordinator worked at an SME. In recognition of the considerable time and energy spent by the study participants, interviewees and workshop participants were offered a pizza as compensation. Only six of the twelve project coordinators accepted the offer to have food purchased for them during the study. All participants were also offered the chance to see some of the MICS project’s recent progress, including mock-ups for the MICS platform.

2.7. Analysis

Analysis of data for this study focused just on the characteristics related to the SDGs. The analysis approach was based on the six phases of thematic coding described by Braun and Clarke [25]. All the comments made in the workshop and interviews were collated together in a single file. In total, 227 comments were extracted, 217 of which were directly associated with a question, and 10 of which came from the open discussions at the end of each questionnaire. This list of comments was then read and given initial codes (multiple codes were often given to the same comment). In this process, 34 different codes were assigned 364 times across the comments. Once all comments had been coded, the 34 codes were amalgamated into 10 themes, described in Section 3.3. For example, two codes, “question requires knowledge of SDGs” and “SDG not relevant to project”, were combined to form the theme “difficulty with SDG”. The response rates of participants were also analysed, giving a quantitative indication of which questions they were interested in answering and which areas they considered they were having an impact in. Response rates were calculated by dividing the total number of responses given per question by the maximum possible number of responses, i.e., the number of participants who were asked that question. Positive response rates were calculated by dividing the number of responses, which indicated whether a project was having an impact on the related SDG by the total number of responses given to a question (not the total number of possible responses). Analysis of the questions was also aggregated to the level of the SDGs by associating each question with the corresponding goal, target, or indicator. The response rates and themes connected to each question could then be combined for each SDG.

3. Results

3.1. Question and Answer Revision

Over the course of the workshop and interviews, the original list of 41 questions related to the SDGs was expanded to a group of 51 questions. The number of changes to the questions are summarised in

Table 1. Summary of number of changes made to questions and answer options.

	Workshop	Interviews (Round 1)	Interviews (Round 2)
Questions	41	41	43
Comments from participants	21	115	81
Changes made to questions	7	19	28
Questions added	-	2	11
Questions removed	-	-	3

.

In total 13 questions were added and 3 questions were removed. Most of the new questions which were added arose from questions being split apart for clarity. For example, a question related to sustainable cities was split into two questions, one asking generally about sustainable cities and one specifically focussed on air quality.

3.2. Questionnaire Responses

During the workshop and interviews, it was possible for participants to skip answering a question if they deemed it was not relevant to their project. Analysing the response rate of each question can therefore give an indication of which questions—and by extension, which SDGs—were relevant to the projects which were interviewed. For the 43 questions asked in the second round of interviews which are related to the SDGs, the average response rate was 85%, i.e., on average, only 10 of the 12 participants would choose to answer each question. In comparison, the overall average response rate for all the questions was 90%, and was 93% for questions not related to the SDGs. Of course, responding to a question is not necessarily an indication that the question is relevant to a project, since it was possible for project coordinators to answer “no” or “I don’t know” to most questions. Another indicator of a question’s relevance to projects was, therefore, their positive response rate: the proportion of responses to a question that suggest a project had an impact in that area. To determine which could be classed as positive responses, the answers to each question were compared with the corresponding SDG goal, target, or indicator, and the options which suggested a project had an impact in that area were highlighted. In the examples below, the positive responses are indicated in bold. Question 154 is a single-select question, whilst multiple answers can be selected for Question 143. In the case that multiple answers can be selected for a question, only one of the positive answers has to be selected for the response to be classed as a positive response, i.e., an interviewee would not have to select all of the bold answers to Question 143 for their answer to be categorised as a positive response.

Question 154: Does the project explicitly promote diversity and inclusion among all relevant participant groups?

• Yes
• No
• I don’t know

Question 143: How does the project contribute to the formal education of participants (for example, by working with schools)?

• Pre-primary-level education to children
• Primary-level education to pupils
• Secondary-level education to pupils (middle and high school)
• Tertiary education to students (university, college and vocational courses)
• Adult education or life-long learning
• The project does not contribute to the formal education of participants.
• I don’t know

The average positive response rate for the questions related to the SDGs was 41%. The average positive response rate for all questions was 60%, and the average positive response rate for non-SDG related questions was 67%. The response rates for questions grouped by SDG are summarised in

Table 2. Response rates for questions related to each Sustainable Development Goal.

SDG Number	SDG Title	Total Number of Questions	Response Rate (%)	Positive Response Rate (%)
1	No poverty	2	100	42
2	Zero hunger	2	46	9
3	Good health and well-being	6	94	5
4	Quality education	5	98	75
5	Gender equality	4	100	40
6	Clean water and sanitation	1	42	100
7	Affordable and clean energy	1	0	0
8	Decent work and economic growth	4	96	15
9	Industry, innovation, and infrastructure	0	n/a	n/a
10	Reduced inequalities	2	96	21
11	Sustainable cities and communities	1	67	100
12	Responsible consumption and production	3	75	48
13	Climate action	2	96	65
14	Life below water	1	25	100
15	Life on land	1	42	100
16	Peace, justice, and strong institutions	2	88	79
17	Partnerships for the goals	5	100	25

. For example, 2 questions relating to SDG 2, “Zero Hunger”, were asked to all 12 participants. Only 11 responses were given across these 2 questions. The highest possible number of responses to these questions was 24, giving a response rate of 46% (11 divided by 24). Of these 11 responses, only 1 answer indicated a project was having an impact related to SDG 2. This gives a positive response rate of 9% (1 divided by 11).

3.3. Comments and Discussion

Across the workshop and interviews, the 12 participants made a total of 227 comments in response to questions related to the SDGs. These comments can be described with 10 themes, which are summarised in

Table 3. Summary of the themes arising from the thematic analysis of the workshop and interview comments. Quotes from participants are given in italics. Themes are ordered by number of corresponding comments.

Theme	Definition and Examples	No. of Comments
Justification of answer	These comments describe the activities of the project rather than the project coordinator’s thoughts on the question. “Our data are not open access because it’s all interview and workshop transcripts. But all the outputs of the project are open access”. “We talk about climate change and its impact on water quality and quantity. And we give education on how to use storm water or mention what is the source of water pollution that we measure in the creek and explain the environmental challenges, just not in very deep detail. We explain the probable source of pollution and then list some counter measures. So we do some small education”.	78
Change to wording	Includes all comments suggesting changes to the questions and answers, for example, requests to add another answer option, a definition of a key term, or a new question on a related topic. “It would be good to have an option where you can say ‘no’ because we’re only targeting a single group”. “Disadvantaged is a little loaded. I would say disadvantaged or historically marginalised”. “It would be good to devote a question to data ownership if it’s not already covered in an upcoming question”.	63
Uncertainty (answer)	This theme includes comments where the interviewee understood the question but was still unsure which answer they should select. “It’s always communicated what is the origin of air pollution and how it can be avoided. Would that fall under sustainable lifestyles?” “Very indirectly researches sustainable agriculture. Not researching how to farm but just looking at the impacts. [I] said yes but [I’m] not confident” “I don’t know which category our education falls into. we have diverse participants from different age and educational backgrounds”.	57
Positive reaction	Positive reactions to questions included comments that the question was interesting and easy to understand or answer. “I really like that you have put this here because I think there are so many projects which haven’t even thought or don’t know what the SDGs are and it really makes you think about that”. “It’s an interesting [question], not one that I’d thought about before”. “These questions are easy, even if I answer no to them all”.	34
Difficulty with SDGs	These comments indicated that the interviewee was having difficulty with the SDG related to the question, either because they felt they did not have enough knowledge about the SDGs or because the SDG was not relevant to their project. “I think the SDG parameters are at the very high level, it’s very difficult to collect in a citizen-science project, it’s really at a different level than our project”. “It would be good to have some explanation about the SDGs. I know vaguely what they are but I don’t know them in detail”.	32
Specificity required	Many of the questions include the term “explicitly”. This theme includes comments where interviewees indicated that they had an indirect impact on a topic, or that having an impact on the topic was not an aim of their project, so they did not feel they could say they explicitly had an impact on the topic. “I would clarify, is the direct goal of the project about this. Again, air quality monitoring might provide data which is helpful for respiratory diseases but health isn’t the primary focus of the project”. “Explicitly no. I’m sure there will be a link. We just don’t investigate it”.	32
Uncertainty (question)	This theme includes comments where the interviewee was unsure about their interpretation of the question. “Are you asking about sustainable development in the context of the SDGs? Not everyone sees it through the SDG framework”. “Need to adapt some of these questions to more natural language rather than buzz words and technical [or] academic language” “What does that mean? [It’s] Not a familiar term”.	31
Project ambition	Sometimes, interviewees were reluctant to answer “no” to a question and preferred to say “not yet”, indicating that, even if the project did not currently have an impact on this topic, it had an ambition to do so. “Not yet, but we hope it will. We talk about it in our social media but it’s not currently a focus of the project”. “I want to select good health because we have ambitions to talk about well-being but it’s not something we directly address yet by a research question”.	17
Difficulty with the topic	This theme includes comments of dissatisfaction with the topic of the question. For example, participants complained that certain impacts were too broad, too ambitious, or not meaningful to citizen science. “Maybe long term but within the span of the project it would be too ambitious to say we have a positive impact on the health of participants”. “So few projects… are linked to expectations of economic benefit or gain. I can guess what you’re trying to ask it’s just not meaningful in citizen science”.	13
Dependency on country	The interpretation of some questions depended on the very specific national context of the interviewee. “I would say it’s just the secondary level education. This is the challenge of doing a platform in many different countries because the education system will look different. But I think it’s still possible to answer these”. “I think people in different countries, if they’re familiar with the SDGs, they might be more used to their local wording”.	7
	Total	364

. (Some comments fit several themes; hence, the total number of comments in Table 3 is 364).

3.4. Updated Questions Related to SDGs

The updated set of 51 questions related to the SDGs are presented in Figure 1 and

Table 4. Updated questions and answers ordered by SDG.

Question and Answer Options	Question ID	SDG Description
Which socially-relevant issues are directly addressed by the project? (We’ll ask details about each issue later.) Poverty Hunger and nutrition Health and wellbeing Education Gender equality Decent work and economic growth Industry, innovation and infrastructure Reduced inequalities Responsible consumption and production Peace, justice and strong institutions	131	Goal 1: End poverty in all its forms everywhere. Goal 2: End hunger, achieve food security and improved nutrition and promote sustainable agriculture. Goal 3: Ensure healthy lives and promote well-being for all at all ages. Goal 4: Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all. Goal 5: Achieve gender equality and empower all women and girls. Goal 8: Promote sustained, inclusive and sustainable economic growth, full and productive employment and decent work for all. Goal 9: Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation. Goal 10: Reduce inequality within and among countries. Goal 12: Ensure sustainable consumption and production patterns. Goal 16: Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels.
Which environmental issues are related to the aims of the project? (We’ll ask details about each issue later.) Sustainable agriculture Freshwater Affordable and clean energy Sustainable cities and communities Air quality Responsible consumption and production (including food waste and chemical pollution) Climate action Marine water Life on land	414	Goal 2: End hunger, achieve food security and improved nutrition and promote sustainable agriculture. Goal 6: Ensure availability and sustainable management of water and sanitation for all. Goal 7: Ensure access to affordable, reliable, sustainable and modern energy for all. Goal 11: Make cities and human settlements inclusive, safe, resilient and sustainable. Goal 12: Ensure sustainable consumption and production patterns. Goal 13: Take urgent action to combat climate change and its impacts. Goal 14: Conserve and sustainably use the oceans, seas and marine resources for sustainable development. Goal 15: Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss.
Does the project actively engage participants from disadvantaged or historically marginalised backgrounds? Yes No I don’t know	156	Target 1.4: By 2030, ensure that all men and women, in particular the poor and the vulnerable, have equal rights to economic resources, as well as access to basic services, ownership and control over land and other forms of property, inheritance, natural resources, appropriate new technology and financial services, including microfinance.
Does the project foster resilience, or does it foster learning and adaptation (which then leads to resilience)? Yes and it has been measured Yes, but it has not been measured No After five minutes, I still don’t understand the question I don’t know	165	Target 1.5: By 2030, build the resilience of the poor and those in vulnerable situations and reduce their exposure and vulnerability to climate-related extreme events and other economic, social and environmental shocks and disasters. Target 13.1: Strengthen resilience and adaptive capacity to climate-related hazards and natural disasters in all countries.
Does the project contribute to secure plant and animal genetic resources in either medium- or long-term conservation facilities? Yes No I don’t know	430	Indicator 2.5.1: Number of (a) plant and (b) animal genetic resources for food and agriculture secured in either medium- or long-term conservation facilities.
How does the project contribute to sustainable agriculture? Increasing the proportion of agricultural area being managed sustainably Increasing investment in rural infrastructure, agricultural research and technology development Increasing agricultural productivity Researching sustainable agriculture	415	Target 2.3: By 2030, double the agricultural productivity and incomes of small-scale food producers, in particular women, indigenous peoples, family farmers, pastoralists and fishers, including through secure and equal access to land, other productive resources and inputs, knowledge, financial services, markets and opportunities for value addition and non-farm employment. Indicator 2.4.1: Proportion of agricultural area under productive and sustainable agriculture. Target 2.a: Increase investment, including through enhanced international cooperation, in rural infrastructure, agricultural research and extension services, technology development and plant and livestock gene banks in order to enhance agricultural productive capacity in developing countries, in particular least developed countries.
Does the project have a positive impact on the physical health of participants? Yes, and it has been measured Yes, but it has not been measured No I don’t know	136	Goal 3: Ensure healthy lives and promote well-being for all at all ages.
Does the project aid in the investigation of diseases? Yes, directly as part of the project activities Yes, indirectly from the project activities No I don’t know	133	Target 3.3: By 2030, end the epidemics of AIDS, tuberculosis, malaria and neglected tropical diseases and combat hepatitis, water-borne diseases and other communicable diseases. Target 3.4: By 2030, reduce by one third premature mortality from non-communicable diseases through prevention and treatment and promote mental health and well-being.
Does the project directly research mental health concerns? Yes No I don’t know	137	Target 3.4: By 2030, reduce by one third premature mortality from non-communicable diseases through prevention and treatment and promote mental health and well-being.
Does the project have a positive impact on the mental health of participants? Yes and it has been measured Yes, but it has not been measured No I don’t know	140	Target 3.4: By 2030, reduce by one third premature mortality from non-communicable diseases through prevention and treatment and promote mental health and well-being.
Does the project explicitly investigate the link between pollution and health? Yes, air pollution Yes, water pollution Yes, some other kind of pollution No I don’t know	135	Target 3.9: By 2030, substantially reduce the number of deaths and illnesses from hazardous chemicals and air, water and soil pollution and contamination.
Does the project aid in the research and development of vaccines and medical interventions? Yes, directly as part of the project activities Yes, indirectly from the project activities No I don’t know	134	Target 3.b: Support the research and development of vaccines and medicines for the communicable and non-communicable diseases that primarily affect developing countries, provide access to affordable essential medicines and vaccines, in accordance with the Doha Declaration on the TRIPS Agreement and Public Health, which affirms the right of developing countries to use to the full the provisions in the Agreement on Trade-Related Aspects of Intellectual Property Rights regarding flexibilities to protect public health, and, in particular, provide access to medicines for all.
How does the project contribute to the formal education of participants (for example, by working with schools)? Pre-primary-level education to children Primary-level education to pupils Secondary-level education to pupils (middle and high school) Tertiary education to students (university, college and vocational courses) Adult education or life-long learning The project does not contribute to the formal education of participants. I don’t know	143	Target 4.1: By 2030, ensure that all girls and boys complete free, equitable and quality primary and secondary education leading to relevant and effective learning outcomes. Target 4.2: By 2030, ensure that all girls and boys have access to quality early childhood development, care and pre-primary education so that they are ready for primary education. Target 4.3: By 2030, ensure equal access for all women and men to affordable and quality technical, vocational and tertiary education, including university.
Do participants gain new skills from taking part in the project? Yes and it has been measured Yes, but it has not been measured No I don’t know	146	Target 4.4: By 2030, substantially increase the number of youth and adults who have relevant skills, including technical and vocational skills, for employment, decent jobs and entrepreneurship.
What kind of specific training does the project provide to participants? Written instructions Training video In-person training Online training Training is not required I don’t know	142	Target 4.4: By 2030, substantially increase the number of youth and adults who have relevant skills, including technical and vocational skills, for employment, decent jobs and entrepreneurship.
Does the project provide technical support to participants? Yes No I don’t know	525	Target 4.4: By 2030, substantially increase the number of youth and adults who have relevant skills, including technical and vocational skills, for employment, decent jobs and entrepreneurship.
What support is provided to educational institutions by the project? In-person sessions run by the project Lesson plans Educational resources (for example, work sheets or classroom activities) Explicit links to the institution’s curriculum I don’t know	144	Target 4.c: By 2030, substantially increase the supply of qualified teachers, including through international cooperation for teacher training in developing countries, especially least developed countries and small island developing States.
Does the project explicitly contribute to gender equality? Yes No I don’t know	159	Goal 5: Achieve gender equality and empower all women and girls.
Does the project use enabling technology, in particular information and communications technology, to promote the empowerment of women? Yes No I don’t know	526	Target 5.b: Enhance the use of enabling technology, in particular information and communications technology, to promote the empowerment of women.
Does the project use mobile phones as a primary tool (for example, using an app to collect observations)? Yes No I don’t know	523	Indicator 5.b.1: Proportion of individuals who own a mobile telephone, by sex.
Is participation possible without a phone connected to the internet? No; it’s nearly impossible Yes; participation is possible without a phone, but a computer connected to the internet is necessary. Yes; participation is possible without a phone or a computer connected to the internet. I don’t know	524	Indicator 5.b.1: Proportion of individuals who own a mobile telephone, by sex.
With regards to freshwater, which of the following activities does the project include? Ambient water quality/pollution Water-related ecosystems and biodiversity Involving local communities in water management Water-use efficiency Water stress of water scarcity Correct treatment of wastewater River restoration	416	Indicator 6.3.1: Proportion of domestic and industrial wastewater flows safely treated. Indicator 6.3.2: Proportion of bodies of water with good ambient water quality. Target 6.4: By 2030, substantially increase water-use efficiency across all sectors and ensure sustainable withdrawals and supply of freshwater to address water scarcity and substantially reduce the number of people suffering from water scarcity. Indicator 6.4.2: Level of water stress: freshwater withdrawal as a proportion of available freshwater resources. Target 6.6: By 2020, protect and restore water-related ecosystems, including mountains, forests, wetlands, rivers, aquifers and lakes. Indicator 6.6.1: Change in the extent of water-related ecosystems over time. Target 6.b: Support and strengthen the participation of local communities in improving water and sanitation management.
How does the project contribute to affordable and clean energy? Increasing the share of clean energy in the energy mix Increasing energy efficiency Researching clean energy	418	Target 7.2: By 2030, increase substantially the share of renewable energy in the global energy mix. Target 7.3: By 2030, double the global rate of improvement in energy efficiency. Target 7.a: By 2030, enhance international cooperation to facilitate access to clean energy research and technology, including renewable energy, energy efficiency and advanced and cleaner fossil-fuel technology, and promote investment in energy infrastructure and clean energy technology.
Does the project explicitly improve economic productivity through diversification, technological upgrading or innovation? Yes; the project doubled the GDP of a country Yes; but it wasn’t as successful as that! No I don’t know	316	Target 8.2: Achieve higher levels of economic productivity through diversification, technological upgrading and innovation, including through a focus on high-value added and labour-intensive sectors.
Does the project explicitly promote the formation and growth of micro-, small- or medium-sized enterprises/businesses? Yes No I don’t know	307	Target 8.3: Promote development-oriented policies that support productive activities, decent job creation, entrepreneurship, creativity and innovation, and encourage the formalization and growth of micro-, small- and medium-sized enterprises, including through access to financial services.
Does the project have a positive impact on the livelihoods of participants? Yes No I don’t know	310	Target 8.5: By 2030, achieve full and productive employment and decent work for all women and men, including for young people and persons with disabilities, and equal pay for work of equal value.
Does the project have explicit health and safety regulations in place? Yes No Not applicable I don’t know	228	Target 8.8: Protect labour rights and promote safe and secure working environments for all workers, including migrant workers, in particular women migrants, and those in precarious employment.
Does the project increase demand for the services of organisations (for example, by promoting sustainable tourism)? Yes No I don’t know	309	Target 8.9: By 2030, devise and implement policies to promote sustainable tourism that creates jobs and promotes local culture and products.
Does the project explicitly promote diversity and inclusion among all relevant participant groups? Yes No I don’t know	154	Target 10.2: By 2030, empower and promote the social, economic and political inclusion of all, irrespective of age, sex, disability, race, ethnicity, origin, religion or economic or other status.
Has the project been designed to give access, where possible, to all participants, including those with “functional diversity” [https://en.wikipedia.org/wiki/Functional_diversity_(disability)] (accessed on 28 February 2022)? Yes No I don’t know	152	Target 10.2: By 2030, empower and promote the social, economic and political inclusion of all, irrespective of age, sex, disability, race, ethnicity, origin, religion or economic or other status.
Does the project include objectives to protect or enhance cultural heritage components? Yes No I don’t know	162	Target 11.4: Strengthen efforts to protect and safeguard the world’s cultural and natural heritage.
With regards to air quality, which of the following activities does the project include? Involving local communities in air quality management Ambient air quality/pollution (including fine particulate matter e.g., PM 2.5 and PM 10)	421	Indicator 11.6.2: Annual mean levels of fine particulate matter (e.g., PM 2.5 and PM 10) in cities (population weighted).
With regards to Sustainable cities and communities, which of the following activities does the project include? Involving local communities in urban planning and management Correct collection and treatment of solid waste Green/blue public spaces	419	Indicator 11.6.1: Proportion of municipal solid waste collected and managed in controlled facilities out of total municipal waste generated, by cities. Target 11.7: By 2030, provide universal access to safe, inclusive and accessible, green and public spaces, in particular for women and children, older persons and persons with disabilities.
With regards to responsible consumption and production, which of the following activities does the project include? Food waste Harmful chemical in the air Harmful chemical in the water Harmful chemical in the soil Waste generation and management (including prevention, reduction, recycling and reuse)	423	Target 12.3: By 2030, halve per capita global food waste at the retail and consumer levels and reduce food losses along production and supply chains, including post-harvest losses. Target 12.4: By 2020, achieve the environmentally sound management of chemicals and all wastes throughout their life cycle, in accordance with agreed international frameworks, and significantly reduce their release to air, water and soil in order to minimize their adverse impacts on human health and the environment. Target 12.5: By 2030, substantially reduce waste generation through prevention, reduction, recycling and reuse.
Does the project collaborate with external companies to enable the adoption of sustainable practices? Yes No I don’t know	408	Target 12.6: Encourage companies, especially large and transnational companies, to adopt sustainable practices and to integrate sustainability information into their reporting cycle.
Does the project explicitly disseminate information on sustainable development or lifestyles? Yes No I don’t know	404	Target 12.8: By 2030, ensure that people everywhere have the relevant information and awareness for sustainable development and lifestyles in harmony with nature.
Does the project educate participants on environmental challenges? Yes No I don’t know	405	Target 13.3: Improve education, awareness-raising and human and institutional capacity on climate change mitigation, adaptation, impact reduction and early warning.
With regards to marine water, which of the following activities does the project include? Marine nutrient pollution Marine plastic pollution Ocean acidification Marine protected areas Overfishing Marine technology	425	Target 14.1: By 2025, prevent and significantly reduce marine pollution of all kinds, in particular from land-based activities, including marine debris and nutrient pollution. Target 14.3: Minimize and address the impacts of ocean acidification, including through enhanced scientific cooperation at all levels. Target 14.4: By 2020, effectively regulate harvesting and end overfishing, illegal, unreported and unregulated fishing and destructive fishing practices and implement science-based management plans, in order to restore fish stocks in the shortest time feasible, at least to levels that can produce maximum sustainable yield as determined by their biological characteristics. Target 14.5: By 2020, conserve at least 10 per cent of coastal and marine areas, consistent with national and international law and based on the best available scientific information. Target 14.a: Increase scientific knowledge, develop research capacity and transfer marine technology, taking into account the Intergovernmental Oceanographic Commission Criteria and Guidelines on the Transfer of Marine Technology, in order to improve ocean health and to enhance the contribution of marine biodiversity to the development of developing countries, in particular small island developing States and least developed countries.
With regards to life on land, which of the following activities does the project include? Sustainable management of forests Desertification or land degradation Mountain ecosystems Other terrestrial ecosystems Biodiversity Species extinction Invasive alien species Wildlife poaching or trafficking Animal behaviour	427	Target 15.1: By 2020, ensure the conservation, restoration and sustainable use of terrestrial and inland freshwater ecosystems and their services, in particular forests, wetlands, mountains and drylands, in line with obligations under international agreements. Target 15.2: By 2020, promote the implementation of sustainable management of all types of forests, halt deforestation, restore degraded forests and substantially increase afforestation and reforestation globally. Target 15.3: By 2030, combat desertification, restore degraded land and soil, including land affected by desertification, drought and floods, and strive to achieve a land degradation-neutral world. Target 15.4: By 2030, ensure the conservation of mountain ecosystems, including their biodiversity, in order to enhance their capacity to provide benefits that are essential for sustainable development. Target 15.5: Take urgent and significant action to reduce the degradation of natural habitats, halt the loss of biodiversity and, by 2020, protect and prevent the extinction of threatened species. Target 15.7: Take urgent action to end poaching and trafficking of protected species of flora and fauna and address both demand and supply of illegal wildlife products. Target 15.8: By 2020, introduce measures to prevent the introduction and significantly reduce the impact of invasive alien species on land and water ecosystems and control or eradicate the priority species. Target 15.c: Enhance global support for efforts to combat poaching and trafficking of protected species, including by increasing the capacity of local communities to pursue sustainable livelihood opportunities.
Does the project lead to an increase in the commitment of organizations to public participation in decision making? Yes, they have made a formal commitment Yes, but not as a formal commitment No I don’t know	231	Target 16.7: Ensure responsive, inclusive, participatory and representative decision-making at all levels.
Does the project help organisations to increase their capacity for public participation in decision making? Yes No I don’t know	232	Target 16.7: Ensure responsive, inclusive, participatory and representative decision-making at all levels.
Are the outputs generated by the project open access? Yes; all outputs Yes; some outputs No I don’t know	210	Target 16.10: Ensure public access to information and protect fundamental freedoms, in accordance with national legislation and international agreements.
How can the outputs generated by the project be used by external parties? Outputs can be shared Outputs can be edited or combined with other material to produce a new work Outputs can be used for commercial purposes Outputs can be used without attribution to the author No I don’t know	211	Target 16.10: Ensure public access to information and protect fundamental freedoms, in accordance with national legislation and international agreements.
Does the project enhance international cooperation on science, technology or innovation? Yes No I don’t know	527	Target 17.6: Enhance North–South, South–South and triangular regional and international cooperation on and access to science, technology and innovation and enhance knowledge-sharing on mutually agreed terms, including through improved coordination among existing mechanisms, in particular at the United Nations level, and through a global technology facilitation mechanism.
Does the project explicitly develop, transfer or disseminate information about environmentally-sound technologies? Yes No I don’t know	522	Target 17.7: Promote the development, transfer, dissemination and diffusion of environmentally sound technologies to developing countries on favourable terms, including on concessional and preferential terms, as mutually agreed.

. Figure 1 shows a flow chart of six questions which assess the project’s knowledge of the SDGs and their contribution to the monitoring of SDG indicators. These six questions cover all the goals and indicators of the SDGs. Table 4 presents the remaining 45 questions, grouped by the SDG they relate to. These questions correspond with 16 of the 17 SDGs, 52 of the 169 SDG targets, and 8 of the 231 unique SDG indicators. Some of the questions allow multiple answers to be selected, whilst some require one answer only to be chosen. The wording of the questions has been adapted from the original wording in the SDGs so the strength of relation between the questions and corresponding targets and indicators varies between questions. Some questions very closely match the SDGs; for example, question 408 uses almost identical wording to SDG Target 12.6, as follows:

Question 408: Does the project collaborate with external companies to enable the adoption of sustainable practices?

SDG Target 12.6: Encourage companies, especially large and transnational companies, to adopt sustainable practices and to integrate sustainability information into their reporting cycle.

Other questions have become less directly linked to the original SDG. For example, question 156 asks about inclusivity within the project.

Question 156: Does the project actively engage participants from disadvantaged or historically marginalised backgrounds?

SDG Target 1.4: By 2030, ensure that all men and women, in particular the poor and the vulnerable, have equal rights to economic resources, as well as access to basic services, ownership and control over land and other forms of property, inheritance, natural resources, appropriate new technology and financial services, including microfinance.

Whilst inclusion is an element of SDG target 1.4., it is predominantly about reducing economic inequality and poverty. Someone answering “yes” to question 156 would therefore not be guaranteed to be contributing to SDG target 1.4.

4. Discussion

This study presents a set of questions, tested and refined over a series of interviews, assessing the impact of citizen science projects on the SDGs. Analysing the responses of citizen science project coordinators to these questions helps to reveal the potential connections between citizen science and the SDGs, as well as the issues that arise when trying to empirically assess these connections.

4.1. Relevance of the SDGs to Citizen Science

Including a significant number of questions related to the SDGs in a general impact assessment framework for citizen science projects only makes sense if the SDGs are relevant to citizen science projects. Analysing the response rates for each question gives some indication of which of the SDGs the project coordinators found relevant. Five SDGs have response rates lower than 50% (SDG 2, 6, 7, 14, and 15). Of these, three SDGs (6, 14, and 15) had a 100% positive response rate, whilst SDG 2 and 7 had positive response rates of 9% and 0%, respectively. The low response rate for these five goals is perhaps unsurprising given that the questions related to these SDGs are very thematically specific. For example, a project coordinator is very unlikely to answer a question related to “life below water” (SDG 14) unless it focuses on marine environments. The difference in positive response rates, however, suggests that, although the questions related to SDGs 6, 14, and 15 were not relevant to every project, for projects which worked on these topics, they were very relevant and easy to answer. Meanwhile, the low positive response rates for SDG 2 and 7 suggests that these questions were not relevant, at least for the projects interviewed. The thematic specificity of these questions could explain why the response rates and positive response rates were, on average lower, for the SDG questions than the other questions. The low response rates for these SDGs in this study do not necessarily suggest that they are not worth including in an impact assessment framework; they simply reflect the thematic bias in the project coordinators who participated in the workshop and interviews. It is clear, however, that not every SDG will be relevant to every citizen science project.

In contrast, the three SDGs with both high response rates and high positive response rates (SDG 4, 13, and 16) cover topics which are relevant to the discipline of citizen science, regardless of the thematic focus of the project. For example, questions related to SDG 4 ask about the education and training of citizen scientists, which is a common objective of citizen science projects. Similarly, one question related to SDG 13 asks about the education of participants about environmental challenges, which is relevant to most citizen science projects with an environmental objective. The questions related to SDG 16 are about public participation in decision making and whether project outputs are open access. Again, it is likely that these questions are relevant to most projects, given the strong relation between citizen science, open science, and democratisation.

It is also interesting to examine the SDGs with high response rates but low positive response rates (SDG 1, 3, 5, 8, 10, and 17). This pattern could indicate topics relevant to projects, but on which the projects do not have an impact. For example, the questions based on SDG 1, 3, 5, and 10 are all related to various forms of diversity and inclusion, which are commonly discussed topics among citizen science practitioners, but which projects often do not have time to tackle directly. The questions on economic growth (SDG 8) and international cooperation (SDG 17) might also be related to the long-term objectives of projects but are too broad or ambitious for most projects to impact directly.

In a previous study [24], citizen science project coordinators were interviewed about their current approaches to impact assessment and their ability to assess impact. In these interviews, project coordinators gave the highest proportion of negative comments about their ability to assess impacts for SDGs 3 and 4, and gave the highest proportion of positive comments about their ability to assess impacts for SDGs 6, 14, and 15. The relation between SDGs and citizen science impact on them, shown by the previous study, is generally supported by the response rates in this study. For example, SDG 3 has a very low positive response rate, whilst SDGs 6, 14, and 15 all have a 100% response rate. The main exception between the two studies is SDG 4, “Quality education”, which had a high positive response rate in this study. This difference could be because, in the previous study, participants were asked about their ability to quantify their impact on the topic, whereas in this study, project coordinators could answer “yes/no” questions, which are likely to be easier to answer.

Whilst the topics of some SDGs might be thematically relevant to citizen science projects, comments from the workshop and interviews indicated that the SDG framework as a whole might not be as relevant to citizen science projects. This is especially the case for projects whose local geographic scale or short timeframe do not match the broad scale of the SDGs. One of the themes which arose from the thematic analysis was “Difficulty with SDGs”, which had 32 corresponding comments. Out of these 32 comments, 19 were made during the wrap-up discussions at the end of questionnaires rather than in direct response to a question. The remaining 13 comments were relatively evenly spread among the SDGs, with most SDGs only receiving 1 comment on this theme. This distribution could suggest that, whilst project coordinators are generally happy with the topics covered in the SDGs, they have more difficulty with the SDG framework as a whole. For example, one participant commented: “I’m not linking my project directly to the SDGs. Not for any particular reason, but I hadn’t thought of it. For a project which isn’t directly linked to the SDGs, it might not feel like a natural way to assess the project. They might have to do quite a bit of research to understand the questions”. Other common comments were that the SDGs are at “the very high level” and do not “make sense in the context of citizen science”. Another potential limitation of the SDGs is their timeframe. One participant explained: “All this discussion about the SDGs, they were established in 2015, so depending on the period of the project you’re evaluating, they might not be relevant to the project. Our participants were in training between 2013 and 2016”. The difference in timeframe is a critical downside of integrating the SDGs within an impact assessment framework for citizen science. Even if the topics covered by the SDGs are relevant to some projects, as a policy framework, they might not be relevant at the scale or timeframe of an individual project. The MICS questions, therefore, confine specific references to the SDGs to the 6 questions outlined in Figure 1, and avoid explicitly mentioning the SDGs for the other 45 questions, even if they are based on the SDGs.

4.2. Monitoring Progress vs. Contribution towards Achieving SDGs

Citizen science can have two different impacts on the SDGs: monitoring progress towards achieving the SDGs (Figure 1) and contribution towards achieving the SDGs (Table 4). Currently, the number of citizen science projects which support the monitoring of SDG indicators is limited. Still, citizen science has been identified as an important potential source of data for SDG monitoring [8]. To capture this impact, the questions in Figure 1 include explicit references to the SDGs and exactly replicate the wording of the goals and indicators. However, whilst a project needs to be aware of the SDGs to contribute to their official reporting, it is possible to contribute to their achievement without knowing what they are. Avoiding mentioning the SDGs in some of the questions (Table 4) therefore means that people who are not aware of the SDGs are not put off from answering the question, and any impact they are having on the achievement of the SDGs is captured. A general need in impact assessment is to capture unexpected or unintended impacts, which, in the case of this study, is only possible if project coordinators answer all the relevant questions, and are not put off from answering a question because it uses unfamiliar terminology, for example, referencing the SDGs.

Moving beyond the language used in the SDGs also allows the wording of the questions to be adapted to suit a wider range of projects. The very specific wording of the SDGs might make them seem irrelevant to projects, even when the project has an impact in that area. Examples which arose in this study include the inclusion of sustainable agriculture in SDG 2 “Zero hunger” and air quality in SDG 11 “Sustainable cities and communities”. In response to a question on SDG 11, one participant commented: “[the project] is not urban air quality in this sense, it’s air quality but also around non-urban areas”. In recognition of these cases, the questions deviate from the original wording of the SDG to make the question easier for respondents to understand, even if they are not familiar with the SDGs. Departing from strict replication of the language used in the SDGs also allows for inclusion of particular topics, such as mental health. Mental health and well-being are only mentioned as a small component of target 3.4 of SDG 3, “Good health and well-being”, and the one corresponding indicator, 3.4.2, addresses suicide mortality rate, which, whilst important, does not reflect the broad range of mental health issues people face in the Global North [26]. Given the potential for citizen science to positively impact participants’ mental health and well-being [27], the topic is given a prominence in the questions that deviates from the SDG wording. Adapting the language of the SDGs for the questions in Table 4 helps to ensure that all contributions to these topics are captured, even if the project coordinator is not aware of the SDGs or the details of their targets and indicators. Equally, maintaining explicit references to the SDGs in a subset of questions (Figure 1) means the direct contributions to the monitoring of SDGs are still recorded and that the MICS platform will introduce the SDGs to project coordinators who might not have considered them before.

4.3. Relevance of the Questions to the SDGs and Limitations of the Approach

The MICS questions give a good indication of which SDGs a project might have an impact on. However, there are several limitations to the extent to which the questions can be used to assess citizen science’s impacts on the SDGs.

Firstly, while there are questions which relate to every goal and a question which asks about the monitoring of every indicator, with regard to a project’s contribution to the achievement of the SDGs, the questions only cover a small proportion of the targets (30%) and indicators (3%). Of particular note is SDG 9, “Industry, innovation and infrastructure”, which currently has no related questions to its targets and indicators. The selection of questions, including most of the questions related to the SDGs, was based on a literature review of current citizen science impact assessment methods [18]. Some additional questions were added based on the SDGs which were chosen to match the authors’ knowledge of existing citizen science projects and the topics which they cover. Whilst it is still possible that projects are related to a target or an indicator not covered in the questions, it is likely that the questions will be sufficient for the majority of existing citizen science projects. Nevertheless, because they do not cover all the targets and indicators, the questions cannot be considered a truly comprehensive assessment of impact towards the SDGs.

Furthermore, there is uncertainty as to how well the questions capture a project’s impact. Two of the themes which arose from the thematic analysis were related to uncertainty: one for the uncertainty of how to interpret the question and one for the uncertainty of which answer to select. In total, there were 88 comments made by participants in these themes across 43 questions (with a total of 500 possible responses). This number of comments shows that, on average, 18% of responses to the questions involved some level of uncertainty that the question had been interpreted and answered correctly. Over the course of the study (including two rounds of edits to the questions; one after the workshop and one after the first round of interviews), there was no significant change in the proportion of comments relating to uncertainty, implying that the initial refinement of the questions did not help to improve the certainty of project coordinators in their responses. One potential reason for this could be because the questions were only tested with a limited number of project coordinators, which might not have been enough to perfectly refine the wording of the questions and answers.

The testing process revealed that even small changes to the wording of the questions can significantly alter their interpretation. For example, one of the themes which arose from the thematic analysis relates to the seven questions which contain the word “explicitly”. The comments made by interviewees suggest that, whilst the project coordinators might have answered positively to the questions thinking of a tenuous impact, having seen the word explicitly, they answered that they had not had that impact. Even if, in this case, the wording helped to reduce the ambiguity of the question, a level of subjective interpretation is inevitable within the questions. When the questions are implemented within the MICS platform, they will therefore be displayed alongside help and information text which will be written to aid the interpretation of the question. For example, key terms in the question will be defined and examples will be given of project activities which would fulfil the requirements of the question. The language of the questions and answers was further refined after the second round of interviews, to be tested alongside the help and information text in the MICS platform.

The MICS questions are designed so that they can be used to assess a project at any stage, i.e., to predict the impact of a project which has not yet started, to measure the current impact of a project, and to summarise the impact of a project which has finished. Therefore, whilst efforts are being made to reduce the ambiguity in the wording of the questions, the questions must be sufficiently open to interpretation, such that they can be used at these different project stages. In the interest of usability, the final decision of how to interpret questions will be left to the user. However, to help aid this decision, when the questions are implemented in the MICS platform, specific guidance will be given as to how the questions should be answered at each project stage, so that there is increased consistency as to how the questions are answered both between projects and within the same project (for example, not answering about future impact for some questions and current impact for other questions). Given this limitation, it is important that the interpretation of any impact assessment based on these questions matches the interpretation of the questions, so that a project coordinator’s answers to a question can be understood in the correct context.

Finally, because the wording of the SDGs has been adapted and altered to make the questions more accessible, some questions no longer exactly match the original target or indicator. As discussed in Section 3.4, this means that a positive response to some of the questions does not guarantee the project is having a positive impact on the corresponding SDG. Given these limitations, any claims made about the impact of projects on the SDGs based on the questions of this study should follow the principles of impact integrity, as defined in the SDG impact standards: not overstating positive impacts, placing impacts in the appropriate context, and being transparent about gaps and limitations in understanding about impacts [28].

5. Conclusions

This paper presents a practical approach to assessing the impact of citizen science towards the Sustainable Development Goals. This approach reflects two distinct pathways to impact for citizen science: monitoring progress towards achieving the SDGs, and contribution towards the direct achievement of the SDGs. Building on existing impact assessment frameworks for citizen science (for example, Kieslinger et al. [29]), this paper shares an exhaustive set of indicators, in the form of 51 questions and—for the first time—the corresponding answers, to assess the impact of citizen science towards the SDGs. This set of indicators is part of a more extensive framework, comprising of over 200 questions to measure the impact of citizen science across five domains: society, the economy, governance, the environment, and science and technology. These questions will be incorporated into an interactive interface within the MICS platform alongside additional guidance and recommendations for citizen science impact assessment. Given the potential limitations of the approach identified in this paper, the questions will continue to be tested with a larger number of project coordinators from a diverse range of projects as part of the development of the MICS platform. This will help to ensure that the questions cover all relevant SDG targets and indicators, the wording of the questions and answers is clear and easy to understand, and that users’ interpretation of the questions is in line with the intended meaning of the questions.