Bridging Science and Practice-Importance of Stakeholders in the Development of Decision Support: Lessons Learned

Abstract

User-friendly, evidence-based scientific tools to support sanitation decisions are still limited in the water, sanitation and hygiene (WASH) sector. This commentary provides lessons learned from the development of two sanitation decision support tools developed in collaboration with stakeholders in Uganda. We engaged with stakeholders in a variety of ways to effectively obtain their input in the development of the decision support tools. Key lessons learned included: tailoring tools to stakeholder decision-making needs; simplifying the tools as much as possible for ease of application and use; creating an enabling environment that allows active stakeholder participation; having a dedicated and responsive team to plan and execute stakeholder engagement activities; involving stakeholders early in the process; having funding sources that are flexible and long-term; and including resources for the acquisition of local data. This reflection provides benchmarks for future research and the development of tools that utilize scientific data and emphasizes the importance of engaging with stakeholders in the development process.

1. Introduction

In recent years, the water, sanitation and hygiene (WASH) sector has witnessed an increase in the development of various approaches to support service delivery [1]. These are mainly frameworks, planning guides and documents that present technical infrastructure options [1,2,3,4,5]. For instance, the Community-led urban environmental sanitation planning guidelines provide WASH practitioners with decision-making processes that allow for broad community involvement for inclusive service delivery [3]. On the other hand, the Compendium of sanitation systems and technologies provides practitioners with infrastructure options that allow for safe excreta management from containment to end-use or safe disposal [5]. Whilst these approaches are often freely accessible, their use by WASH practitioners in developing countries is still limited [1,6]. The reasons why uptake has been poor include: complexity, comprehension by decision makers, not adaptable or relevant to the local context and reality, lack of interconnectedness with other related societal services and poor dissemination [1,6,7,8]. Thus, to improve the usability of decision-making support tools (DST) by WASH practitioners, these limitations need to be revisited and minimized in future tool development.

To date, most existing tools in the WASH sector fall short in terms of their ability to illustrate to practitioners how changes and improvements in wastewater or fecal sludge infrastructure or management practices can impact pathogen reduction and mitigate subsequent health impacts from exposure to pathogens in the environment [9,10]. The United Nations Sustainable Development Goal 6 (SDG6) on access to water and sanitation for all, and the WHO guidelines on sanitation and health both emphasize the need to go beyond infrastructure to integrating service quality and management to mitigate excreta-related pathogens [11,12]. The Global Water Pathogen Project (GWPP; https://www.waterpathogens.org/) (accessed on 17 July 2020), which started in 2015, created an open access knowledge hub on pathogens in excreta, including an online book (Sanitation and Disease in the 21st Century: Health and Microbiological Aspects of Excreta and Wastewater Management) which was meant to serve as a resource and add value to existing DSTs [13]. However, to increase the value of this resource, it was noted early on that the complex scientific language presented in the online book needed to be translated into lay terms and made more accessible for use by practitioners to support WASH decision making [6,14,15,16]. Studies show that a wide gap still exists between science and practice, and there is an increasing realization that processes enabling knowledge exchange between scientists and decision makers are fundamental in promoting the development of effective DSTs [17,18]. To move science beyond its traditional technical boundaries and translate knowledge into understandable and usable forms requires stakeholders to be placed at the forefront as active contributors in the development of tools [17,19]. This philosophy was embodied by the Knowledge to Practice (K2P) project, which aimed to develop online tools to complement the GWPP book and support sanitation decision making. The goal of the K2P project was to develop tools that were user-friendly and suitable to the needs of decision makers. The tools were designed to help practitioners select sanitation and wastewater treatment technologies and management practices to ultimately reduce pathogens in receiving environments. The team placed stakeholders at the forefront of the development of the DSTs. While engagement with stakeholders is always identified as important in the development of tools [20,21,22], the process is often inadequately documented and thus can be difficult to study and replicate. We think it is important that others learn from our experiences so that they can develop relevant tools for the WASH sector or other sectors seeking to bridge science with practice.

Consequently, this commentary documents the evolution of the K2P project in the development of two DSTs (Pathogen Flow and Mapping tool, and the Treatment Plant Sketcher tool) and provides lessons learned from the perspective of the implementing team. The DSTs developed by the K2P team provide improvements over existing planning approaches and tools in the WASH sector, to allow for better control over health outcomes [2,3,4,23]. The tools both utilize models to predict the fate and transport of pathogens through the sanitation service chain and the environment, with additional inputs about the design and operation of the sanitation systems. Users can also collect and upload their own data or link to external data sources.

The Pathogen Flow and Mapping (PFM) tool enables WASH practitioners to spatially visualize pathogens in the environment under current sanitation conditions and under future conditions (as selected by the user). The model behind the tool uses population, sanitation coverage (access, toilet type), pathogen shedding and wastewater treatment removal data to simulate pathogen emissions into receiving surface water environments. It allows users to identify hotspots and areas for improving sanitation coverage and test how changes in the sanitation service chain (emptying of toilet, etc.) can affect pathogen reduction and help with the prioritization of sanitation upgrades. The model produces maps that illustrate relative differences between baseline and potential future scenarios where the user can vary inputs such as sanitation coverage, toilet type, wastewater treatment type, onsite sanitation emptying frequency, population growth, etc. Areas with relatively higher amounts of pathogens reaching the environment (hotspots) during baseline analyses can become priority locations for action (www.waterpathogens.org/) (accessed on 25 April 2021).

The Treatment Plant Sketcher tool allows practitioners to predict pathogen flows and reduction through an existing wastewater or fecal sludge treatment facility, or a new facility, by “sketching” that facility’s configuration and specifying information about the treatment unit processes used. The tool allows users to make design changes to the plant and see how these changes would impact the pathogen removal efficiencies of the wastewater system (reduction in viruses, bacteria, protozoa and helminths), and the fraction of pathogens emitted from the treatment plant in the treated liquid effluent vs. the sludge/biosolids (www.waterpathogens.org/) (accessed on 25 April 2021).

2. Methods

2.1. Development of the Knowledge to Practice (K2P) Sanitation Decision Support Tools

The development of these tools was a collaborative effort between scientists from different academic institutions, information technology (IT) specialist and partner WASH practitioners in Uganda. These scientists were from Michigan State University, San Diego State University, Temple University, Wageningen University and Research, and Venthic Technologies. In Uganda, the team mainly collaborated with the National Water and Sewerage Cooperation, the Ministry of Water and Environment and the Kampala Capital City Authority. The team also engaged with different WASH practitioners locally and internationally, to gain their input in the development of the tools. Michigan State University provided overall coordination of the project, San Diego State was responsible for the modelling of the pathogen flows and the development of the Treatment Plant Sketcher tool, Temple led the stakeholder engagement component, Wageningen was in charge of developing the Pathogen Flow and Mapping tool, and Venthic Technologies was in charge of information technology and the development of a user-friendly web-based interface of the tools (www.waterpathogens.org) (accessed 25 April 2021). In-country collaborators in Uganda as well as WASH practitioners globally at the municipal and national level were the target users of the tools. Academic and other capacity development institutions, however, were also interested in using the tools in training of existing and the next generation of WASH service professionals.

2.2. Approach

The goal of this commentary was to summarize the K2P team’s experiences in the development of two sanitation decision-making support tools, specifically related to stakeholder engagement. The information presented herein is a compilation of the experiences of the first author, who led the most stakeholder engagement activities in Uganda as well as collected data through open-ended questionnaires from the team members. Members of the K2P team were asked to complete a self-administered questionnaire with largely open-ended qualitative questions. This was followed by a virtual interview to follow-up on responses that needed more clarification or questions that were not adequately answered. The first author developed and sent out the questionnaire and then interviewed the team members who were also co-authors of this paper, to understand their view of the process for developing the tools and their experience with stakeholder engagement.

By academic training: two of the respondents were environmental engineers, two were environmental scientists, one was a microbiologist, one was a social scientist and one was an information system specialist. Three members of the project team had (non-academic) implementation experience in developing countries.

3. Experiences in Developing Sanitation Decision-Making Support Tools

This section summarizes the overall experiences of the K2P team’s engagements with Uganda stakeholders in the development of the tools. Table S1 summarizes some of the key points from the different team members’ perspectives.

3.1. K2P Project Team Engagement with Stakeholders and Expectations

The K2P project started with a high-level launch meeting in Uganda (July 2018) to create awareness and start meaningful stakeholder interactions. The meeting targeted top management in both government and non-government organizations and institutions to make the project and its objectives known as well as gain buy-in to kick-start the engagement processes with stakeholders in the WASH sector. The first meeting was followed by a series of engagement meetings with medium-to-large groups of participants, or in one-on-one settings. The team keenly documented stakeholder contributions during engagements and frequently shared updates with stakeholders (through e-mail and face-to-face meetings) on how their feedback was used to improve the tools. The K2P team from the onset was determined to develop tools that were responsive to the needs of WASH practitioners so that they could use them for sanitation decision making. For example, one of the members of the team said:

“I expected that stakeholder engagements would prevent all of us from spending time producing outputs that are not useful to practitioners. It would enable us produce tools that meet the needs of decision makers.”

This resonates with past research that revealed why some of the existing tools in the WASH sector are not used by practitioners, especially in developing countries, citing their lack of knowledge of their existence, complexity or them not being tailored to the needs of the decision makers [1,6,8]. To limit such shortfalls, the implementing team identified various expectations they hoped to achieve by engaging stakeholders in the development of the tools such as:

(i)

Understand stakeholder needs and collaborate in the development of tools that are relevant to their decision needs;

(ii)

Gain feedback on the concepts of the tool(s) to guide the development process and buy in on the tools to increase the likelihood of them being used;

(iii)

Gain insights for improvement and better use of the tool(s);

(iv)

Let WASH practitioners take the lead in the testing and validation of the tools;

(v)

Let stakeholders own the development process of the tools and use as much as possible.

Kizito and colleagues (2009) also highlight some of the aforementioned expectations in their study on the development of DSTs for decentralized urban water supply management in Uganda. The authors highlight how user involvement in the development of the tools and targeting the right users to participate in the engagement process are fundamental for effective decision support provided by the tools [21].

3.2. Influence of Stakeholders on the Direction of the Tools

From the first interaction with WASH practitioners in Uganda [6] and a series of interactions that followed, the K2P implementing team had a number of reflections that influenced the direction of the tools. In an interview with the lead author, one of the members on the team said:

“After a year of engagement with Ugandan stakeholders, we have learned a lot of their priorities and realities. We have also had a reality check about communication between academics and practitioners.”

The first engagement with stakeholders led to changes in the concept of the Treatment Plant Sketcher tool and to the development of simplified fact sheets on sanitation technologies that were easy to understand. Although the main concept of the mapping tool was unchanged following feedback from WASH practitioners, practitioners proposed real-world applications with regard to using the tool that were not originally envisioned by the K2P team and that shaped the development of the tool. For example, the stakeholders emphasized the importance of including onsite sanitation systems as one of the inputs for the mapping tool since they were missing from the first iteration. Following the workshop, the team had more clarity on the target audience for the different components of the tool(s), with the mapping tool regarded by stakeholders as more versatile, while the pathogen flow tool was most relevant to specialized groups that focused on centralized wastewater and fecal sludge treatment-like utilities. At the end of the first workshop, stakeholders also identified how they wanted to be engaged in the development of the tools moving forward [6].

The K2P team expressed open-minded sentiments from the start of the project, they recognized the relevancy and contribution of stakeholders in developing tools that would be usable by WASH practitioners. One of the team members said:

“While the team had some ideas on what they wanted to realize out of the project, they were not deeply rooted in them. This flexibility allowed stakeholders to voice their opinions and not seem forced in a certain direction.”

As many studies indicate, the involvement of stakeholders in the development of tools creates trust in the tool results as well as the acceptance and use of results in decision making [24,25].

3.2.1. Evolution of the Tools

At the start of the project, the K2P team presented the initial concepts of the tools to WASH stakeholders in Uganda for their insights and to gain an understanding of whether the tools were responsive to their decision-making needs. Through the feedback gathered from the first workshop and other engagements that followed, the team refocused and refined the tools so they were less complex while at the same time ensuring that they were robust enough to meet stakeholder needs. The implementing team continually engaged with WASH stakeholders at different prototype stages to ensure their inputs were integrated, as well as noting new areas for improvement (see Supplementary Material Figures S1 and S2 for the evolution of the tools). The first workshop conducted by the K2P team was held over a 2-day period with a large number of stakeholders. The subsequent engagements were held at the stakeholders’ workplaces, in small groups or in one-to-one settings. These meetings were purposefully done to reduce the travelling burden on stakeholders and to encourage more participation. In the wake of the SARS-CoV-2 pandemic, virtual meetings were adopted to maintain engagement with stakeholders. The success of the engagement process was largely due to the time and investment made in working actively with stakeholders all through the development process. One of the K2P team members was based in-country, hosted by one of the local collaborating partners in Uganda, to ensure continuous interactions with all engaged stakeholders. Following engagements with stakeholders, the tools went through 5–7 different iterations, evolving into the Treatment Plant Sketcher tool and the Pathogen Flow and Mapping (PFM) tool. One team member shared the view of the tool evolution process as follows:

“The tools have changed since we started, and in a way, become more simplified to meet the needs of stakeholders. For instance, there was need to split apart onsite and centralized portions of the PFM tool to meet the needs of two different organizations who work separately on different sanitation components of the sanitation service chain.”

3.2.2. Collaborative Case Studies

Case studies were developed collaboratively between the K2P implementing team and some WASH stakeholders (KCCA and NWSC). The case studies were based on key sanitation questions the organizations wanted to answer. The K2P implementing team, led by the stakeholder component, held meetings with stakeholders to understand the kind of decisions the organizations were making. Through those discussions, key questions were identified that both parties thought the tools might be able to support. The process resulted in two case studies; one focusing on on-site sanitation using the Pathogen Flow and Mapping tool and the other on centralized treatment systems—using the Treatment Plant Sketcher tool. Through the case studies, stakeholders expressed a better understanding of how the tools can be used to inform sanitation planning and investment decisions and can relate with the outputs produced from the tool models. One other important thing that resulted from the case studies was the importance of the visualization of outputs (how the results were displayed). The case study on onsite sanitation systems illustrated how pathogen emissions in the city corresponded to areas that were prone to cholera outbreaks in 2019, and hotspots that could guide sanitation prioritization efforts. On the other hand, the centralized treatment case study illustrated that if the Lubigi wastewater treatment plant in Kampala added two maturation ponds after the facultative pond and stored fecal sludge and pond sludge in drying beds for about 90 days, they would see an estimated 1.5- to 2.0-fold increase in overall pathogen reduction efficiency. On average, this translated to potentially preventing from ~1.5 to ~2.2 trillion pathogens from contaminating surface waters and farms each year.

From the case study experience, a member of the K2P team remarked:

“I did not think that visualization of the outputs were important until later when we were doing the case studies to answer stakeholder questions and were wondering what this information would look like.”

A key lesson learned during the development of the above case studies was that for stakeholders, a case study is very different from what most academics perceive it to mean. This was a challenge even within the study team. For over a year, the team debated as to what a case study meant from the perspective of a stakeholder. The academics on the team believed that a case study was an exercise where the tool developers tried the tools with data provided by stakeholders to see if they worked. For others on the team with more experience in implementation, they maintained that stakeholders should drive a case study in the context of this project and ideally the stakeholders should be able to use the tools independently to answer their own questions and then the K2P team would only assist when needed. It took the K2P team over a year to reach a consensus on the topic and they established that two forms of case studies were required: one set that is an academic exercise and then a second set that is driven by stakeholders to test the true utility of the tools. Unfortunately, the project was cut short and only one fully stakeholder-driven case study was conducted by NWSC where they used the Treatment Plant Sketcher Tool to answer some of their own questions regarding the upgrading of wastewater treatment systems in the country. The K2P team believed that numerous more case studies driven by stakeholders from different contexts around the globe are required to ensure the versatility of the tools. For those working on similar projects, this is a key recommendation from the project team.

3.2.3. Simplification of Decision-Making Support Tools with Global Datasets

The K2P team realized that, in many contexts, there is an absence of data and therefore, an absence of the tools needed to include a default data option to simplify their use by a wider audience. Some of the inputs for the K2P tools populated default data from the JMP, GWPP and other existing literature, such as on disease prevalence. The default data, when run, provide baseline scenarios for decision makers. However, where country-specific or regional data are available, users of the tools can also use their own data to re-define their baseline or produce scenarios to help inform their sanitation decisions. Default data were found to be necessary for situations where groups did not have their own data. The WHO and UNICEF are the global bodies responsible for the JMP and tracking progress towards the water and sanitation sustainable development goal 6 and provides global data on the progress of most countries [26]. The K2P team leveraged this respected, global dataset to be utilized in the Pathogen Flow and Mapping tool.

3.2.4. Stakeholder Motivations for Engagement in the Development of the Tools

The motivation of WASH practitioners to participate in the development of the DSTs evolved based on the value they saw in the tools. From the start, stakeholders were made aware that the project had no provisions to pay for their participation in the K2P workshops, and therefore, were not compensated or paid throughout their involvement in the project activities. However, some stakeholders asked about compensation—as they were accustomed to compensation for their participation in other project workshops. Based on the above premise, among all the stakeholders involved from the start, three major partners emerged (Ministry of Water and Environment, National Water and Sewerage Corporation, and Kampala Capital City Authority) who led the sanitation service delivery. While some participants were whole-heartedly engaged from the start, others became more interested gradually over time. The commitment of some stakeholders in this project concurs with what was reported in a study on the design of a decision support tool for visualizing E. coli risk on agricultural land [17]. Oliver and colleagues (2017) observed that the enthusiasm of different stakeholders involved in the design of their tool were not simply figureheads representing their organizations but showed up to be engaged and provided constructive criticism and feedback which led to the development of credible tools. Additionally, local academic institutions also emerged as an important stakeholder group in our process. The engagement with local institutions like Makerere University allowed the project team to collaborate with local professors who are training future WASH practitioners and can further utilize the tools and integrate them in training modules for students. Moreover, these faculty members could serve as consultants for running the tools for interested organizations that do not have the capacity or resources to run them independently but would be interested in contracting someone to do it for them. For example, one of the team members observed that while some practitioners were interested in the outputs from the tools, they were not interested in running the models. Whilst the K2P team tried to make the tools as simple and user-friendly as possible, there are likely going to be stakeholders who will need a skilled group of people to run the tools for them and provide them with the outputs to inform their decision making. For example, engineers and planners from organizations or utilities may need to run the models and provide the synthesized output for top management to support decision making and resource prioritization. In some cases, academics from the local universities may also be able to provide this service.

3.3. Lessons from Series of Stakeholder Engagements and the Overall Project

In this section, we summarize the key lessons the K2P team learned from its engagement with stakeholders and provide recommendations for scientists interested in bridging science and practice through the development of tools.

• The tools need to be as simple as possible. The tools need to be as simple as possible for those who will use them. While some members on the team mentioned knowing that the tools needed to be simple, they admitted that you cannot know when something is easy unless you have tried it out. This explains why some decision support tools are never utilized by decision makers [1,8]. The issue of the complexity of the tools and guides in the missing link between practice and theory is reported as one of the reasons for not using some of the existing tools by decision makers [8]. Engagement with stakeholders was important in figuring out whether people were able to easily use the tools developed. The information technology expert on the team stated:

‘’It is always simple to circulate ideas, but these should be tested in frequent and actionable engagement sessions with end-users.’’

2.

An enabling environment needs to be created for stakeholders to encourage active participation. From the engagements with WASH practitioners, we learned that having an environment where people can speak freely and provide honest feedback is important to improve the tools and for stakeholders to use them. One of the team members said:

“It becomes much more difficult to engage in development of such tools when participants or members of the team do not speak openly due to alternative or competing interests.”

The K2P team encouraged honest feedback on the tools from WASH practitioners and tried to do lots of attentive listening while remaining open minded to obtain as much input as possible. Another member on the team expressed the following:

“Putting ourselves in the shoes of stakeholders is important. For example, our ability to reframe some of our discussions with the stakeholders to suit the decisions they were making helped us dive in into how these K2P tools were going to be relevant in meeting their needs. We figured out ways to communicate with stakeholders to be on the same page.”

The organization and individuals’ decision-making needs should be understood and considered. Understanding the needs of the organizations and stakeholders engaged is important in informing the development of the tools to tailor them to the service questions they seek to solve. An important aspect that was mentioned by a team member is drawing a line between what is feasible and not within the project scope early in the process. This member stated:

“While all feedback raised by a stakeholder is important, not all may be used.”

In the case of the K2P tools, we had to explain to stakeholders where our tools complement existing tools in the WASH sector and what suggestions were outside the scope of the project. For example, stakeholder views on costs, other treatment outside pathogens, and the distribution of risk with underground water reinforced the fact that the K2P project addresses one specific need within the decision-making cycle. These suggestions also bring out areas for future considerations in improving the versatile application of the developed tools. Understanding the needs of stakeholders (individuals, groups or organizations) and how this affects their participation has been identified in previous studies as important in influencing the success of the projects [3,27].

3.

A dedicated team should be responsible for planning and executing stakeholder engagement activities. The K2P team had an ambitious plan of developing the tools with stakeholders in Uganda in under two years. One of the lessons the team learnt is that managing different inputs from stakeholders can be quite complex, requiring more preparation, multiple engagements, many revisions and most importantly, time. Thus, depending on the nature of the tools, developers and funding agencies need to be flexible and patient to allow for sufficient engagement with stakeholders. One of the members of the team said:

“Having social scientists on the team that know how to relate with stakeholders and keep long-term relations with them is necessary before any meaningful work can be accomplished.”

The above quote equally relates to another member who said:

“When I was contacted to be part of the team, I knew we needed to identify a strong person on the ground to be involved in the management of stakeholders.”

The K2P team recognized that having a dedicated person on the ground to bridge the team with stakeholders and execute their contributions was crucial—something not all projects led by foreign institutions recognize. This concurs with previous research that also shows the importance of improving communication with stakeholders as well as the intensity of interactions between academics and policy makers in informing decision support tools and influencing the use of academic research [28,29].

4.

There needs to be early engagement with stakeholders and selecting the “right” stakeholder partners. Whilst it takes time to build these relationships, having interested organizations with interested individuals that already have the questions for which the tools can seek to answer expedites the development process. It is important to involve stakeholders in the development and proposal stage of projects for better buy-in and engagement even before the project receives funding or begins. The advantage of engaging with stakeholders in the early development of tools is to understand their needs and questions such that the tools are tailored for them right from the beginning. In addition, a clear plan on the implementation and co-creation of the tools can be conceived. This is different from what was done with the K2P tools where we (the scientists) started with concepts for the tools and then went to engage stakeholders. It would have been ideal if there could have been more time to select and work with stakeholders, even earlier in the process. One of the members of the team said:

‘‘I would prefer to go to stakeholders at the start and say that these are the things that could potentially be important to improve sanitation decisions. Would you be interested in them? This is different from imposing the tools on stakeholders. This is however constrained by the nature of funding agencies who want to look at what you are going to do before giving you funds. It would be ideal to engage with the stakeholders to develop the proposal, but it is hard to get funds to support this type of engagement before the project start.”

This lesson is supported by previous studies in fisheries which highlighted the importance of involving stakeholders in the early stages of framing projects and through the lifecycle of the projects for the effective use of evidence in decision making [30,31]. In a study on the design of a decision support tool for visualizing E. coli risk on agricultural land using a stakeholder-driven approach, authors reported that the involvement of stakeholders within all stages of the support tool design—from inception, idea formulation through to testing—promoted enthusiasm for the end-product, created trust and understanding of the processes as well highlighted aspects that are often never considered by technical project teams [17].

5.

A team with the right composition and ability to collaborate is essential. The inclusion of local scientists is important in research and development projects stemming from the “North”. The experience of good local collaborators and scientists is important to ensure the team understands the local context and processes in engaging with stakeholders as well as ensuring that the planned tools are versatile enough to fit into country contexts. One of the reasons cited for the failure in use of decision support tools has been their inability to be adapted or contextualized to the local context and realities [1]. Having a social scientist on the K2P team was an advantage, especially in issues related to stakeholder engagements and ensuring the use of simplified terminologies instead of more scientific/academic language. One of the members of the team stated:

“We have a great team that is engaged and works together well. However, even with this great team, we can get lost if we do not communicate enough and regularly. You need to build a good system for the communication within the team and with stakeholders.”

6.

Longer term funding is needed from donors/sponsors so that meaningful engagement is possible. From this project, we learned that tools that engage stakeholders can take longer than initially planned due to:

(a) Delays in coordinating meetings with stakeholders above and beyond their daily work schedules;

(b) Allowing for sufficient time and resources to make multiple revisions to tools to respond to the stakeholder needs;

(c) Including a time lag for developing a common understanding with stakeholders to realize importance of the tools;

(d) Having the tools pre-tested by stakeholders; and

(e) Understanding local data limitations that delay the contextualization of the tools to stakeholder settings.

Whilst the K2P project planned to develop sanitation decision support tools within two years, our experience from Uganda is that this was an unrealistic timeframe, and in the end, it took 2.5 years to have tools ready for widespread testing. The project was however, cut short at 2.5 years due to the COVID-19 pandemic and funding constraints. At least an additional 6 months would have been needed to adequately test and refine the tools with a larger stakeholder community in different country settings. The consecutive stakeholder engagements to ensure that the tools communicated to users in terms of language and context proved to take much longer than initially expected. In addition, the tools transformed so much from the initial concepts that major revisions to the architecture and user interfaces of the tools were needed—which was time consuming and costly. In such situations, it is important that funding for these types of projects is flexible enough to allow for continuity so that they can achieve their desired goal of tools that will be used by stakeholders. A member of the team said:

“To have these tools extremely iterative means going through many versions, which could take 5–8 years to have them fully working and applicable in a wide variety of context and application so that they are quite user friendly.”

7.

Resources/funding for acquisition of local data. Projects of this nature should allocate sufficient resources/funding for the collection or acquisition of local data to further refine and validate the tools in a variety of contexts before widespread use. This is important in illustrating the relevance of the tools and that the results produced are realistic for a variety of settings. This lesson concurs with findings from previous studies that emphasize the importance of local evidence in informing decision making [27,28]. A study on support tools for evidence-informed policy making in health contends that local evidence is more directly relevant in informing decisions than studies conducted elsewhere [28].

4. Conclusions

This commentary summarized the K2P team’s experiences in the development of two evidence-based sanitation decision-making support tools. To the best our knowledge, this is the first study in the sanitation sector to share lessons learned in the development of decision support tools. The team concluded that active stakeholder engagement was key in the development of the tools and that adequate resources and time be dedicated to involving stakeholders early in the process. It was also imperative that the tools be as easy as possible to use and be tested with stakeholders in their context to ensure they are responsive to their needs. The reflections presented herein provide benchmarks for future research involving the development of tools that utilize scientific data and emphasize the importance of stakeholders in the development process.