Understanding Rural Water Services as a Complex System: An Assessment of Key Factors as Potential Leverage Points for Improved Service Sustainability

Round 1

Reviewer 1 Report

No comments.

Author Response

Reviewer 1 did not provide any comments to which the authors can respond. 

Reviewer 2 Report

Please see the attached pdf. 

Comments for author File: Comments.pdf

Author Response

Comment

 

Many references cited in the manuscript follow by an error message “Error! Reference source not found”, e.g. in lines 170, 175, 210, 232, 267, 278, 335, 365,  376, … To avoid potential mislabeling, please also provide the names of authors for every reference.

 

 

Response

We regret to hear that some of the references in this manuscript generated error messages. In reviewing the manuscript, we do not see these errors appear in either the .doc or .pdf versions of the document.  When we try to review the submitted files, we receive a message “Error 404 File not found”. 

 

Regarding the citation style, we are using the required MDPI style for this manuscript which requires sequential numbering of references in place of authors’ names. Please advise if error messages are present in the resubmission of the manuscript. 

Comment

 

Lines 231-234: "normalized score" for each feedback loop. Since it is a complex system, how would you ensure that the normalized score of the entire feedback loop is the average value of the direct effects of adjacent factors? This calculation method reflects a linear relationship and does not seem to reflect the complex non-linear relationship between factors (refer to lines 58-68).

 

 

Response

Thank you for this comment.  We calculated a ‘normalized score’ for each feedback loop in order to create a relative ranking of the loops through which the most – and least – dominant loops. As the unique combinations of factors and causal links present in the feedback loops are themselves the direct product of complex systems analytics (i.e. Vensim system dynamics software modeling, Lines 235-237), the process of quantifying the loops for the use of a comparative ranking system using an average of influence values does not diminish the complex insights that each feedback loop represents. We have added a clarifying statement to this paragraph that explains this distinction. 

Comment

 

Lines 244-245: The indirect influence matrix is generated by multiplying the cross-impact matrix by itself in iterations. The matrix product seems to be mainly used to describe the relationship between linear equations, not the relationship of equations in non-linear systems. How do you capture the non-linear relationship between complex systems of Rural Water Services?

 

 

Response

The indirect influence matrix that is generated via matrix multiplication is a non-linear process; that is, the matrix calculation generates influence values for each factor-on-factor influence that exponentially increase with each iteration. This is similar to, but different than, the non-linear behavior that results from the compounding or balancing behavior generated by combinations of inverse or positive polarity connections between the factors (described in Lines 248-257). The resulting behavior of both of these dimensions of the causal loop analysis is similar to, but should not be confused with, non-linear relationships in the non-linear algebra, differential or integral calculus.  Thank you for allowing us to point out this important distinction.

Comment

Lines 53-57: Interactions between factors are no less important than the effects of individual factors. The following four papers by Li et al. discussed the interaction between factors in a nonlinear dynamic system of water supply, that is, the impact of the combination of investment and labor input on the equilibrium of a water supply system. A comparison is recommended.

 

Li, K.B.; Ma, T.Y.; Wei, G.; Zhang, Y.Q.; Feng, X.Y. Urban Industrial Water Supply and Demand: System Dynamic Model and Simulation Based on Cobb–Douglas Function. Sustainability 2019, 11, 5893, doi:https://doi.org/10.3390/su11215893. Li, K.B.; Ma, T.Y.; Dooling, T.; Wei, G. Urban Comprehensive Water Consumption: Nonlinear Control of Production Factor Input Based upon the C-D Function. Sustainability 2019, 11, 1125, doi:https://doi.org/10.3390/su11041125. Li, K.B.; Ma, T.Y.; Wei, G. Multiple Urban Domestic Water Systems: Method for Simultaneously Stabilized Robust Control Decision. Sustainability 2018, 10, 4092, doi:https://doi.org/10.3390/su10114092. Li, K.B.; Ma, T.Y.; Wei, G. Robust Economic Control Decision Method of Uncertain System on Urban Domestic Water Supply. International Journal of Environmental Research and Public Health 2018, 15, 649, doi:https://doi.org/10.3390/ijerph15040649.

 

Response

Thank you for providing these references for review. While we acknowledge that these studies appear to have a similar scope to the work in our manuscript, each uses methods that are similar to ours in the context of terminology, only. The main difference in our approaches is that we are not developing, or seeking to develop, unique computational models based on the individual dimensions of factors, with an expressed goal of identifying unique coefficients within each model. In fact, all of the factors present in our study by their nature are unquantifiable, which is why we use focus groups and the relative ranking of influence scores to quantify the feedback loops in our study. 

 

Comment

Furthermore, please check if the following are relevant:

 

What are the characteristics of this kind of complex systems?

 

 

Response

As we describe in the introductory section (Lines 64-77), the systems which we are modeling in this study exhibit many of the hallmarks of complex systems; they are comprised of a multitude of ever-changing factors many of which have non-linear relationships between them; the result of this behavior is unanticipated and non-linear outcomes; many of these factors and their relationships are unquantifiable; there are multiple different perspectives through which to understand the structure of the systems, none of which are objectively ‘correct’; and, the characteristics of each of these systems is believed to be highly context-specific. 

Comment

 

Some of the selected references are too old. Latest references should be included.

 

 

Response

We have reviewed our reference list and determined that of the 80 citations in the manuscript, nearly half are from within the past five years, and 90% are within the past ten years.  Thus, we believe that the sources we use are appropriately up to date and most relevant to our study.   

Comment

 

How is the data obtained? Is it subjective data or objective data?

 

 

Response

The data for the systems analysis is obtained through a facilitated group model building (GMB) workshop called ‘Factor Mapping’. In this process we convene knowledge local stakeholders to describe the factors within the local system and their relationships to one another. Through this process we elicit information that we use to quantify the relationships between factors using a relative influence rating process. This allows us to translate qualitative information about the system into quantifiable data of complex factor interaction. We have added a citation in the Methods section (Lines 299-301) to studies which contain protocols for the factor mapping workshop. 

Comment

 

The „Factor Mapping‟ workshop format is used in this paper, but a specification of

 

„Factor Mapping‟ workshop format is not provided.

 

Response

Thank you for this comment, as we noted in a re-review of the manuscript that we did not include a specific reference to the studies in which we provide protocols for the Factor Mapping workshop practice. Please see Lines 299-301 of the Methods section for more information on this process. 

 

Valcourt, N.; Walters, J.; Javernick-Will, A.; Hollander, D. Summary Report of Baseline Iterative Factor Mapping and Learning (IFML) Analyses For Concept 1 Activities In Ethiopia And Uganda; Sustainable WASH Systems Learning Partnership: United States Agency for International Development: Washington, DC, 2019;

 

Valcourt, N.; Walters, J.; Javernick-Will, A.; Hollander, D. Summary Report of Baseline Iterative Factor Mapping and Learning (IFML) Analyses For Concept 3 Activities In Kamuli District; Sustainable WASH Systems Learning Partnership: United States Agency for International Development: Washington, DC, 2019

 

Comment

 

Through the analysis of the results, what meaningful conclusions have been reached? It should be clearly expressed in the paper.

 

 

Response

We believe that through the analysis presented in the manuscript we have shown that while participants across the contexts tend to identify a common set of factors, the interactions amongst those factors and their individual ability to influence service sustainability varies considerably across contexts. These findings suggest that a more intentional focus on factor interactions in WASH systems could lead to more effective strategies for improving service sustainability

Author Response File: Author Response.docx

Reviewer 3 Report

Thank you for this work and attempting to identify factors to improve water access. My primary concern is that this manuscript comes off as an academic exercise, rather than a true learning point for practitioners, NGOs, or local governments. There is an excess of buzzwords and layers of technical jargon/approaches that muddy the methods instead of clarify the findings.

Abstract - please be more explicit on the methods and findings. How many countries/contexts? How many group discussions? Etc. The basics of the full document should be included in the abstract.    Introduction - this section is well referenced, but please be more explicit with terminology and examples. For example, ..."  the sector has developed frameworks to measure the individual and collective capacity of factors to sustain services[13]."  Please, note the frameworks. Not all readers will be fully immersed in the sustainability literature, some descriptions are required.    Lines 58-71 - this paragraph seems out of place or unnecessary.    Users/beneficiaries were not consulted?    Was consensus needed between the groups to score the matrix?    The Causal Loop Diagram looks nice but I assume it is not very useful. Very busy and hard to see much of anything specifically.    Was this process worth it? The results are somewhat expected or at least not surprising, with more contexts/countries, would there be different or clearer results?   Are there next steps for this work?   

Author Response

Comment

My primary concern is that this manuscript comes off as an academic exercise, rather than a true learning point for practitioners, NGOs, or local governments. There is an excess of buzzwords and layers of technical jargon/approaches that muddy the methods instead of clarify the findings.

 

Response

Thank you kindly for this comment. This comment echoes much of the of the initial feedback and skepticism we received on this work, however, after conducting nearly a dozen of these workshops over the past five years in multiple countries and communities worldwide, we have seen that this approach has led to actual changes in how NGOs and local governments make important decisions around WASH. As one specific example,, the work we conducted in Kabarole District Uganda led to the participants  creating a task team that developed a District-wide Master Plan for WASH based on findings from the exercise. We have added more information regarding some of the material outcomes of the Factor Mapping process to the Limitations section (Lines 546-550) 

 

Comment

Abstract - please be more explicit on the methods and findings. How many countries/contexts? How many group discussions? Etc. The basics of the full document should be included in the abstract.    

 

Response

Thank you for this comment. To clarify, we conducted four workshops, one in each geographic context presented in the study. We have added this information to the abstract. 

 

Comment

Introduction - this section is well referenced, but please be more explicit with terminology and examples. For example, ..."  the sector has developed frameworks to measure the individual and collective capacity of factors to sustain services[13]." Please, note the frameworks. Not all readers will be fully immersed in the sustainability literature, some descriptions are required.    

 

Response

Thank you for this comment. In fact, the number of different frameworks the sector has developed for looking at the sustainability of WASH services are quite numerous and we did not think it appropriate to identify them in the introductory section without having adequate space in the text to discuss the relative characteristics of each. We believe that it is the existence of these frameworks, and not their specific merits, that are germane to our discussion in the Introduction. We have revised this section to note that there are ‘a wide range of’ frameworks that have been developed for this purpose.  The citation we provide for this statement (Boulenouar, J.; Schweitzer, R.; Lockwood, H. Mapping sustainability assessment tools to support sustainable water and sanitation service delivery) is a comparative analysis of many of the most prominent sustainability frameworks in the sector. 

 

Comment

Lines 58-71 - this paragraph seems out of place or unnecessary.    Users/beneficiaries were not consulted?   

 

Response

Your assumption is correct that much of the existing literature is based on external expert perspectives, with little input from users/beneficiaries. As the data in our study is directly elicited from local stakeholders, we believe that this section of the Introduction is quite important for readers to understand the intellectual merit of the study and how it compares to existing literature on the subject. 

 

Comment

Was consensus needed between the groups to score the matrix?    

 

Response

Yes. Values in the matrix were based on a group consensus basis. This was determined by the local facilitators in each context based on local practices and customs for group discussions, but most commonly consisted of a simple majority (more than 50% of the participants in agreement).  We have added a note to the Methods section (Lines 211-213) to clarify this point. 

 

Comment

The Causal Loop Diagram looks nice but I assume it is not very useful. Very busy and hard to see much of anything specifically.   Was this process worth it? 

 

Response

The Causal Loop Diagram (CLD) presented in the study is shown for illustrative purposes; it is not the primary output of the process. Through our work, we have found that workshop participants find the most value in reviewing the relative rankings of Influence, Dependence and Feedback scores for each factor displayed in a form similar to Table 6. Our local partners in this work have told us that presenting data in this form has spurred very insightful discussion and led to specific decision actions, including the development of a district-wide Master Plan for WASH in Kabarole District (Lines 546-550). This is also the reason we chose to use a cross-impact matrix to conduct the workshops as opposed to a CLD; it is often too complicated, visually, for the participants (and the facilitators) to follow the connections.  Therefore, we find that it is in the information contained in the CLD that is most useful itself, and not necessarily the specific graphic presented here in this study for the purposes of illustrating the physical connections between the factors. 

 

Comment

The results are somewhat expected or at least not surprising, with more contexts/countries, would there be different or clearer results?   

 

Response

Thank you for this question. Based on our knowledge of current practices in the WASH sector, we believe that the results of our study are both quite unexpected and of a very important intellectual merit.  Conventional wisdom in the WASH sector suggests that there are a wide range of factors required for sustaining WASH services and that these factors operate in identical or similar partners with other factors across varying geo-political contexts (Lines 83-91 - Introduction). In a clear contrast to this perspective, our work shows that local stakeholders tend to identify a common set of priority factors across contexts that are necessary for sustaining services, but they then go on to describe how the connections between these factors are in fact quite unique and context-specific (Lines 468-481 - Discussion). These findings both confirm certain dimensions of the status quo of systems approaches in the sector, and present challenging reflections for further implementation of these approaches in WASH (Lines 499-509 - Discussion). Thus, we believe that this work presents essential insights for the sector at this unique inflection point in the operationalization of systems approaches to WASH.

 

Comment

Are there next steps for this work?

 

Response

Yes, there are. In the concluding sentences of the manuscript (Lines 554-557) we outline some of the dimensions of this process which we are continuing to explore in future work, including; the repetition of workshops over time in the same contexts with the same participants; the influence of indirect influence values on Influence, Dependence and Feedback rankings; and, levels of alignment in cross-impact matrices for different groups of stakeholders in the same contexts.