Handrails through the Swamp? A Pilot to Test the Integration and Implementation Science Framework in Complex Real-World Research

Round 1

Reviewer 1 Report

The article " Handrails through the Swamp? Testing the Integration and Implementation Science framework in complex real-world research " offers the application tools of a unified approach to these studies on the example of heterogeneous research projects.

Since such tools allow for the integration of various research projects, this ensures the stability of the development of scientific knowledge.

Author Response

Please see the attachment

Author Response File: Author Response.docx

Reviewer 2 Report

The article is interesting and proposes a tool for carrying out a valuable comparative analysis of various case studies in the context of i2s. However, some points need to be clarified.

1. The selection of these and not other case studies requires wider justification. Criteria should also be provided to precisely classify the studied cases to a specific type (type: covering a range of environmental and agricultural problems?). It seems that in the case of the presented research it is difficult to talk about their representativeness. Therefore, it is necessary to clearly indicate common features of the studied cases in order to avoid the impression of generalization to any case studies concerning complex real-world research. Therefore, certain definitions are missing.

2. As for the test procedure itself, more space should be devoted to the description of the limitations of the tests performed. (Maybe the study should be treated as a pilot?)

3. "After testing, we identified 43 (discrete?) Elements across the three domains, with domain 1 containing 19, domain 2 containing 9 and domain three containing 15. Appendix A presents a table in which the three domains, the four questions in each domain, and the elements associated with each of the questions are listed. "
What was the aforementioned testing procedure?

4. Why was the interview technique used to study some case studies and the survey technique to study others?

5. Why was the evaluation carried out by three analysts? What were their competences and experience in this type of work?

6. We are dealing with ordinal variables. What type of correlation was calculated? Rho?

7. It seems to me that using the quantitative analysis tools in the description of qualitative phenomena looks better if the set of surveys/assessments is wider (higher number of respondents or cases). If it is impossible to increase the number of analyzed case studies, maybe it is worth collecting more evaluations (i.e. asking a larger number of users)?

Author Response

Please see the attachment

Author Response File: Author Response.docx

Reviewer 3 Report

We can agree with the authors that socio-environmental challenges are complex, difficult, and uncertain. The objective of the paper appears sound to help disciplinary scientists navigate these “swamps”.

Nevertheless, the paper does not evaluate a broad enough state of the art. The framework for studying complex real-world problems needs to be situated regarding other approaches such as system engineering [1] [2] and model-based approaches [3]. As well the question of interoperability is crucial when heterogeneous participants (partners, systems domains) must talk each other [4] [5]. These approaches are not dedicated to socio-environmental studies, but they attempt to link domain and go from concepts to proposition of system solution. The Integration and Implementation Science (i2S) framework would gain in interest if it were related and situated regarding these works.

Then, the 7 transdisciplinary case studies presented in the i2S framework are not sufficiently didactically presented and detailed. We agree, with comparison to complex system engineering that the mastering of concepts in the i2S framework during the planning and delivery of research can increase the usefulness of the research for next users, and here again interoperability is key. But the mastering of concepts (semantical approaches for instance with ontology) is not demonstrated clearly how and how much it can be interesting for domain users.

The complexity of systems studied makes growing the needs of solution project management, integration. The i2S dimensions can propose a solution if more formalized and demonstrated. Nevertheless, the increased usefulness for the end users should be discussed with the concept of user-experience and should be more debated.

So, maybe that the i2S framework could provide handrails for researchers but is not clear with the presented methodology how and how much it will help users navigate the swamps when engaging with the complexity of socio-environmental problems.

1. Blanchard, B. S. (2004). System engineering management. John Wiley & Sons.
2. Forsberg, K., & Mooz, H. (1991, October). The relationship of system engineering to the project cycle. In INCOSE International Symposium (Vol. 1, No. 1, pp. 57-65).
3. Estefan, J. A. (2007). Survey of model-based systems engineering (MBSE) methodologies. Incose MBSE Focus Group, 25(8), 1-12.
4. Chen, D., & Doumeingts, G. (2003). European initiatives to develop interoperability of enterprise applications—basic concepts, framework and roadmap. Annual reviews in control, 27(2), 153-162.
5. Zacharewicz, Gregory; Daclin, Nicolas; Doumeingts, Guy; Haidar, Hezam. 2020. "Model Driven Interoperability for System Engineering" Modelling 1, no. 2: 94-121.

Author Response

Please see the attachment

Author Response File: Author Response.docx