Systematic Review of Agent-Based and System Dynamics Models for Social-Ecological System Case Studies

Round 1

Reviewer 1 Report

I think that this is an outstanding paper reviewing literature contributions of the use of Agent-Based Modeling (ABM) and System Dynamics (SD) modeling in the domain of social-ecological systems (SES), with an emphasis on the use of real-world case studies  to inform policy making processes. The context and goals of the study are clear and well presented. The review methodology is also clearly presented, as well as the data sets used for selecting papers. The analysis of the selected papers is well done and the authors provide an interesting summary of the strengths and limits of the ABM, SD and hybrid (ABM+SD) approaches. This synthesis results in some suggestions for future research in the domain. Well done!!   

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

This paper presents a literature review of ABM and SD (and their hybrids) for real-world SES case studies, focusing on their policy analysis applicability. I applaud the authors for tackling the difficult task of writing a review paper. I also think that the topic is essential and timely. Below I provide a few major and minor limitations of the manuscript that require refinement and that, hopefully, will improve its quality and, ultimately, recognition in the SES modeling community.

MAJOR

[1] The methodology used for paper selection: The fact that the authors focused only on the last ten years is troubling and needs justification. A cursory query on Google Scholar with a timespan before 2010 can provide several relevant papers worth mentioning.  The idea of combining ABM and SD is not new and dates back to the late 1990s. Also, narrowing down the search to ‘agent-based’ leads to the omission of papers that use alternative names for ABM, like ‘multi-agent’ or ‘individual-based’. The latter is more common in ecology (ref. to work by Volker Grimm), but there is no need to assume that ABM in SES is only applicable to social systems, so the inclusion of ‘individual-based’ as a keyword is a requirement.

[2] Positioning the paper’s objective within the literature reviewing modeling SES and the ABM-SD hybridization: in the introductory section, the authors posit that their review focusing on applied ABM and/or SD is unique. I would agree if I saw what had been published so far on SES modeling, which is missing from the paper. Quite a few papers are talking about different approaches to SES models. A couple of examples (but there’s more!):

Kelly et al. (2013). "Selecting among five common modelling approaches for integrated environmental assessment and management"

Mallampalli, Varun Rao, et al. 2016. “Methods for Translating Narrative Scenarios into Quantitative Assessments of Land Use Change.” Environmental Modelling & Software 82: 7–20.

Filatova et al. (2013) Spatial agent-based models for socio-ecological systems: Challenges and prospects, Environmental Modelling & Software, Volume 45, July 2013, Pages 1-7

Borshchev, A., and A. Filippov. 2004. From System Dynamics and Discrete Event to Practical Agent Based Modeling: Reasons, Techniques, Tools. The 22nd International Conference of the System Dynamics Society. Oxford, England.

Badham, J. (2010). A Compendium of Modelling Techniques. Integration Insights.

The latter two are not SES-oriented but provide an excellent classification of modeling approaches and demonstrate the major differences between SD and ABM, where the latter is top-down aggregate quantitative modeling and the former is bottom-up individual-based modeling.

While these do not focus explicitly on policy analysis, there should be a section discussing what we have learned so far from such reviews, before delving into the specific objective of this manuscript.

A related topic is ABM-SD hybridization. There has been considerable discussion in the literature on the pros and cons of hybrid models, both within and outside of SES. See for example:

Parker, Dawn C. et al. 2008. “Complexity, Land-Use Modeling, and the Human Dimension: Fundamental Challenges for Mapping Unknown Outcome Spaces.” Geoforum; Journal of Physical, Human, and Regional Geosciences 39 (2): 789–804,

Swinerd, Chris, and Ken R. McNaught. 2012. “Design Classes for Hybrid Simulations Involving Agent-Based and System Dynamics Models.” Simulation Modelling Practice and Theory 25 (0): 118–33.

Ligmann-Zielinska et al. (2020), Principles of Participatory Ensemble Modeling to Study Complex Socioecological Systems, Innovations in Collaborative Modeling, Edited by Laura Schmitt-Olabisi, Miles McNall, William Porter and Jinhua Zhao, Michigan State University Press,

Since I mentioned hybridization, I think that extending this topic would add value to the manuscript. Unfortunately, not much can be said about this topic based on two cited papers. Extending the search to papers older than a decade may provide more opportunity for this discussion.

[3] Handling uncertainty: one critical aspect not covered in the paper, which, in my opinion, is one of the most significant weaknesses of the manuscript, is handling uncertainty. There is no review of whether uncertainty and sensitivity analysis was done in the first place and, if so, to what extent and how. First, it is a critical aspect if we want to build stakeholder trust in modeling, and second, it is a requirement when we move to policy decision-making that directly impacts SESes. Without uncertainty exploration, modelers cannot really attribute differences in policy scenario outcomes to the scenario assumptions or the model mechanics. Without this knowledge, specific policy recommendations/interventions may be useless. Also, one of the significant challenges in applying SES models to real-world policy-oriented problems is communicating uncertainty – a topic closely related to Aspect 3.

[4] ‘Aspect 3 – stakeholder involvement’ misses a case where the stakeholders evaluate the products of models rather than the models themselves. In such cases, a user-friendly interface is unnecessary since the stakeholders deal with the products of modeling (reports, maps, diagrams, etc.), not the models themselves. Another stakeholder involvement aspect is role-playing games. Certain ABMs were developed strictly for this purpose, and some of them are case-specific (not just abstract simplified SES models) – check ‘companion modeling’ for example. Also, the authors should delve a little more into the topic of participatory modeling in SES – this is a critical aspect when the researchers want their models to be used in practice.  Topics to consider are participatory model building, mediated modeling etc. Two experts who have published a lot on this topic in SES are Moira Zellner (using mainly ABM) and Laura Schmitt Olabisi (SD), for example:

Zellner M (2008) Embracing Complexity and Uncertainty: The Potential of Agent-Based Modeling for Environmental Planning and Policy, Planning Theory & Practice Volume 9, Issue 4

Schmitt Olabisi et al. (2013) "Modeling as a Tool for Cross-Disciplinary Communication in Solving Environmental Problems." In Enhancing Communication & Collaboration in Interdisciplinary Research. M. O'Rourke, et al. (Eds.), 271-291.

Schmitt Olabisi, Laura et al. 2010. “Using Scenario Visioning and Participatory System Dynamics Modeling to Investigate the Future: Lessons from Minnesota 2050.” Sustainability: Science Practice and Policy 2 (8): 2686–2706.

Van den Belt, M (2004). Mediated Modeling: A System Dynamics Approach to Environmental Consensus Building. Washington, DC: Island Press.

[5] Sections 4.1. and 4.2. I would argue that the limitations listed explicitly for ABM and explicitly for SD are actually expected for both. For example, regardless of the modeling approach, the results of models should not be taken at face value when dealing with policymaking (l.333+). SD also becomes a simulation method when the model is run with uncertain inputs, so constraining this argument to just ABM seems superficial. Another example is related to hybrid models. The conclusion that SD-ABM hybridization is aimed at reducing computational can be disproven if the modeling team tightly couples the two models. Then, the coupling may significantly slow down the computation.

MINOR

[1] l. 47+ I’d argue that the ‘three-pillar model of sustainability’ also applies to SES and should be included in the cited frameworks

[2] l.90, I would avoid talking about emergence in the context of SD when also talking about ABM. One of the major characteristics of ABMs is emergence, while I have not heard about this concept in SD. I suggest changing to ‘explicit representation of system-level dynamics …’

[3] For the examples of applications cited (l. 84 and 94) please be specific that these are just a few of many examples of using ABM and/or SD in modeling SES.

[4] l 107+ this statement is incorrect. While I agree that it is not as common as we’d like, it’s not as uncommon as the authors suggest – this conclusion may be due to the methodology used to extract the pool of papers as discussed above.

[5] The conceptual framework used to define ‘Aspect 2’ (p. 5) omits essential elements of SES – businesses/industry and engineering systems. I’d argue that these are critical elements of SES, especially if we focus on the last decade. While these are not expected (perhaps even nonexistent) in the reported applied ABM/SD of SES, they should be present in a framework that tries to capture the essential components of SES. Then, the authors could elaborate upon why these elements are important in their discussion section.

[6] ‘Aspect 4 – practical application …’ p. 6, talks about decision-making. The authors should narrow down the term to policy decision-making. When talking about ABM, decision-making has many different meanings.

[7] Many of the observations related to the analysis (section 3) may be attributed to the nature of ABM versus SD (e.g., scale, heterogeneous decision-making, representation of feedback loops), which is not surprising. The authors should discuss this in detail in their discussion section.

[8] Figure 4 is a nice visual but needs clarification of abbreviations. It is cumbersome for the reader to browse through the paper for explanations of the abbreviations. The authors should either provide a separate table with the names behind the abbreviations or extend the caption with a legend listing the abbreviations and the terms behind them.

[9] l. 339+, there has been a long-term debate about whether ABMs are appropriate for prediction or whether models’ predictive ability is a proper objective when dealing with complex systems like SES. See Epstein below. Hence, I’d argue that this is not necessarily their limitation, as it may not really matter when researchers build and use empirical ABMs.

Epstein, J. M. (2008). "Why Model?" Journal of Artificial Societies and Social Simulation 11(4): 12: http://jasss.soc.surrey.ac.uk/11/4/12.html

[10] l.351 ‘SD can integrate disparate data types..’ and so can ABMs. This is not a unique property of SD.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

This manuscript reviews models applied to study complex social-ecological systems. Publications of papers focusing on Systems dynamics and Agent-Based models are reviewed.  This review presents several advantages of the two main methods reviewed. The study also presents improvements that, according to the authors, could be achieved to augment the utility of these modeling methods.

General comments

As the manuscript is Very well-organized and written, I have only minor suggestions for the authors. Improve the presentation of Table 1. I think it could be clearer.

Final remarks

This is a well-written manuscript. In the field of systems dynamics and simulation, this review will serve as a comprehensive reference. I have no reserve to recommend its publication in Systems.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

The authors adequately addressed the comments I posed in the previous version of the paper except for one. My only critique that I wholeheartedly disagree with is the following statement, which is counterfactual:

"We had excluded articles that used conceptual model (uncertain data input)."

It's quite the opposite. Conceptual models do not deal with uncertainty at all! They demonstrate components, relationships, events, and processes as concepts to represent some phenomenon or theory (as the name suggests). It is the empirical models (including models for policy analysis) that have uncertain inputs. A very simple example would be an agent-based model where farmer agents select portions of their land for conservation (fallow land). Then, the uncertain input would be the percentage of land excluded from cropping. This could simply be represented as a probability distribution (like a min-max range) where each agent could be assigned a different value. Obviously, the sampling would differ from run to run, making the model stochastic and therefore imbued with uncertainty. The authors seem to confuse the concept of model uncertainty with theoretical modeling. 

Now, I understand that this is outside the scope of the paper, and I appreciate that the authors discussed uncertainty in their limitations. My comment above is just for the authors' information and does not refer to the paper directly.

Consequently, I'm also still not sold on the statement, "With this context, SD has better predictive power than ABM." The authors' reasoning does not support this claim. Perhaps it is true, but it does not stem from the uncertainty argumentation provided in the response letter.