A Concept for Testing Decision Support Tools in Participatory Processes Applied to the ToSIA Tool

Abstract

ToSIA (Tool for Sustainability Impact Assessment) offers a transparent and consistent methodological framework to assess impacts of changes (technological, policy, management, etc.) in the forest-based sector. This tool is able to facilitate the decision making process within and between diverse groups of stakeholders (e.g., forest managers and policymakers) as it provides a neutral, transparent and data-driven platform for stakeholder interaction and communication. To test these capabilities of ToSIA, a practical approach to test if a decision support system is suitable for participatory processes was developed based on a set of evaluation criteria for participatory processes. ToSIA’s performance was assessed and discussed in different categories against a selection of criteria for successful participatory processes: six criteria were fulfilled by ToSIA, in nine, ToSIA is potentially helpful, in two, criteria ToSIA has no influence, and for three criteria, no experiences exist until now. As a result, ToSIA’s conceptual suitability as a participatory decision support system was confirmed for two interlinked roles: as a decision support system to assess alternative scenarios, and as a communication platform for stakeholder interaction.

1. Introduction

Today’s regional planning and development processes are expected to meet sustainability requirements, and to be based on sound scientifically approved results that are open to public scrutiny. This poses challenges for decision makers and calls for decision support tools which are suitable for application as part of participatory processes [1].

The concept of sustainability has evolved and grown during recent decades, in response to demands from the ongoing policy process and taking commitments made by EU member states into account. The European Commission decided on a Forestry Strategy for the European Union [2] to establish a framework for forest-related actions in support of sustainable forest management (SFM). The Strategy emphasises the importance of the multifunctional role of forests and SFM for the benefit of society. It also links to international commitments (e.g., UN Conference on Environment and Development in 1992 (UNCED) [3] and its follow-up conferences, and the Ministerial Conferences on the Protection of Forests in Europe (MCPFE), now Forest Europe [4], and stresses the importance of active participation in all forest-related international processes. The need for improved co-ordination, communication and co-operation in all policy areas of relevance to the forest sector is highlighted, and the current focus of sustainability discussions becomes clear: participation and representation of diverse stakeholder groups, and the possibility to assess multiple ecosystem services and related issues as part of integrated impact assessments.

Along with the evolution of the definition of sustainability, the need also increased for methods, tools and models, which are capable of calculating sustainability impacts [5]. In response, ex ante sustainability impact assessment (SIA) methods have been developed in Europe [6,7,8], which assess the impacts of changes such as policies, market, technology or industry developments in comparison to a baseline. ToSIA, the Tool for Sustainability Impact Assessment, was developed to assess the sustainability impacts of changes in forest wood chains (FWCs) [9].

While a range of methods and tools have been developed and used to assess sustainability in a general or in a specific setting [10], they are often not practical when engaging with stakeholders and practitioners from a variety of mixed professional backgrounds due to a variance in methodological assumptions and suitability, (non-)available user guidance and data availability. Decisions regarding land-use management are often heavily scrutinized by the public. As these decisions are made by responsible authorities and affect the environment and wider ecosystem services, they are under significant pressure from increasingly complex public demands [11,12,13]. Large-scale, sometimes global interactions between agencies and institutions at an international level influence decision-making even at a local level [14]. Borja et al. [15] stress the importance of integrative tools and approaches, which combine different levels of detail (e.g., from local to national scale) and to maintain the quality of the assessment. Integrated landscape management tools, which address this complexity, pose the risk of becoming too complicated for the end-user (be it an expert or a novice), and consequently the tools may never be used in practice. Giupponi [16] states that despite the many DSS developed in the field of environmental management, the risk of such systems failing to meet the challenge of real-world problems is reported to be high, as greater attention should be given to the needs of potential users and to the identification of the concrete application contexts. Even the criteria for judging whether a DSS has been successful or not, are often a matter for discussion [17,18,19].

There is a widely recognized need to develop new support tools for decision-making in this field, with greater attention to identify both the needs of potential users and the context of the application itself [16]. Menzel et al. [1] explore the combination of DSS and participatory approaches for democratic decision making and developed a list of features and criteria for testing the suitability of DSS for participatory approaches. Against these criteria and features they assessed the currently available DSS as listed on FORSYS (Forest Management Decision Support Systems (=FORSYS) was a Cost Action (FP0804) which gathered decision support tools for forestry in an online wiki-database [20]). One conclusion of Menzel et al.’s [1] study was that there is a serious lack of such tools, and a call to combine DSS tools with participatory process approaches. Further we will refer to decision support systems with participatory approaches as “participatory decision support systems” (pDSS).

The main focus of our study was stakeholder involvement for decision support concerning sustainable regional development issues. For this ToSIA was chosen as a DSS tool, as its application is neutral and flexible. A participatory stakeholder approach, which follows the requirements of the tool, was developed to serve two interlinked roles: as a decision support system to assess sustainability impacts of alternative scenarios, and as a communication platform for stakeholder interaction. The aim of this study was to test the suitability of the ToSIA tool as a pDSS against the features and criteria developed by Menzel et al. [1] to show if the criteria are suitable, how they can be applied in practice and thus also can be applied to other DSS. One further aim of developing and applying such a practical testing approach was also to demonstrate how ToSIA can be used and improved. This study also gives recommendations for executing such a participatory process with stakeholder involvement.

2. Material and Methods

As different methods are combined in this paper, first, a short description of how ToSIA works as a tool is given with references for further reading. Then Menzel et al.’s [1] features and criteria for pDSS are described, and based on these, a method to assess the suitability of pDSS is presented using the example of the ToSIA tool.

2.1. ToSIA: Tool for SIA—and for Participatory Decision Making?

ToSIA is a decision support tool for sustainability impact assessment. It compares quantified impact on sustainability between a baseline and scenario(s) [6,9]. These impacts are calculated from indicators and material flows of value chains or Forestry Wood Chains (FWCs), and are reflected as changed indicator values between alternatives [21]. ToSIA is a flexible and entirely data-driven tool, with the possibility to include additional indicators [22,23,24] as well as analysis tools such as Multi-Criteria-Analysis [25] and Cost-Benefit-Analysis [26]. ToSIA was applied in Northern European case study regions in North Scotland, North Sweden and East Finland as part of a project called Northern ToSIA in close collaboration with local stakeholders (details about these case studies can be found in [27,28,29,30,31]). For this purpose, the structure for undertaking SIA assessments with the ToSIA tool was used for interacting with users according to the following three-step participatory approach:

Step 1: Building a Case with Stakeholder Involvement

A case consists of process chain topologies, linked by products, which describe the activities (=processes) and input/output products to these production processes of clearly defined and graphically formalized process chains. These topologies are linked with material flows and indicator values per process for both a baseline and one or more scenarios. All this data is entered into a database via a server-based graphical user interface, and from there loaded into the actual ToSIA.

Step 2: Presentation and Explanation of Baseline and Scenario Results

In the ToSIA engine material flows, and indicator impacts are calculated for the baseline and scenarios. They can be compared and analysed per process, per chain or user-defined sub-segment.

Step 3: Comparison and Analysis Tools

In addition to the indicator-based comparison functionality, there exists also a range of analysis tools such as Multi-Criteria-Analysis, Cost-Benefit-Analysis and Policy Analysis. These tools are optional to use in addition to the unweighted indicator-based impact comparison. They help to translate values of different units into a non-dimensional preference (MCA), monetary values (CBA) or to refer to policy documents with potential limitations for selected indicators.

2.2. Evaluation Framework: Criteria and Method for Testing Suitability for pDSS

Menzel et al. [1] identify four features of DSSs that may support their application in participatory processes: (i) group decision support, (ii) possibilities to include other values than timber production, (iii) flexibility of system to include non-traditional forest data and management options, and (iv) multi-criteria decision analysis tools. These features are the basic requirements that Menzel et al. [1] found to be a pre-requisite for DSS to qualify as suitable for participatory processes. In addition to these four features, they also identified 20 evaluation criteria as relevant to successful participatory processes, out of which they identified seven criteria that had particular relevance for DSS. These seven were: Opportunity to influence outcome, quality and selection of information, challenging of status quo and fostering creative thinking, structured decision-making process, transparency, and independence and neutrality of process, search for common values. These criteria were shortlisted (see criteria in italics and marked with *) from a more comprehensive long-list of 20 criteria (Table 1).

Table 1. List of criteria related to successful participatory planning selected by Menzel et al. [1] and based on a review of the literature. Seven main criteria of relevance to DSS identified by Menzel et al. [1] are in italics.

Evaluation Criteria	Criteria Definition
Fairness	Access to the process and power to influence process and outcomes [32].
Relationships and social capital building	Referring to issues of social capital through new and existing social networks developed during the process/project, for example, trust, reciprocity and collaboration [33].
Structured group interaction	Relates to principles about the structural characteristics of the process; for example, planning of meeting—time, location, “physical arrangements”. Locus of control is with the planner of the process [34].
Facilitation of constructive individual/group behaviour	Relates to principles about personal behaviour of individuals taking part in the process; for example, ground rules [34].
Representation	Referring to the spread of representation from affected interests; including how legitimate the representation is seen to be; the diversity of views is important not just that representatives from different groups are invited [33].
* Opportunity to influence outcome	Referring to the participant’s opportunity to influence (enough time; involved early enough; access to policy makers and leaders; organisational structure) [33]. Giving people the opportunity to express their preferences and values (co-authors).
* Quality and selection of information	Referring to the adequacy, quality and quantity of information provided [33].
Cost-effectiveness	Referring to the improvements created through the process in relation to the costs accrued [33].
Accessibility of process	The issue of physically getting people present and involved in deliberative settings [34].
Adequate resources	Public participants have access to the appropriate resources to enable them to successfully fulfil their brief [35]. Sufficient time and supporting technical resources. Participatory processes takes time and organisers must ensure provision of sufficient technical resources to allow participants to formulate sound opinions based on timely and reliable information [36].
Opportunity to influence process design	The public is involved as early as possible in the process as soon as value judgments become salient [35]. The decision-making process is clearly structured, with inclusion of stakeholders in the process design and transparency on how final decisions will be reached [37].
* Challenging status quo and fostering creative thinking	Process encourages questioning the status quo and encourages the imagination of alternative futures [38].
* Structureddecision-making process	The participatory process uses appropriate mechanisms for structuring and displaying the decision making process [35] .
Clear mandate and goals	Expectations towards participants are clearly laid out at the beginning of any process [36]. The nature and scope of the participation task are clearly defined; scope, expected output and mechanisms for the procedure are defined [35].
* Transparency	Referring to both internal, whereby participants understand how decisions are made; and external, whereby observers can audit the process [33].
Acceptance of outcome	Social and political acceptability [39] Groups and individuals interested in or affected by public land decisions report that the resultant plan addresses their needs, concerns, and values, and they will not appeal it [40].
Accountability	Referring to whether the representative’s core constituencies are satisfied, including expectations [33].
* Independence and neutrality of process	The process is conducted in an independent, unbiased manner. Participants are free to conduct themselves in a voluntary and self-directed manner without coercion, and process management is neutral. The process seeks the common good, not just accommodating specific interests [37].
Legitimacy	Referring to whether the outcomes and process are accepted as authoritative and valid [33].
* Search for common values	A participatory decision-making process places strong emphasis on the value-based character of a policy dispute and the mechanisms by which it is managed [32].

Table 1. List of criteria related to successful participatory planning selected by Menzel et al. [1] and based on a review of the literature. Seven main criteria of relevance to DSS identified by Menzel et al. [1] are in italics.

Evaluation Criteria	Criteria Definition
Fairness	Access to the process and power to influence process and outcomes [32].
Relationships and social capital building	Referring to issues of social capital through new and existing social networks developed during the process/project, for example, trust, reciprocity and collaboration [33].
Structured group interaction	Relates to principles about the structural characteristics of the process; for example, planning of meeting—time, location, “physical arrangements”. Locus of control is with the planner of the process [34].
Facilitation of constructive individual/group behaviour	Relates to principles about personal behaviour of individuals taking part in the process; for example, ground rules [34].
Representation	Referring to the spread of representation from affected interests; including how legitimate the representation is seen to be; the diversity of views is important not just that representatives from different groups are invited [33].
* Opportunity to influence outcome	Referring to the participant’s opportunity to influence (enough time; involved early enough; access to policy makers and leaders; organisational structure) [33]. Giving people the opportunity to express their preferences and values (co-authors).
* Quality and selection of information	Referring to the adequacy, quality and quantity of information provided [33].
Cost-effectiveness	Referring to the improvements created through the process in relation to the costs accrued [33].
Accessibility of process	The issue of physically getting people present and involved in deliberative settings [34].
Adequate resources	Public participants have access to the appropriate resources to enable them to successfully fulfil their brief [35]. Sufficient time and supporting technical resources. Participatory processes takes time and organisers must ensure provision of sufficient technical resources to allow participants to formulate sound opinions based on timely and reliable information [36].
Opportunity to influence process design	The public is involved as early as possible in the process as soon as value judgments become salient [35]. The decision-making process is clearly structured, with inclusion of stakeholders in the process design and transparency on how final decisions will be reached [37].
* Challenging status quo and fostering creative thinking	Process encourages questioning the status quo and encourages the imagination of alternative futures [38].
* Structureddecision-making process	The participatory process uses appropriate mechanisms for structuring and displaying the decision making process [35] .
Clear mandate and goals	Expectations towards participants are clearly laid out at the beginning of any process [36]. The nature and scope of the participation task are clearly defined; scope, expected output and mechanisms for the procedure are defined [35].
* Transparency	Referring to both internal, whereby participants understand how decisions are made; and external, whereby observers can audit the process [33].
Acceptance of outcome	Social and political acceptability [39] Groups and individuals interested in or affected by public land decisions report that the resultant plan addresses their needs, concerns, and values, and they will not appeal it [40].
Accountability	Referring to whether the representative’s core constituencies are satisfied, including expectations [33].
* Independence and neutrality of process	The process is conducted in an independent, unbiased manner. Participants are free to conduct themselves in a voluntary and self-directed manner without coercion, and process management is neutral. The process seeks the common good, not just accommodating specific interests [37].
Legitimacy	Referring to whether the outcomes and process are accepted as authoritative and valid [33].
* Search for common values	A participatory decision-making process places strong emphasis on the value-based character of a policy dispute and the mechanisms by which it is managed [32].

To evaluate the suitability of a DSS for participatory decision making processes a checking procedure covering two stages a, and b (see Figure 1) was adopted.

Figure 1. Stages used for checking the suitability of a tool, in this case ToSIA as a pDSS, building on Menzel et al. [1].

Figure 1. Stages used for checking the suitability of a tool, in this case ToSIA as a pDSS, building on Menzel et al. [1].

Table 2. Categories and details for categorisation of tool performance against pDSS evaluation criteria.

Categorisation of Tool Performance	Details on How Categorisation was Done
Tool entirely fulfils criteria (3):	Criteria marked (3) were judged to have been fulfilled already through the design of the tool and nature of the approach. Fulfilment of criteria is entirely dependent on tool characteristics, no influence of a facilitator.
Facilitation-dependent. Tool potentially helpful to fulfil criteria (2)	Criteria marked (2) are able to be fulfilled, although successful implementation depends strongly on how the stakeholders are guided through the process (dependent of facilitation). For the fulfilment of this criteria, the tool is potentially helpful.
Criteria does not influence tool (1):	Criteria marked with (1) depend on a wider range of factors than those directly related to the tool or the approach (strongly dependent on organisation of participatory interaction or facilitation). For the fulfilment of this criteria tool cannot help.
No experience available yet (0):	A question mark (0) indicates that sufficient experiences are still missing, so that no categorization is currently possible.

Table 2. Categories and details for categorisation of tool performance against pDSS evaluation criteria.

Categorisation of Tool Performance	Details on How Categorisation was Done
Tool entirely fulfils criteria (3):	Criteria marked (3) were judged to have been fulfilled already through the design of the tool and nature of the approach. Fulfilment of criteria is entirely dependent on tool characteristics, no influence of a facilitator.
Facilitation-dependent. Tool potentially helpful to fulfil criteria (2)	Criteria marked (2) are able to be fulfilled, although successful implementation depends strongly on how the stakeholders are guided through the process (dependent of facilitation). For the fulfilment of this criteria, the tool is potentially helpful.
Criteria does not influence tool (1):	Criteria marked with (1) depend on a wider range of factors than those directly related to the tool or the approach (strongly dependent on organisation of participatory interaction or facilitation). For the fulfilment of this criteria tool cannot help.
No experience available yet (0):	A question mark (0) indicates that sufficient experiences are still missing, so that no categorization is currently possible.

3. Results

The results were obtained by applying the pDSS assessment method described above using the pDSS evaluation features and criteria to evaluate the suitability of the ToSIA tool, for decision making with active stakeholder involvement in a participatory process, as described in Table 3.

3.1. Stage A: Evaluating ToSIA against the Four Features

In the following the four features are used for an initial evaluation of the suitability of ToSIA as pDSS, with an explanation in which ways ToSIA addresses each feature.

The Possibility of Including Values Other than Timber Production

Material flows along the value chains are calculated in ToSIA in tonnes of organic Carbon. By means of conversion factors, these material flows can also be expressed per product in Euros, m3, tonnes of material, etc. Even though the main focus of ToSIA applications has been on forest-based Carbon, this is no limitation. Other value chain calculations are also possible, for example reindeer-based organic Carbon [31] and fossil Carbon chains [28].

Sustainability impacts are assessed using indicator values for individual processes along the process chain. These processes can be related to timber production, as well as to any other process along the (forest) value chains. In terms of indicator values there are no limitations as long as they can be quantified and related to process units. Indicators can be selected for other values than timber production such as biodiversity, recreation, employment, emission, hazardous waste, husbandry herd balance, foraging resources, etc., [23,24,41]. A limiting factor however is data availability and data quality: ToSIA results are only as good as its input data for indicator value and material flow calculation. These two factors restrict the applicability and the quality of the calculated results. Data collection is a time-demanding job, and needs special attention at using statistics, models, surveys and expert opinions that use different assumptions for externally generating input data to ToSIA. For participating stakeholders the origin of the data and transparency of calculation proved to be very important, so that they could/would trust in the calculated results.

The Ability of the System to be Sufficiently Flexible to Include Non-Traditional Forest Data and Management Options, for Example, the Possibility of Including Uneven-Aged Forests

ToSIA uses process chains, consisting of processes, products and links, i.e., chain topologies. The scope, content and level of detail is determined by the user. There is no limitation to the inclusion of even-aged forest management processes but any type of forest management is possible, just as any process within the forest-based sector (FBS) or other sectors can be assessed [6,22]. However, ToSIA is not a forest growth or forest management simulator. Each process and product has to be defined by the user, so ToSIA only knows as much as the user who provides the data. Information about growth dynamics of uneven-aged and mixed forests would need to be derived from inventory data or suitable growth simulators. At an aggregate process level for snap-shots in time, ToSIA has been tried and works well, and this use works also well in participatory approaches as stakeholders are not overloaded by too detailed information about growth processes and potential management options.

Tools for Multi-Criteria Decision Analysis

In addition to its comparison and reporting function, ToSIA offers evaluation tools that are built into ToSIA, in particular Multi-Criteria-Analysis [25], Cost-Benefit-Analysis [26] and Policy Analysis [42,43] However, these tools need to be used with care, as many include subjective preferences (like stakeholder preferences in MCA) or assumption for hypothetical monetary values of non-monetary aspects (like biodiversity in CBA). These subjective values get recalculated over the SIA results, and thus for stakeholders it was difficult to see how the final preference results were influenced by the actual case study results, subjective preferences, or limitations imposed by specific data, the case study or scenario options.

Group Decision Support

The tool can be used for group decision support. It requires a facilitator who applies it in a way that follows principles of effective stakeholder involvement. Stakeholders can incorporate their individual preferences for making assessments [25,44].

It has to be stated, however, that ToSIA as such does not give any “recommendations” regarding which alternative to choose, it only provides quantitative assessments of the impacts of alternative scenarios compared to a baseline. The ultimate choice relies on the user of the tool, regardless if it is a group or an individual.

One of the core ideas behind the ToSIA tool was to make its application as flexible as possible and to keep limitations imposed by the tool to a minimum. This was achieved by making it to a high extent data-driven. All information connected to the topic, the content of the chain, the indicators and material flows, comes from the data that is loaded into ToSIA, which again is determined by the user. This makes it very flexible and transparent, but also very time and data-intensive which slows down the application of ToSIA for SIA assessments.

3.2. Stage B: Detailed Assessment of Using ToSIA in a 3-Step Participatory Approach against the pDSS Suitability Criteria as Evaluation Categories 0 to 3

Three steps of the ToSIA approach are assessed in turn to check if and how each pDSS evaluation criteria fits into the categories 0 to 3. The reason why this check is done separately for each step of using ToSIA is because each step addresses different criteria. In some cases there are overlaps, where one pDSS criteria is covered by two or even all three steps (see Table 3 for details).

Table 3. List of pDSS evaluation criteria categorised according to level of influence a criteria has on participatory processes, and which are covered by ToSIA at which stage. Criteria in italics with * were chosen by Menzel et al. [1] as mainly relevant, which are in partial contrast to pDSS category 3 (Tool entirely fulfils criteria) as chosen by the authors. Definitions of criteria can be found in Table 1.

Evaluation Category	Coverage by ToSIA	Step
3—Tool entirely fulfils criteria
* Independence and neutrality of process	yes (Tool structure elements: process chains, indicators)	1	2	3
Clear mandate and goals	partially (depending on facilitation)	1		3
* Structured decision-making process	yes (3 steps tool structure)	1	2	3
*Challenging status quo and fostering creative thinking	yes (scenario assessment embedded in tool structure)	1	2
Opportunity to influence process design	partially (data-driven approach)	1	2	3
3—Tool entirely fulfils criteria
* Quality and selection of information	yes (metadata available and visible)	1	2
* Transparency	yes (whole tool setup is based on transparency and userfriendlyness)	1	2	3
Acceptance of outcome	missing info(has not yet been tested for ToSIA)			3
2—Facilitation-dependent. Tool potentially helpful to fulfil criteria
Fairness	Partially (depending on facilitation)	1	2	3
Accessibility of process	partially (depending on facilitation)	1	2	3
* Opportunity to influence outcome	partially (depending on facilitation)	1	2	3
* Search for common values	partially (depending on evaluation/assessment)	1	2	3
Relationships and social capital building	Partially (depending on participants)	1	2	3
Cost-effectiveness	missing info (not assessed with ToSIA so far)	1	2
Legitimacy	missing info(sufficient range of experiences is missing for ToSIA; see also “Acceptance of outcome”)		2	3
1—Criteria does not influence tool:
Representation	partially (depending on facilitation)	1	2	3
Accountability	yes (for data management: structured approach, DCP, clear framework conditions)	1	2	3
Adequate resources	no (depending on many factors; esp frame conditions)			3
Structured group interaction	No (outside ToSIA’s influence)	1	2	3
Facilitation of constructive individual/group behaviour	partially (depending on facilitation)	1	2	3
0—No experience available yet:

Table 3. List of pDSS evaluation criteria categorised according to level of influence a criteria has on participatory processes, and which are covered by ToSIA at which stage. Criteria in italics with * were chosen by Menzel et al. [1] as mainly relevant, which are in partial contrast to pDSS category 3 (Tool entirely fulfils criteria) as chosen by the authors. Definitions of criteria can be found in Table 1.

Evaluation Category	Coverage by ToSIA	Step
3—Tool entirely fulfils criteria
* Independence and neutrality of process	yes (Tool structure elements: process chains, indicators)	1	2	3
Clear mandate and goals	partially (depending on facilitation)	1		3
* Structured decision-making process	yes (3 steps tool structure)	1	2	3
*Challenging status quo and fostering creative thinking	yes (scenario assessment embedded in tool structure)	1	2
Opportunity to influence process design	partially (data-driven approach)	1	2	3
3—Tool entirely fulfils criteria
* Quality and selection of information	yes (metadata available and visible)	1	2
* Transparency	yes (whole tool setup is based on transparency and userfriendlyness)	1	2	3
Acceptance of outcome	missing info(has not yet been tested for ToSIA)			3
2—Facilitation-dependent. Tool potentially helpful to fulfil criteria
Fairness	Partially (depending on facilitation)	1	2	3
Accessibility of process	partially (depending on facilitation)	1	2	3
* Opportunity to influence outcome	partially (depending on facilitation)	1	2	3
* Search for common values	partially (depending on evaluation/assessment)	1	2	3
Relationships and social capital building	Partially (depending on participants)	1	2	3
Cost-effectiveness	missing info (not assessed with ToSIA so far)	1	2
Legitimacy	missing info(sufficient range of experiences is missing for ToSIA; see also “Acceptance of outcome”)		2	3
1—Criteria does not influence tool:
Representation	partially (depending on facilitation)	1	2	3
Accountability	yes (for data management: structured approach, DCP, clear framework conditions)	1	2	3
Adequate resources	no (depending on many factors; esp frame conditions)			3
Structured group interaction	No (outside ToSIA’s influence)	1	2	3
Facilitation of constructive individual/group behaviour	partially (depending on facilitation)	1	2	3
0—No experience available yet:

In general it was found that ToSIA makes facilitation easier because of its clear structure which offers predefined, logical steps for active participation. This is in response to category relevance (compare category 3 and 2). The converse argument however does not hold. For some pDSS criteria a distinction is made about how the ToSIA tool (as such) supports a criteria, and how the facilitation process in terms of presenting the tool functionality influences the successful application of the tool.

3.2.1. Tool entirely fulfils criteria (3)

The basic principles of the ToSIA tool were judged to meet the following criteria, which correspond to ToSIA elements as detailed below and are independent of facilitation:

* Independence and Neutrality of Process

The structure in which the data is collected and entered in terms of tool-functionality is neutral and independent. Cases are described in terms of user-chosen processes and products, and the data consists of material flows and relative indicator values. Quantitative and qualitative indicators cover economic, environmental, social and cultural aspects; new indicators can be defined according to a formalised approach, if needed to present aspects which are of importance to the different stakeholders. There are no restrictions or forced limitations by the tool itself. However, which data is set up by the user and what his/her motives for selection are cannot be influenced by the tool.

* Structured Decision-Making Process

The process of using ToSIA gives a clear structure following the actions described below from a to e, which demands decision making and facilitates communication if these decisions are taking jointly. The structure is:

(a)

Problem definition: hereby the focus of the case is decided, as well as the system boundaries. In the Northern ToSIA project cases to which the case publications refer, this decision was entirely made by the involved stakeholders and thus produced diverse cases which were entirely focused on research questions and cases of stakeholder interest [27,29].

(b)

Collecting information for material flows and indicator calculation for baseline and scenarios. The raw data was provided by the participants on request from the researchers and then calculated to suit the format which ToSIA requires [30]. This process takes time, and required iterative communication with the participating stakeholders; however it also helped to make the decision process on what to include or to exclude transparent and created trust into the data. Also first tentative scenarios were developed.

(c)

Calculating and analysing scenario impacts. Based on collected data the impacts of baseline and first scenarios were shown. In this process the development of additional and more refined scenarios came naturally [30], but also critique at scenario options were voiced [28,29].

(d)

Evaluating results via stakeholder preferences for an alternative, using Multi-Criteria-Analysis preferences. This option was not carried out in all cases, only in the Finnish case. Here the experience gathered from the Finnish case highlighted the importance of a continuous participatory stakeholder engagement. If for example the scenario selection is made too narrow, there is a risk that results may not find the approval of certain stakeholder groups. This is particularly the case if it is not clear to all stakeholders how tool, data, stakeholder preferences or scenario options influence recommendation of certain scenarios. The main problem was a change of the involved stakeholder group and too limited explanation of how decisions in previous activities a to c were made during the facilitation of the workshop.

(e)

In cases where the participating stakeholder remained always the same persons throughout the process, however, the provided sustainability impact assessment information on case alternatives (=baseline and possible scenarios) led to informed decisions on resource use, and considerations on how to react to policies [31,45].

In addition to the value-by-value comparison of indicator results between baseline and alternative scenarios for each process, ToSIA offers an impact indicator value comparison for different aggregations: between sub-sections of the process chain, for specific sustainability dimensions or only for user-selected indicators. This comparison is purely indicator value based and shows the quantified impacts calculated alternative options (=scenarios) have. It does not give any recommendations or preferences (unlike analysis tools, which can be used on top of these unadulterated indicator value results), but offers an objective and quantified basis for discussion and decision making.

To aid decision-making by giving the possibility to incorporate different stakeholder preferences, Multi-Criteria-Analysis, Cost-Benefit-Analysis and Policy Analysis, are analysis tools which can be used in addition to the ToSIA indicator value comparison. A tool interface exists.

* Challenging Status Quo and Fostering Creative Thinking

A baseline (e.g., status quo) is needed to provide a reference against which scenarios results can be compared. This basic tool requirement automatically challenges thinking about the current status quo and “what would happen if”-consequences. Scenarios can be defined in several ways, although the most common characteristics of potential changes were found to be:

▪

Material volume changes: what happens if the raw material availability increases (e.g., new forms of energy from short-rotation plantations, storm events producing large volumes in short time periods) or decreases (e.g., change in market conditions, new trade regulations and policies).

▪

Indicator values change: what happens if, for example, labour or fuel prices increase, or efficiency in fuel consumption increases and emission from production decreases.

▪

Topology changes: assessment of what happens if new processes (e.g., new bio-refinery technology) are included into chain.

* Quality and Selection of Information

At all stages of using the tool adequate information is available for the user as metadata, help and info files. Metadata like data source, data quality, assumptions are displayed on mouse-over at every value, however only if this information has been provided beforehand. The calculation of data in ToSIA is based on mathematical formulas which ensure high quality of data, such as:

Material flows: All processes are linked by products, and the material flow of these products is calculated in tonnes of Carbon on a strict mass-balance of inputs and outputs. No material gets lost or increased unaccounted for.

Indicator calculation: for each process relative indicator values are provided. Material flows are multiplied by the relative indicator values to reflect both aspects in the overall indicator value.

However, as ToSIA is entirely data-driven, and dependant on the data which the user provides, the quality and selection of the information is entirely dependent on the quality of the input data and data availability. In some cases, especially for cultural indicators, no data may exist and thus need to rely on expert estimates. The source and quality of each data item, however, has got a place for documentation of this metadata in ToSIA.

Clear Mandate and Goals

ToSIA tool provides a clear framework and platform for sharing information about goals. Through its data-driven baseline-scenario assessment approach it mandates the user automatically to define clear, quantifiable and well described goals. In all cases (Scotland [27,45], Sweden [30,31], Finland [28]) the contents of the case studies were worked out by the stakeholders jointly in guided discussions. The results and sustainability impacts of the baseline and scenarios of the case studies were discussed jointly with the stakeholders.

Opportunity to Influence the Process Design

The ToSIA tool and concept strongly advocate through its data-driven focus the opportunity to influence the process design, by defining (non-pre-filled) data elements which are crucial for calculation. However neither tool nor concept can guarantee these; the execution depends on the approach taken by the facilitators. In our study a main goal was to have local and regional decision makers and stakeholders included in the process of identifying relevant issues, definition of the case and the selection of indicators, as well as provide the data, select relevant scenarios and to discuss the effects of their decision options with their peers and other stakeholders for the same forest resource.

* Transparency

Transparency was a main goal for the design of the ToSIA tool, and the reason for focusing on a data-driven approach. The included metadata (on data sources, data quality, assumptions and definitions) enhances the transparency of the tools calculations to stakeholders. Clear references to metadata and assumptions are available to the user on mouse-over on indicator values, indicator and process definitions, as well as in info boxes for each step of the tool interface for easy checking at every point of the assessment.

Acceptance of Outcome

This criterion was so far not sufficiently assessed with ToSIA, as it requires a follow-up in decision making at further levels of discussion after the assessments are made. This will need to be further assessed in future. Directly after the interactions, the reactions of the involved stakeholders were in general very positive. In Scotland the tool was suggested to be used also for the planning of the Cairngorm’s National Park Plan for 2013. In Sweden the involved Sami village were positive about the potential usefulness of the tool for decision support, but also asked for further long-time studies before any definite decision can be made on the general possibility to accept outcomes from ToSIA (unpublished; project report).

3.2.2. Facilitation-dependent. Tool potentially helpful to fulfil criteria (2)

Criteria marked (2) are not under the direct influence of ToSIA tool or approach, but dependant on how the tool is used by a facilitator in interaction with participants. However, ToSIA can be potentially helpful for facilitation in these cases, as it gives a clear structure and transparent information at all steps.

Fairness

ToSIA’s use can promote fairness, since all data is displayed transparently and since any user’s voice can be included without any weighting. This attribute however relies on how the user of the tool designs his/her participatory process, and can potentially also open the door for lopsided results, if the group of participating stakeholders is having a strong representation of individual lobby groups and if this fact is not taken into consideration by the facilitator. In the studied cases, the results for the baseline were calculated based on data provided by stakeholders and for scenarios which were also chosen by the same group of participants in Scotland and Sweden. However in Finland different groups were involved, and consequently there very some critiques by participants on how the scenarios had been chosen.

Accessibility of Process, * Opportunity to Influence Outcome, Search for Common Values

The ToSIA tool and concept strongly advocate through its data-driven focus the accessibility of process, opportunity to influence the process design and outcome and to promote the search for common values. However neither tool nor concept can guarantee these; the execution depends on the approach taken by the facilitators, but ToSIA provides a platform that allows these criteria to be fulfilled better. In our study a main goal was to have local and regional decision makers and stakeholders included in the process of identifying relevant issues, definition of the case and the selection of indicators, as well as provide the data, select relevant scenarios and to discuss the effects of their decision options with their peers and other stakeholders for the same forest resource. Even though this makes the work very data and time intensive, this approach can be recommended if the goal is not to get quick results, but to aid understanding of the challenges of the involved stakeholder groups for each other problems, to stimulate constructive discussions and to promote a search for commonly acceptable solutions.

Relationships and Social Capital Building

This criterion links strongly to the before mentioned Search for common values. The ToSIA tool provides a clear framework and platform for communication through its baseline-scenario assessment approach. The fulfilment of these criteria however relies strongly on the participants and partially also on the facilitators. In our study the contents of the case studies were worked out by the stakeholders jointly in guided discussions. The results and sustainability impacts of the baseline and scenarios of the case studies were discussed jointly with the stakeholders.

Cost-Effectiveness

Cost-effectiveness strongly depends on the content of the case, and how the costs of the process are calculated against the “savings” which occurred from avoiding disadvantageous decisions. No such calculation was done with ToSIA so far.

Legitimacy

Similarly, as with the criterion Acceptance of outcome, the long-term perception of the legitimacy of ToSIA results has not yet been assessed as it is a rather new tool. More details under Section 3.2.4 “No experience available yet”.

3.2.3. Criteria does not influence tool (1)

Criteria in this category are such on which a tool cannot have any influence at all, as they are not connected, but dependent on organisation and facilitation of participatory processes. For completeness sake, however they are also evaluated here on why these criteria cannot be influenced by a tool, or in this case by ToSIA.

Representation and Accountability:

Both criteria depend strongly on the planning and the conduct of the facilitation process. In our study local and regional decision makers and stakeholders were approached and included in the process of identifying relevant issues which are of interest for them, in order to get good representation of stakeholders. Accountability or adherence to democratic decision making was assumed to be a basic condition and not checked more thoroughly. In terms of data management this criteria is adhered to.

Adequate Resources

In our work, researchers were the experts whose function was to provide access to the appropriate resources, both time- and knowledge-wise, by communicating the required data format, and facilitate the process of comparing results from defined baseline scenarios to a larger stakeholder group. The ToSIA setup was potentially helpful for facilitation, but it cannot guarantee a successful adherence to these criteria as they are outside the reach of a tool (such as sufficient time for coffee breaks), (Step 3 only).

Structured Group Interaction, Facilitation of Constructive Individual/Group Behaviour

ToSIA tool cannot influence human interaction or behaviours, this is entirely facilitation dependant. In the case studies researchers acted as facilitators whose function was to define the elements needed to build a case, to help calculate data in the required format, to help in the process of comparing results and defining scenarios which follow the ToSIA approach. The ToSIA setup was potentially helpful for giving a framework for facilitation, but it cannot guarantee a successful adherence to these criteria as they are outside the reach of a tool.

3.2.4. No experience available yet (0)

For the following criteria there is not yet sufficient information available yet for ToSIA. This may be different for other tools tested by the same approach.

Legitimacy, Acceptance of the Outcome

Legitimacy of ToSIA outputs (Steps 2–3) is one of the least evaluated criteria in the application of ToSIA to date, so that experience is missing to make an informed judgement. Also Acceptance of the outcome (Step 3) cannot yet be rated, as sufficient experience from broader policy processes is missing. Case study results, which were presented, were legitimate and accepted as they were chosen mainly by the stakeholders involved in each case. These stakeholders had built up the cases and provided the initial data, which led to a very high acceptance of the results, even if they were sometimes surprising.

* Cost-Effectiveness

As mentioned before cost-effectiveness is a strongly purpose-dependent criteria. The purpose can be avoided cost of disadvantageous decisions or costs for public relations and good governance related activities of involving and informing relevant stakeholder. For neither purpose calculations were done with ToSIA so far.

4. Discussion

4.1. How Can ToSIA Support Stakeholders in Decision Making?

The results of this article challenge many experiences which suggest that scientific tools and stakeholder are not easily combined [33,46]. The fact that ToSIA as a tool responds to all four features and is fully (6 criteria) or potentially (9 criteria) helpful for fulfilling 15 out of 20 criteria selected by Menzel et al. [1] already sets it apart from the majority of DSS tools listed on FoRSYS webpage [20]. ToSIA was expected to support decision making processes, where different aspects of sustainability and alternatives in decision making need to be taken into account. Because this is one of the main ideas behind ToSIA: to quantify impacts of a scenario compared to a baseline, in a transparent and objective way. Experiences from tool applications, however, showed that not only the tool but also the very process of applying the ToSIA approach in the process of decision making, is valuable for structured and transparent decision making and communication with different interest groups. This was when the idea of assessing ToSIA’s suitability as a DSS tool for participatory processes (pDSS) was born.

Stakeholder involvement opened up a range of new experiences on participation and ToSIA use. Surprisingly, data procurement was a very successful element of the participatory process. Relevant data is sometimes hard to find. The experience from the project was that the data collection must be based on the stakeholders’ own activities and realities, and that it is delivered or aided by the local stakeholder (=Participation). Similar experiences are also made in other fields like in developing country contexts, where participatory mapping [47,48] is a useful tool for stakeholder involvement, following similar concepts as we did. When stakeholders took responsibility for their data procurement and quality, the acceptance of the results was often greater as they “owned” the data, and hence also, to some extent, the outputs. Such collaboration often led to discussions about potential improvements to modelling and engagement processes, and helped lift cultural barriers restricting the use of models and tools.

Another key participatory process is scenario building. Participatory approaches in scenario building create alternative options which are of very high interest and considered “legitimate” by the involved stakeholder groups, may however have certain limitations in covering all aspects and interrelation ships of the system, or by being accepted by wider audience [49]. In one case, ToSIA was applied to the ongoing regional policy processes and this limited the scope of the case study and its findings [28]. The scenarios were the same as those used in previous Forestry Programmes. Stakeholders who participated in a Multi-Criteria-Analysis workshop to choose between pre-calculated scenarios for the use of regional forests were unsatisfied because of the lack of real alternatives, e.g., a significant increase in conservation areas. They expressed the wish to have been involved earlier in the project to widen the range of scenarios that were analyzed than those used in the ongoing policy process. This stressed the importance of the participatory processes criteria representation of all involved stakeholder groups throughout the whole process, so that they have a chance to influence the process design and outcomes.

In addition to choosing between alternative scenarios, ToSIA also proved to be very useful in decision defence, when decision makers and stakeholders needed to explain to other users or the public why certain decisions have been taken (=accountability and transparency). With ToSIA, evidence can be provided as a quantified backup for debate, and also the quantified impacts of alternative options can be opened to public scrutiny. This application relates again back to the quality and origin of data. If the stakeholders know the origin of their data, they may trust it more and use the results more convincingly and securely in public debate.

As a third field of potential application, ToSIA can also be used to assess decisions, which are already made. By means of a decision impact assessment, ToSIA can provide evidence either to support or challenge past decisions.

4.2. Further Ideas for Development and Application

Nevertheless, when it comes to participators decision making, it became clear that the processes and work needed to apply a tool, such as ToSIA, is at least as valuable as the outputs generated by the tool itself, if stakeholders are involved in all steps [46]. A major responsibility is carried by the facilitator or expert guiding stakeholders through the process. To a large extent, they determine success or failure, and cannot easily be replaced unless users invest the time and effort to learn how to use the tool itself. In comparison to other DSS in the forestry sector, as scrutinized by Menzel et al. [1], the possibility that the tool can support group interaction was particularly welcome and also a point where many other DSS were judged less favourable. As for the other main characteristics, according to Menzel et al. [1], only three tools included non-traditional timber-production related variables in a participatory approach. HEUREKA [50] has been developed in such a way. Within ToSIA this is a strength which is covered through its flexible concept and approach. Only few systems, e.g., SIMO, could be developed by the users, while in ToSIA the whole concept is data-driven and as such the content of each assessment is solely developed by the user. Multi-Criteria-Decision-Analysis cover a range of powerful tools, which need careful introduction and guidance when used in stakeholder interaction [51]. Only few DSS included MCDA in an integrated pDSS setting and use it in an enabling participatory setting, for example Mesta (details for all mentioned DSS can be found under [20]) [52]. Responding to this feature, ToSIA can include any indicator units and has a choice of analysis tools including Multi-Criteria-Analysis to supplement the analysis of impacts on individual indicator values. At the same time we need to highlight that the distinction between comparison of calculated indicator values and weighted indicator values (as done in MCDA) according to stakeholder preferences needs to be made clear very explicitly. This experience for participatory MCDA is shared by Mendoza and Martins [51].

In general, as the ToSIA methodology needs explanation and guidance for setting up cases, depending on the familiarity of users with formalised systems and process-driven approaches, and their (mis)trust of scientific models and procedures, the speed of implementing cases was dependant on the acceptance of the process by the users (stakeholders) and the amount of time and effort they invested into it.

5. Conclusions

The most outstanding conclusion is the importance of involving stakeholders throughout the entire process of applying a tool for sustainability impact assessment such as ToSIA, not just to select scenarios and indicators at the start and to be informed of the results at the end. There are multiple opportunities to engage stakeholders in refinement of the rules used to assess the suitability of individual sites for conversion in land use or forest management through workshops with stakeholders. These can help to increase the quality and legitimacy of an organisation’s planning and policymaking procedures. The study also demonstrated the value in linking with a “real” planning or policymaking process. Models and policies/plans can then evolve together, and all stakeholders, including researchers and policymakers, can more easily focus on tangible useful outputs rather than on a hypothetical exercise. Data, assumptions and outputs from models can be discussed as part of the stakeholder consultations for that policy or plan.

In this aspect the authors want to stress the importance and purpose of decision support tools as instruments which facilitate and aid participatory processes, and which do not just focus on producing outputs. There is a danger that these outputs will not or cannot get digested and accepted by the public as the way they were made was not transparent or based on democratic principles. Decision support systems can only function satisfactorily if they do not remain expert tools, but tools that can involve affected stakeholders in the decision process. For this reason the authors introduced a practical categorisation and application of a system for testing tools on their participatory decision support functions, which goes beyond Menzel et al.’s [1] criteria listing and selection.