*3.5. Reference Values (Step 4)*

The classification of the metrics' result is made by associating each answer to a resilience development level, related to the reference values. The metric results are classified as (i) incipient (non-existent or at early stage of development); (ii) progressing (significant steps have already been taken); and (iii) advanced (consolidated results). The assessment of each metric is made according to reference values, defined from an extensive literature review on each metric. A resilience development level between 0 and 3 is then assigned, based on reference values, namely (i) incipient [0, 1], progressing [1, 2], and advanced [2, 3, 26].

Given the RAF structure (Figure 1), the results of the metrics contained in each criterion might be averaged to a criterion resilience development level, and further on, upwards to an objective and then to a dimension development level. Table 1 presents examples of three metrics, their reference values and the set of references used to support them.

**Table 1.** Examples of data, procedure, and model-based metrics and corresponding reference values proposed in the RAF for nature-based solutions (NBS).


\* DB: Data based, PB: Procedure based, MB: Model based.

#### **4. Conclusions**

This paper presents the methodology adopted for the construction of a RAF to assess NBS contribution to urban resilience and the developed structure of the RAF for NBS, focused on solutions for stormwater management and control. This resilience assessment proposal ensures the evaluation of the main *attributes* of the urban resilience and the relevant *aspects* for the NBS evaluation. The RAF aims to support NBS diagnosis and to assist decision-making in its planning, implementation, and management. Also, this framework allows to identify NBS with potential to contribute to city resilience.

The determination of the metrics relevance and feasibility of application to cities is a fundamental step in the validation step. Considering the metrics relevance, the stakeholders' opinion allowed to conclude that most metrics are considered essential for the assessment of the NBS contribution for urban resilience. Regarding the feasibility of application, a higher variability in the stakeholders' responses was obtained. As expected, procedure and model based metrics were labelled as medium or low feasibility of application, mainly at the service and infrastructure level.

In this sense, the criteria identified as with lower feasibility of application correspond to the following criteria: (i) flexible service, (ii) reliable service, and (iii) infrastructure preparedness for recovery and buildback. Regarding the flexible service criterion, the lower feasibility of application is due to the lack of awareness of ES and the difficulties related to the use of the NBS retained water for other purposes, at city level. From another hand, the lower feasibility of application of the reliable service and infrastructure preparedness for recovery and buildback criteria is because metric determination is carried out based on a mathematical model.

The stakeholders' participation highlighted the relevance of each metric and criteria defined in the RAF. Stakeholders' opinion allowed to identify the RAF adequacy to diverse city maturity levels and helps to select adequate metrics for the cities according to the urban resilience development level. Based on this analysis, the consolidation of metrics definition and required data will be carried out.

The RAF consolidation will be carried out after its complete application to the case study. In this step, the metrics' definition and the required information will be verified. In this sense, future work will focus on the consolidation of the RAF and on the proposal of a roadmap for the RAF application to any city.

The RAF application should follow the proposed roadmap and consider pre-selected metrics. The pre-selected metrics and the required information depend of the urban resilience development (incipient, progressing, advanced). For example, only for cites with advanced urban resilience development the model based metrics will be determined, and, consequently, the information provided by the mathematical model, which is more detailed and difficult to obtain, will be required.

The determination of the RAF will be carried out by a multidisciplinary team composed by human resources of the entities in charge of NBS management, stormwater management services and green space management. The RAF can be applied to assess the contribution to urban resilience of all existing NBS in the city, a group of NBS or a specific NBS. For this reason, the NBS under assessment should be identified in the complementary profile.

**Supplementary Materials:** The following are available online at http://www.mdpi.com/2071-1050/12/6/2537/s1. Table S1. Objective, criteria, and metric of Dimension I "Integration of NBS in the city".; Table S2. Objective, criteria, and metric of Dimension II "Operation and service of NBS".

**Author Contributions:** The conceptualization of the work and of the methodology were carried out by P.B. Original draft preparation was developed by P.B. and review and editing was accomplished by P.B., R.S.B., and A.G. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Portuguese Foundation for Science and Technology (FCT) through the PhD fellowship PD/BD/135216/2017.

**Acknowledgments:** The authors gratefully acknowledge the support of the Portuguese Foundation for Science and Technology (FCT), through the PhD fellowship PD/BD/135216/2017. The authors acknowledge the RESCCUE project for the opportunity to participate in this special issue. The authors would like to thank the following organizations for the collaboration and all constructive comments during this work and for the collaboration in the RAF validation step: Águas de Coimbra, Águas do Porto, Ajuntament de Barcelona, Bristol City Council, Câmara Municipal de Lisboa, Câmara Municipal do Porto, City of Vancouver, and SMAS de Almada.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.
