Real-Time Evaluation of City–University Partnerships for Sustainability and Resilience

Abstract

Cities face many challenges in their efforts to create more sustainable and resilient urban environments for their residents. Among these challenges is the structure of city administrations themselves. Partnerships between cities and universities are one way that cities can address some of the internal structural barriers to transformation. However, city–university partnerships do not necessarily generate transformative outcomes, and relationships between cities and universities are complicated by history, politics, and the structures the partnerships are attempting to overcome. In this paper, focus groups and trial evaluations from five city–university partnerships in three countries are used to develop a formative evaluation tool for city–university partnerships working on challenges of urban sustainability and resilience. The result is an evaluative tool that can be used in real-time by city–university partnerships in various stages of maturity to inform and improve collaborative efforts. The paper concludes with recommendations for creating partnerships between cities and universities capable of contributing to long-term sustainability transformations in cities.

1. Introduction

The future of global sustainability and the future of cities are tightly connected. Cities are home to more than half of the world’s population and must play a critical role in mitigating climate change and adapting to its impacts to allow residents to thrive. In fact, one of the 17 Sustainable Development Goals (SDGs) specifically mentions the role of cities and urban areas, and the need for urban sustainability transformation, and several others focus heavily on cities [1]. Cities currently emit over 70% of all global carbon dioxide emissions [2]. Therefore, establishing and maintaining tight urban carbon budgets is key to meeting emissions and warming goals set out by the Paris agreement, the International Panel and Climate Change (IPCC), and UN Sustainable Development Goals [3]. Cities are increasingly feeling the effects of extreme weather and are particularly vulnerable because of their frequent proximity to coasts, floodplains, and dry areas. For instance, extreme wildfires have become a global phenomenon and cities from California to Australia are facing compounding struggles from the fires that seem to worsen every year [3,4,5,6]. In the 2019–2020 fire season, megafires burned across Australia, scorching over 25 million acres of land, killing roughly a billion animals, and destroying nearly 2000 homes. In California’s 2018 fire season, there were over 58,000 wildfires, with the Paradise fire incinerating an entire town and killing 85 people. In 2019, utility companies throughout California chose to preemptively shut off electricity for over 500,000 residents for fear of similarly devastating fires. This urgency is echoed in calls to focus sustainability research and practice on the sustainability transformation of cities and regions [7].

Sustainability problems such as climate change are complex and require innovative systemic solutions that span disciplines and institutions and are often slow to manifest [8]. These complex problems require transformation, or “radical, systemic shifts in values and beliefs, patterns of social behavior, and multilevel governance and management regimes” [9]. Municipal governments are attempting to mitigate and prepare for complex climate and energy challenges by creating sustainability and resilience agendas, which typically take the form of planning documents, civic mandates, and associated policy and programmatic actions [10]. Local governments, including municipalities, counties and regions, are primarily responsible for addressing climate change impacts, decarbonizing transit systems, transitioning to renewable energy, ensuring food access, and building more resilient and sustainable communities. However, they are often limited by institutional design, organizational logic, limited cross-jurisdictional coordination, and a general lack of skill and capacity for dealing with the uncertain and fast-changing nature of sustainability and resilience challenges [10,11]. Municipal plans and policy initiatives necessitate and often explicitly call for cross-sectoral and inter-institutional partnerships and collaborations (i.e., between cities, businesses, universities, NGOs, and community organizations) that can help dismantle institutional barriers and path-dependencies so that more innovative and holistic solutions can be achieved [12]. For instance, in an analysis of the Rockefeller Foundation’s 100 Resilient Cities strategy documents, partnerships and collaborations were the most commonly cited planning, development, and implementation strategy across US cities [13]. Additionally, partnerships and collaborations with other institutions, like universities, have become increasingly important because they can help cities and other municipal governments address complex challenges, develop innovative solutions by operating across departments and jurisdictions and build capacity for sustainability problem solving.

Sustainability science and related fields (e.g., climate science, environmental science) continue to call for greater transdisciplinarity and applied research to increase the rate and real-world impact of discovery for urban sustainability and resilience [10,13]. A 2018 Nature article recognized the urgent need for research on the intersection of cities and climate change [3]. The article, and a subsequent publication from the National Science Foundation (NSF), called for increased understanding of “sustainable urban systems science”, and deeper partnership between researchers, policymakers, and practitioners to co-create knowledge and solutions together [14]. This research underscores the need for collaboration that supports innovation and transformation at the local level that can be shared and scaled globally.

City–university partnerships (CUPs) are emerging as mechanisms for the development, implementation and assessment of sustainability and resilience measures—creating the environment for sustainability science research to be more tightly coupled with sustainability problem-solving by urban policymakers [15,16]. Universities are particularly well suited to advance sustainability research and practices, especially at the local level, due to their position as “anchor institutions” or place-based organizations with a deep connection to the local community [17,18]. When universities and cities partner, there is an increased ability to build a transformative innovation ecosystem, where new ideas and solutions can be co-created, applied, and adopted for immediate impact [18,19,20].

Across the world, increasing numbers of CUPs are forming to support a range of climate change and sustainability-oriented work. For example, in the US, Smart City San Diego is a partnership between a university, municipality, utility company, and non-profit organization aimed at accelerating a regional transition to a green economy [15]. The Sustain-Lite project is a partnership in Singapore between a university, the local government, and a private business, responding to predicted growth of trade and commerce in Asia, and developing knowledge and tools for supply chain innovation [15]. Keeler and colleagues (2016) describe utilizing city–university partnerships across North America, Europe, and Asia to transfer and scale solutions to sustainability problems [21]. CUPs are rapidly developing at a global scale. For instance, the Educational Partnerships for Innovation in Communities–Network (EPIC-N), which unites local governments and communities with universities, now has 37 members spanning four continents, nine countries, and continues to grow [22].

While CUPs have structural similarities, e.g., they all include some form of agreement between researchers or administrators from universities and city administrators to formally collaborate, the partnerships operate in different modes. CUPs established to address complex sustainability problems such as climate change can be understood as falling into one of three modes: routine, strategic, or transformative, summarized in

Table 1. This table shows three types of partnership structures, their attributes, and the context in which they are most applicable.

Partnership Modes	Attributes	Context
Routine	Transactional, consultant-based or fee-for-service; Loose exchanges; One-sided deliverables; Limited joint efforts; One-off projects; Often based on individual (rather than institutional) relationships	Straight-forward problems i.e., community organization manager working with a researcher to develop maps of street trees
Strategic	Loose partnership; Shared vision and desire to co-create; Often externally focused	Complicated problems i.e., Business organization and city department collaborating on the development and implementation of initiatives for minority-owned businesses
Transformative	Formalized partnerships; Deep cross-institutional learning and mission alignment; Focused on internal and external systemic change	Complex problems i.e., City, university, and community coming together to develop holistic equity-focused climate adaptation plan

[23,24]. Routine partnerships are transactional and consultant-based; limited joint efforts that are suited for static and straight-forward problems (e.g., the City of Portland and Portland State University working together to develop a map of street trees) [13,25]. Strategic partnerships focus on co-creation with both the city and university partners contributing to the goals and design of the collaboration. Such partnerships are often addressing more complicated problems that are value-laden and have multiple solutions (e.g., Tempe, Arizona working with ASU to design and implement a process to create a climate action plan). Transformative partnerships are formalized, with deep cross-institutional learning and mission alignment; these are well-suited for complex or wicked problems that include long-term goal setting, contested solution spaces and regular evaluation and updating of developed solutions (e.g., the holistic partnership between the University of British Columbia and the city of Vancouver working to accelerate and navigate urban sustainability transitions) [25,26,27,28]. There is an increasing need for these kinds of transformative partnerships given the growing awareness and pressure to make progress on complex issues [29]. Understanding which partnership mode a CUP is operating within lays the foundation for evaluating a CUP for, among other things, coherence between partnership goals and partnership structure.

Developing effective monitoring and evaluation (M&E) techniques for all modes of CUPs is a vital component for intervention implementation, management, learning, and adjustment; for transformative CUPs, it is imperative. Iterative M&E of interventions provides real-world decision-making strategies for administrators, while also delivering comparable data for long-term research and analysis [30,31]. Appropriate development and implementation of CUP specific M&E tools can fulfil the real need to assess new and ongoing efforts and offer recommendations for improvement.

The partnership evaluation literature across several fields of study provides some common indicators for successful partnerships and collaborations. However, there is little guidance regarding specific strategies for CUPs seeking transformative sustainability and resilience outcomes. In general, partnerships and collaborations tend to be assessed based on: trust and trust-building; understanding context; shared history; mutual respect and understanding; member attitudes and beliefs; member satisfaction; processes, organization, and decision-making; communication; determination of goals and objectives; financial and human resources; and leadership [32,33,34]. However, there remains a need to guide partners in how to implement evaluative practices, relate assessment to outcomes, or integrate findings into ongoing partnership management, especially in the case of large institutions coming together for prolonged change. Additionally, further clarity is needed regarding how the general indicator categories of partnership assessment apply to the specific context of partnerships between cities and universities pursuing transformational sustainability outcomes.

Therefore, this paper aims to develop a research-based evaluation scheme for CUPs working on urban sustainability and resilience transformations. The article chronicles the development of an evaluation scheme to plan, monitor, and optimize CUPs for transformative resilience and sustainability outcomes. In so doing, the paper answers the following research questions: How can city–university partnerships be assessed for their capacity to contribute to long-term sustainability and resilience transformation? Specifically, what should be evaluated, who should be involved in the evaluation, and at what frequency? How is this knowledge formatively integrated into CUPs for their improvement? The research questions are answered through a combination of focus groups and evaluation design and application and results include a step-by step guide for real-time sustainability and resilience CUP evaluation.

2. Materials and Methods

The research team utilized an exploratory and confirmatory iterative focus group methodology as a knowledge elicitation technique to develop an operationalized evaluation scheme for city–university partnerships (CUPs) working on urban resilience and sustainability initiatives [35,36,37]. The focus groups were made up of experts currently involved in the development of transformative sustainability and resilience CUPs and were used to inform possible formative evaluation approaches, indicators, and tools. This consisted of three focus group meetings: one to develop the baseline schema, the second to refine the schema, and the final to reflect on the schema (Figure 1). Between each focus group session, the evaluation technique was applied to the CUP initiatives being undertaken by focus group participants as part of a participatory evaluation technique [38,39].

Focus groups functioned as generative workshops, bringing multiple experts together in one space, and using guiding questions to prompt development of tools, considerations, and opportunities for CUP evaluation. In particular, the groups were asked to think about indicators, metrics, and functional approaches for evaluation based upon their knowledge and skills. The researchers took notes at the focus group sessions which were compiled and sorted to uncover metrics and indicators that met criteria from three prominent collaboration evaluation frameworks: (1) the Collaboration Evaluation and Improvement Framework (CEIF) [40]; (2) the Relationships, climate, and expectations (RCE) framework; and (3) the Extent of collaboration (EC) framework [32]. These findings were then compared to existing literature on transition management and transformative partnerships, specifically, the principals for transferring partnership-based sustainability and resilience solutions across contexts [26]. Finally, metrics and indicators were fit into the deployment mechanics developed by the focus group participants to create the proposed CUP evaluation framework.

Expert knowledge elicitation focus groups were used for this research because they capitalize on communication between research participants in order to generate data. Compared to other types of group interviews, focus groups explicitly use group interaction as part of the method, therefore, people are encouraged to talk to each other, ask questions, exchange anecdotes and comment on each other’s experiences and perspectives [35]. Focus group methodologies are particularly useful for exploratory and applied research; identifying avenues of interest as new fields begin to emerge and when academic literature is thin.

A focus group was developed using a purposive sampling technique. Participants were chosen based upon their experiences in transformative sustainability and resilience CUP development and implementation, connection to inter-institutional partnership initiatives, and experience in research, evaluation, or monitoring of sustainability and resilience related interventions. All participants were currently actively engaged in the implementation of transformative sustainability or resilience initiatives through a CUP at the time of the focus groups, so iterations of the developed formative evaluation scheme could be directly applied.

The individuals selected for focus group participation contained academics and practitioners from five cities around the globe: Portland, Oregon, USA; Mexico City, Mexico; Leuphana, Germany; Karlsruhe, Germany; and Tempe, Arizona, USA. While not statistically representative, this group offered a wide range of experiences and expertise related to sustainability and resilience CUP planning, implementation, and transition management useful for the development of an operationalized evaluation scheme.

The goal of the first focus group session was to determine a starting point for the research and development of a formative evaluation approach for urban sustainability and resilience initiatives that utilize CUPs. There were 10 attendees in the group which consisted of: graduate students, post-docs, faculty members, and practitioners from local government. Attendees were from Germany, Mexico, and the United States. The focus group session was semi-structured, with researchers posing questions and participants responding free form to the questions and to the responses of the other participants.

The session consisted of exploring open-ended questions and prompts, related to how participants currently managed and evaluated their sustainability and resilience CUP work, and what was working well, or experiencing challenges. Questions were used to guide the conversation and to prompt generative and comparative discussion among the participants. Notes were taken and analyzed to develop answers to the questions, which were then combined with best-practices literature (as described above) to develop a formative evaluation scheme. The first version of the evaluation scheme was then applied by the researchers to the focus group participants’ initiatives.

The goal of the second focus group session was to present findings from the first version of the formative evaluation scheme and elicit feedback from the group to refine the scheme. The focus group session was loosely facilitated by the researchers and consisted of exploring open-ended questions and prompts related to the performance of the draft evaluation scheme, how well it represented the work, how findings could be integrated into CUP management, and what might need to be changed. Results from this session were compiled and used to create a refined version of the evaluation scheme, which was subsequently applied by researchers to the participants’ initiatives.

The goal of the final focus group session was to present findings from the application of the refined version of the evaluation scheme and elicit feedback from the group to reflect upon and finalize the scheme. The focus group session was loosely facilitated by the researchers and consisted of exploring open-ended questions and prompts related to the accuracy, usefulness, and overall design of the evaluation scheme. Findings from the session were consolidated and used as final refinements to the evaluation scheme.

3. Results

The focus group sessions and iterative process of evaluation development, deployment, and refinement resulted in a scheme that can be used to assess city–university partnerships (CUPs) for their capacity to contribute to long-term sustainability and resilience transformation. Specifically, results indicated: (1) what should be evaluated, (2) who should be involved in the evaluation, (3) how evaluation data is collected and disseminated, and (4) the frequency at which evaluation should occur. Finally, the results highlight how knowledge generated through the evaluation process can be formatively integrated into CUP management for their improvement.

An in-depth description of the proposed CUP evaluation scheme is described below. It begins with answering the practical questions of who, what, when, where, and why with regard to evaluation. It concludes with a simple step-by-step guide to implement the evaluation.

3.1. Indicators and Measures: What to Evaluate and Why

The focus group sessions and trial evaluations showed that assessing CUP progress requires understanding participant perceptions of both outcome-based and relational aspects of the partnership. Therefore, the proposed scheme includes two domains for evaluation: (1) perceptions of the collaborative project and (2) perceptions of the partnership itself. It is advantageous to gauge the status of these two domains separately, and then integrate knowledge between them for a holistic understanding of the dynamics of the CUP.

When evaluating perceptions of the collaborative project, three core areas, supported by several reinforcing indicators, are assessed. This is referred to as the Foundation, Actions, Impacts (FAI) assessment of CUP project development, implementation, and outcomes. FAI assessment uses short surveys and informal interviews to gauge participant perceptions related to each indicator (process details elaborated upon in Section 3.2 and Section 3.3). Each of the three core areas are described in further detail below:

• Foundation—Measures CUP participants’ perceptions of interest, competency, and capacity related to project undertaking. Seeks to understand feelings towards own organization and partner organization.
• Actions—Assesses perceived ability of all partners to plan and implement project-related change interventions in a co-created and co-managed way.
• Impact—Evaluates the perceived achievement of project goals and identification of opportunities for future work.

The FAI components are additive over the course of CUP project development and implementation (Figure 2). Findings indicate that when there are deficiencies in one of the earlier stages (for example, lack of interest in the foundation stage), it becomes increasingly difficult for the CUP project to thrive in later stages. By applying the FAI evaluation scheme, such deficiencies can be illuminated and mitigated. Additionally, if progress on the initiative becomes stalled or problematic, using this diagnostic tool can help direct where corrective action should be taken. It can also help identify where support is needed and aid the formulation of goals and plans to better match the evolving circumstances.

The second domain of evaluation measures participant perceptions of partnership functioning. This is used to understand partnership-specific dynamics, which may or may not be directly related to the current collaborative project. This part of the evaluation can help identify the partnership mode being utilized by the CUP and help participants match their partnership structure to the types of problems they hope the solve and their individual institutional contexts. For instance, if the CUP has transformative aims but the partnership is operating in the routine mode, then the assessment can be used to identify the mismatch and inform methods to shift modes. To measure partnership functioning, the Interpersonal Context and Empowering Supports (ICES) assessment can be used. The following aspects are considered ICES assessment indicators:

• Interpersonal Context—Measures participant perceptions of collaborative history between institutions, interest to engage, demonstrated motivation to engage, and mutual understanding of need. Seeks to understand perceptions of both own organization and partner organization.
• Empowering Supports—Assesses perceived and/or actual formalization of a partnership, mechanisms of the partnership, and resources committed on all sides of the partnership.

The results of the ICES assessment are compiled to describe the current typology of the partnership and its level of functioning (Figure 3). Findings are divided into three categories, low, medium, and high ICES. Each category relates to a partnership typology, which can be used to understand how well the partnership structure is aligned with desired projects and real-world outcomes. It can also expose when there is malalignment between perceptions of the partnership from different individuals or organizations. This allows for transparent conversations regarding the durability, efficacy, and purpose of the CUP.

Taken together, the Foundation, Actions, Impacts and Interpersonal Context and Empowering Supports (FAIICES, pronounced “faces”) evaluation scheme provides vital information regarding CUP structure and functioning from both a project-based and relationship-based perspective. It offers a mechanism for understanding how partnerships evolve in relationship to project milestones. The FAIICES scheme can be used to find points where the overall CUP system is lacking or out-of-balance, allowing CUP administrators to target efforts in those specific areas. Additionally, it can provide insight into areas where targeted action to develop the partnership, or evolve partnership typology, can be deployed.

For instance, one side of the partnership may be struggling to achieve goals by itself. Through application of the FAIICES evaluation, this could show up as a low score in the Actions category on the project part of the evaluation, and perhaps deficient resources committed on the partnership side. This highlights an opportunity for intervention in specific areas to make an impact; i.e., by facilitating foundational development in content area knowledge or having a tough conversation about shared resources. These capacity-building efforts help develop and align the CUP so that action can flow through the system effectively, and project objectives can be adaptively achieved.

3.2. Actors: Who Evaluates and Who Is Evaluated

The FAIICES evaluation scheme is designed to be participatory and flexible. As a participatory evaluation methodology, it is not meant to be a process where the evaluator is objectively removed, but instead they are an integrated part of the process. This works because the FAIICES scheme is about exploring perceptions of CUP functioning and incorporating findings into CUP management.

Being a participatory method allows for parts of the evaluation methodology to evolve depending on participant needs (especially timing, data collection, dissemination and more discussed further in Section 3.3, Section 3.4 and Section 3.5 below). Therefore, the evaluator can take many forms: From collective team-led evaluation, to a specific person on the team leading evaluation, to a semi-removed evaluator who has ties to the initiative, to an outside evaluator with at least some knowledge of the initiative. Additionally, the evaluator can be more than one singular person. Adherence to the FAIICES scheme is more important than who leads the evaluation.

However, the evaluator also plays an important role in building the evaluative capacity of the team. As evaluation occurs, the evaluator should be sure that the team is understanding the process, purpose, and usefulness of the evaluation, so that it might be conducted by a different entity in the future. In this way, evaluation can become ingrained in CUP management and the responsibility to evaluate can be shared.

Selecting the correct CUP members to be the subjects of evaluation is a more nuanced task. Not everyone who participates in the CUP needs to participate in the evaluation processes. As little as one person from each partner organization is required to complete the data collection portion. Evaluation participants should be central to the functioning of the CUP on both the relational and project-oriented sides. For most CUPS, there are no more than 1–3 key people on each side of the partnership with the insight, power, and positionality needed to be useful for this form of evaluation. The people involved in the evaluation should be the same people who can act on the findings and integrate them into the decision-making and management processes.

3.3. Tools: How to Collect Data and Disseminate Results

The FAIICES scheme can utilize several forms of data collection. We found that a mixture between short surveys and informal interviews, followed by a safe place to collectively examine results, worked well for the participants. One important feature of FAIICES is that it is not rigid; so long as perceptions of the indicators are being gathered, the method in which that occurs is less significant. This is a particularly useful feature when working between multiple institutions. For one side of the partnership, short, pointed surveys with Likert scale answers might best fit into their workflow and norms. Meanwhile, the other side might do better with informal, consultative interviews that get to an understanding of the indicators in a more conversational way. However, they are gathered, compiling the results and fitting them to the FAIICES indicators allows for a subjective, yet informative, comparison between perspectives.

This is a qualitative evaluation tool and evaluators are tasked with interpreting the results from interviews, surveys, etc. and matching them as best they can to the FAIICES attributes. Having a visual representation of results aids understanding and integration of results (see Figure 4 for an example score sheet and visual aid). When sharing the results, it is important to note the qualitative and subjective nature of the findings and note that they should be interrogated and explored. The notion of ambiguity in the results can stimulate more creative thinking and problem-solving in CUP participants.

While only a few people need to actively participate in providing data for the FAIICES analysis, the results of the evaluation can be disseminated more broadly. CUPs vary widely regarding the number of people involved, so dissemination practices need to be developed for specific contexts. If there is a core team of people who meet regularly to work on the CUP initiative, we recommended sharing results at one of these meetings. Here, through active dialogue and discussion, FAIICES findings can be inspected, scrutinized, and affirmed, hopefully leading to the generation of new goals and strategies for improved partnership management and project implementation.

3.4. Timing: When and How Often to Evaluate

Appropriate timing of the FAIICES evaluation is one of the most vital results of this study. The FAIICES scheme should be used iteratively and inform real-time decision-making. Depending on the context of the CUP, the evaluation should be completed about two to four times per project cycle. The timing of evaluation should revolve around key milestones or events, that way findings can be used immediately; results help both reflectively, and for future management decisions (Figure 5). This concept of participatory real-time evaluation has not often been used for CUPs, or other urban sustainability and resilience work. Development and testing of the FAIICES framework showed tangible potential for this approach to be seamlessly integrated into CUP partnership and project management.

3.5. Knowledge Integration: Using Evaluative Results in Real-Time

As mentioned throughout previous sections, it is imperative for the results of the FAIICES evaluation to be integrated into CUP management and decision-making in real-time. The value of the knowledge generated from the evaluation itself does not compare with the value generated through careful exploration and integration of results by CUP administrators. Collaboratively disseminating and investigating the findings from application of the FAIICES scheme helps to bridge gaps in understanding across institutional barriers and norms. Additionally, the process helps spur conversation and dialogue, which ultimately reinforces mutual understanding and trust.

Knowledge integration from FAIICES often highlights concerns, challenges, and opportunities that CUP participants may not have previously considered. In this way, the evaluation offers insight into leverage-points for higher impact interventions, or elicits strategies for navigating complex political, institutional, and real-world systemic barriers. Continuous, iterative, and strategically timed evaluation can help the CUP evolve and prosper through ever-evolving internal and external circumstances.

3.6. Implementation: Quick Guide to the FAIICES Evaluation Scheme

The FAIICES scheme is simple to begin and can change to suit specific contexts and needs over time. Box 1 shows a quick step-by-step guide for getting started on implementing a FAIICES evaluation:

Box 1. Step-by-step guide for getting started with the FAIICES evaluation scheme.

• Define your city–university partnership—Who is involved, what are your goals, why do you want to undertake this collaborative work?
• Choose an evaluator—Determine whether you want to collaboratively conduct the evaluation, or if you want to identify a specific person or people on your team or entity outside your team to undertake the evaluator role.
• Pick your evaluation participants—Choose at least one central figure from each partner institution to participate in the evaluation. These people should understand both the relational and outcomes-oriented sides of the partnership. They will be the subjects of data collection.
• Determine data collection methods—Decide whether open-ended surveys, Likert scales, informal interviews, or focus group sessions will be best for your participants (and feel free to get creative or adjust over time). Develop questions and prompts to explore participants’ perceptions of project foundation, actions, and impact as well as the partnership’s interpersonal context and empowering supports (see Section 3.1). Example open-ended informal interview questions and guidelines and example open-ended and Likert style survey questions are available in Appendix A.
• Conduct evaluation—Choose an appropriate time to conduct your evaluation, usually just before or after a key event or milestone (see Section 3.4). Get survey/interview responses from your key informants on all sides of the partnership.
• Analyze and compile data—Data analysis techniques will vary depending on the data collection methods used. Therefore, either quantitatively, qualitatively, and/or subjectively compile data to show institution-specific and combined responses for each FAIICES category; depict in a visual format if possible (see Figure 4).
• Disseminate and discuss—Soon after results have been compiled, schedule a time to collectively examine results. At minimum, the people who participated in the evaluation should be present, but this can also be expanded to include the larger CUP team. As a group, (typically led by the evaluator) go through the results, question them, add context, change or reinforce the findings.
• Integrate results into CUP administration—Have the management team think about any opportunities, challenges, or interesting findings that were exposed by the analysis. Question whether these findings indicate that a change in CUP typology, strategy, or goals is needed. Pay specific attention to places where modifications could lead to a better partnership trajectory, or tangible impacts. Finally, decide if and how to respond to these findings, and adjust CUP practices accordingly.
• Repeat FAIICES process—Follow the same instructions at the next appropriate evaluation time; you can then also explore how results change over time for deeper understanding of CUP evolution.

4. Discussion

This paper outlines a multi-faceted tool for the real-time monitoring and evaluation (M&E) of sustainability and resilience city–university partnerships (CUPs) derived from analysis of ongoing sustainability and resilience-focused CUPs. The Foundation, Actions, Impacts and Interpersonal Context and Empowering Supports (FAIICES) evaluation tool is useful for CUPs of all types but is vital for CUPS that aim to be transformative and attain transformational outcomes. The tool offers a mechanism for ongoing data collection on CUPs suitable for future research, and immediate, tangible, useful results for adept management of CUP initiatives as they happen. As municipal governments work towards solutions for exceedingly complex sustainability and resilience problems, CUPs are emerging as a strategy to accelerate learning, build capacity, and confront institutional barriers [14,15]. Successful CUPs will match the structure of their partnership to their sustainability goals. However, there is limited research on CUPs to improve their performance. This paper provides the FAIICES evaluation tool as one mechanism to guide the design and management of sustainability and resilience-oriented CUPs, in an effort to improve their contributions to sustainability outcomes.

If a CUP is interested in tackling the complexities of urban resilience and sustainability through long-term collaboration, establishing and maintaining a transformative partnership will be critical. In transformative partnerships, cross-institutional partners retain their identities but are willing to learn from and with each other through prolonged, deep engagement. Partners approach their common purposes in a profoundly collaboratively way and exhibit a greater willingness to promote deeper systemic changes both internally and externally [25,41]. While not all CUPs need to be transformative, many CUPs that are working on sustainability transformations are not achieving their goals or generating real world outcomes. This may be due, in part, to a mismatch between partnership structure and the specific problems and context. Successful transformative partnership administration calls for understanding how to think systemically and manage within systems. The FAIICES scheme offers users a way to reconcile their current partnership mode with their goals and develop pathways toward alignment.

How the FAIICES scheme is implemented matters. Effective implementation must: gauge perceptions of the CUP from all sides of the partnership; explore both relational and outcome-oriented aspects of the CUP; and occur in real-time (i.e., well-timed iterative formative evaluation for adaptive management). Gauging CUP participants’ perceptions of and perspectives on the indicator areas of interest proved to be more useful than measuring quantifiable metrics. Our results confirm that for the purposes of agile management and decision-making, perspectives play a critical role. For example, what one partner perceives as interest to engage from their collaborator matters more for relationship development than the actual measurable interest, i.e., impact is greater than intent. Future work should aim to connect methods of quantitative analysis to the FAIICES findings to better understand how the varying indicators relate to CUP outcomes and the qualitative measures used in this approach.

Additionally, our findings show that the project-based component and relationship-based component of the CUPs should be assessed separately but considered collectively. This is not often done in research on sustainability and resilience collaborations, as most research either focuses solely on project outcomes or solely on the collaboration itself [42]. With the FAIICES evaluation scheme, the relationship between these two domains is better understood, and can be used to make decisions that span across the domains. Future research should apply the FAIICES framework with an eye towards understanding the dynamics between the two domains, and how their interplay impacts CUP functioning and outcomes over time.

One of the biggest value-propositions that is generated by using the FAIICES tool is the ability to both collect data for immediate and longitudinal studies of CUPs while also immediately integrating findings into the CUP development, management, and implementation process. Historically, implementers have tended towards summative evaluation, which entails analysis of results compared to goals at the end of an intervention process used to make a judgement regarding efficacy [43]. Unfortunately, summative assessments often go uncompleted, or they occur after an intervention has ceased, so results cannot be directly integrated into implementation [44]. This is in direct contrast to the formative evaluation strategies that have been suggested by the sustainability and resilience transition management literature. It is suggested that complex work should be constantly re-evaluated and re-adjusted (adaptively managed) in an iterative way that supports agile decision-making and learning [29,45]. Our findings from this study confirm these results.

Finally, while this tool was developed specifically for city–university partnerships that are working on complex urban sustainability and resilience topics, it is possible that it can be useful for a much broader context. The FAIICES tool itself does not ask any resilience or sustainability-related questions; it also is not specific to the constraints of municipal governments or research universities. The metrics are focused on co-management, institutional alignment, and process in such a way that they are likely applicable to a wide range of collaborative efforts, especially those working on exceedingly complex or transformative issues. Further work is needed to understand how FAIICES might be applicable to these varying contexts.