Towards a Reflexive Framework for Fostering Co—Learning and Improvement of Transdisciplinary Collaboration

Abstract

Scholars in sustainability science as well as research funders increasingly recognize that a shift from disciplinary and interdisciplinary science to transdisciplinary (TD) research is required to address ever more complex sustainability challenges. Evidence shows that addressing real-world societal problems can be best achieved through collaborative research where diverse actors contribute different kinds of knowledge. While the potential benefits of TD research are widely recognized, its implementation remains a challenge. In this article, we develop a framework that supports reflection and co-learning. Our approach fosters monitoring of the collaboration processes, helps to assess the progress made and encourages continuous reflection and improvement of the research processes. The TD co-learning framework has four dimensions and 44 criteria. It is based on a substantial literature review and was tested in a Horizon 2020-funded research project ROBUST, which is applying experimental governance techniques to improve rural-urban relations in eleven European regions. The results demonstrate that the framework covers the key facets of TD collaboration and that all four broad dimensions matter. Each research-practice team reflected on how their collaboration is going and what needs to be improved. Indeed, the coordination team was able to see how well TD collaboration is functioning at a project level. We believe the framework will be valuable for actors involved in the planning and implementation of any type of multi-actor, interactive, innovation, transformation and action-oriented research project.

1. Introduction

The global challenges societies are facing are multidimensional, transcending disciplinary boundaries, multi-actor by nature and intertwined with a diverse and dynamic socio-political context [1,2,3,4,5]. Disciplinary and orthodox research methodologies have limited capacity to deal with complex challenges [3,6,7,8]. Sustainability science was established as a new field of research in the late 1990s to respond to the multidimensionality of challenges facing societies [4,9,10,11,12,13,14,15,16,17]. It was designed as a problem-driven and solution-oriented field [4], with the aim to consolidate knowledge and methodologies among “natural sciences, social sciences and humanities to create a new, holistic science” [8] (p. 2). It was also meant to be interdisciplinary.

This shift from disciplinary to interdisciplinary science is crucial to solve pressing global challenges, particularly those related to sustainability. Takeuchi (2014) goes beyond and argues that “transcending the interdisciplinary […] [and] the academic […] is the direction in which we must take scholarship [because] society must be included within the grand arena of sustainability science; it must not be shut out.” [8] (p. 3) Jahn et al. (2012) arrived at a similar conclusion, emphasizing the need to link scientific progress with societal needs [18]. Many other researchers support these views, adding that academic and practitioner knowledge needs to be brought together [7,12,18,19,20,21].

As a result, in the last 30 years theory and methodology on TD research within sustainability science have been evolving and maturing often derived from practical experiences. Early examples of the shift towards TD approaches include TD research carried out from 1999 onward at the Institute for Social-Ecological Research (ISOE) (www.isoe.de/en/) in Frankfurt/M; the research of ISOE led to major socio-ecological and transformation-oriented research programs funded by the Federal Ministry of Education and Research (BMBF) from 2000 [22,23]; and the research and educational activities carried out from 2010 to 2018 at the Lund University Centre of Excellence for Integration of Social and Natural Dimensions of Sustainability (LUCID) (www.lucsus.lu.se) [11].

The need to better link scientific progress with societal needs is also more and more recognized by research funders. Interestingly, related research projects are rarely labelled explicitly as ‘transdisciplinary’ but rather as ‘interdisciplinary with stakeholder involvement’ (examples include 94 funded projects under the Rural Economy and Land Use Program in the UK and 12 projects under Stand-Land-Plus program on rural-urban relations, Germany) or ‘multi-actor.’ The latter term features in the European Commission’s (EC) Horizon 2020 program (by 2019, the European Union (EU) allocated around one billion euros to fund around 180 multi-actor projects related to agriculture, forestry and rural development in the seven years of Horizon 2020 (2014–2020). Over 120 of these projects have already started (https://ec.europa.eu/eip/agriculture/en/about/multi-actor-projects-scientists-and-farmers), where ‘multi-actor projects’ are defined as “projects in which end users and multipliers of research results such as farmers and farmers’ groups, advisers, enterprises and others, are closely cooperating throughout the whole research project period” [24]

While the potential benefits of TD research are widely recognized and such projects are increasingly being funded, implementing TD research remains a major challenge [1,4,7,18,25,26,27,28]. Challenges include alignment of different values and interests, kinds of knowledge and methods; a gap between how TD research is conceptualized, planned and implemented; managing multi-actor collaboration; generating impact at different scales; and institutional barriers [1,4,7,26,28,29,30,31,32,33]. The key question is how to overcome the above mentioned.

Mitchell and Ross in the book of Fam et al. (2017) [2] presented a set of guidelines for improvement of TD research practice where reflexivity and learning are brought to the fore (reflexivity is defined by Bolton (2010) as being aware of and questioning, our own attitudes, thought processes, values, assumptions, prejudices and habitual actions. It leads to an awareness of the limits of our knowledge and of how our own behavior might disregard different views [34], pp. 14, 15). Similarly, many researchers consider either reflexivity or learning (individual learning, mutual, collaborative or co-learning, and social learning), or both elements key for the success of TD research [2,4,6,7,18,20,28,35,36,37,38]. In the literature, the terms ‘mutual learning’ and ‘co-learning’ are often used interchangeably. Mutual learning or co-learning is context-dependent and occurs through social interaction when partners with different expertise and knowledge are collaborating in TD research [2]. Social learning is defined as “a change in understanding that goes beyond the individual to become situated within wider social units or communities of practice through social interactions between actors within social networks.” [39] Westberg and Polk (2016) also discuss different kinds of learning in sustainability science and TD discourse [35]. We will use ‘co-learning’ in this paper to capture these different notions of learning.

Westberg and Polk (2016) emphasize that learning and its connection with reflexivity still do not receive the attention they deserve in TD research, adding that assessing progress in joint learning requires monitoring [35]. The related discussions are between more positivist, technical-rational models that assume that objective assessments are possible and those that argue for the need to acknowledge greater complexity, uncertainty, subjectivity and context specificity. In the latter case, the approach needs to be more deliberative, with an emphasis on inclusiveness and recognition of the validity of a wider range of voices and perspectives in defining legitimacy [2,4,18,19,37,40,41].

In light of the above, the aim of this paper is to develop an approach or framework that supports reflexive activities, facilitates learning and allows monitoring multi-actor research collaborations in order to assess the progress made and encourage continuous improvement. We particularly focus on TD collaboration as one of the key features of TD research. The main research questions are the following:

(1)

What broad dimensions determine TD collaboration?

(2)

What are appropriate criteria and related guiding questions that can support reflexivity and co-learning in TD collaboration?

(3)

What are the lessons learned from a first application of the framework in the EU-funded research project ROBUST on urban-rural relations?

In this paper, we first provide a brief description of the methodology used. This is then followed by a presentation of the TD co-learning framework with four dimensions that we elaborated based on a review of relevant studies. The four dimensions are presented in a tabular form operationalized with a total of 44 criteria, each with guiding questions. Some first lessons learned in applying the framework in the EU-funded research project ROBUST on urban-rural relations illustrate the effectiveness of the approach. We conclude the paper with a brief reflection on the importance of the four dimensions as well as a discussion of opportunities for wider application.

2. Methodology

2.1. Elaboration of the TD Co-Learning Framework Based on a Literature Review

In order to answer the first two research questions abovementioned, a significant body of literature was examined. In the review, we used Scopus, ScienceDirect and Mendeley databases. The keywords included ‘collaboration,’ ‘teamwork,’ ‘team science,’ ‘transdisciplinary,’ ‘participatory,’ ‘living lab,’ ‘evaluation,’ ‘reflection,’ ‘learning,’ ‘monitoring,’ ‘framework,’ ‘criteria’ and ‘indicators.’ Relevant grey literature was identified through supplementary searches. It comprised various reports, project documentation, as well as material published by the EC, in particular by Directorate-General for Agriculture and Rural Development. The search stage was continuous, which allowed spotting relevant publications that would not have been found otherwise through a single search procedure with a set of keywords. Then the titles, abstracts and keywords of the released articles were scanned. Additional criteria were applied at this stage to further narrow down the list of literature. We checked whether the literature includes any of the following terms highly relevant for TD research (1) theoretical frameworks and concepts related to teamwork, collaboration, (meta-)cognition, learning, mutual or collaborative learning, social learning, knowledge integration, social innovation or reflexivity or reflection; (2) theoretical or methodological discussions related to participatory, interdisciplinary, multidisciplinary, cross-disciplinary approaches including monitoring of participatory processes, reflexive assessment, evaluation methods and living lab methodology. Living Labs (LLs) can be defined as “an arena (i.e., geographically or institutionally bounded spaces) and … an approach for intentional collaborative experimentation of researchers, citizens, companies and local governments” [42] (p. 2). This is also the definition adopted in the ROBUST project.

Overall, we selected around 100 publications and several pieces of grey literature for the review. During the elaboration of this framework, particular attention was paid to elaborated frameworks relevant for this paper [9,36,37,43,44,45,46,47,48]. Thematically, the selected literature spans sustainability science in broad terms and specifically, sustainable food systems; socio-ecological and social-ecological systems; rural development; land use; environmental management and urban-rural relations. We also included literature on public health, digitalization and innovation, as well as policy to better capture the diverse nature of TD collaboration and related learning.

The material reviewed predominantly includes studies conducted within the last 30 years. According to Jahn et al. (2012) that is the period when transdisciplinarity is acknowledged to have gained momentum: “It is widely acknowledged that transdisciplinarity gained its current popularity through the works of Funtowicz and Ravetz (1990, 1991, 1993) on post-normal science and further still through those of Gibbons et al. (1994) on a new mode of knowledge production (‘Mode 2’)” [18] (p. 4). The reviewed literature also encompasses various cultural settings and geographical locations (including research carried out in Europe, North and South America, Australia and New Zealand).

We believe that the above steps allowed to base framework development on a sufficiently comprehensive literature review. The resulting TD co-learning framework is presented with relevant references in Section 3.

2.2. Testing of the Framework in the EU-Funded Research Project ROBUST

Recognizing the potential benefits of TD research, the EU-funded research project “Rural Urban Outlooks: Unlocking Synergies (ROBUST)” (For more information on the project, see the project website: www.rural-urban.eu) was designed to be TD. LLs are one way of working in a TD fashion as they bring actors from science, policy and practice together. In ROBUST, eleven LLs located in different European regions are put center stage. Each LL consists of research and practice partner teams. Research partners are represented by universities, research institutes and consulting firms, while practice partners are most often from a municipal government or regional authority overseeing regional development planning and policy. Research-practice partner teams in LLs have been brought together early in the project to collaborate on specific issues that are considered important for the specific region.

To enhance the LL approach, the project planning included a dedicated task to systematically monitor and reflect on TD collaboration processes. The task was also meant to facilitate necessary adjustment over the course of the project. One first result of this task is the TD co-learning framework presented in this article. The other important outcome will be a longitudinal dataset illustrating change in TD collaboration in the eleven LLs over the four-year project period. Three surveys are planned to do so: the baseline, the progress and the final with about a year in between.

The baseline survey was designed based on the framework presented in this paper. The criteria were chosen and the questions were formulated considering the current state of affairs in the project (e.g., what needs and can already be measured). The baseline survey was run via the online survey tool Qualtrics. It was implemented in April 2019 and included 30 questions. The baseline survey was anonymous with only three questions related to personal data: LL team, type of partner (research or practice) and disciplinary background. Out of a total of 79 partners in the LLs, 57 completed the survey, of which 32 were research and 25 practice partners (overall response rate: 72%).

The baseline survey was deliberately implemented only when LLs had started tangible joint work. The survey serves as a benchmark against which to measure change and it was used to refine priorities and processes. The lessons learned from the first application of the framework are presented in Section 4.

3. TD Co-Learning Framework

In this section, we will present the framework. We will describe the general structure of the framework followed by a discussion of each of the four broad dimensions with the related criteria, literature references and guiding questions.

3.1. Structure and Use of the Framework

The TD co-learning framework is structured along four dimensions as the literature review showed that these dimensions are the most essential when assessing the functioning of participatory and TD research processes (see Appendix A

Table A1. Overview of the main literature examined to select the four broad dimensions for the framework (in the alphabetical order).

Source	Thematic Focus	Context	Approach	Process	Outcomes
1. Benson et al., 2014 [117]	Participatory governance, management of water resources, evaluation, learning			X	X
2. Blackstock, Kelly, and Horsey, 2007 [9]	Participatory research for sustainability, evaluation, social learning	X	X 1	X	X
3. Burgess and Chilvers, 2006 [51]	Participatory technology assessment, evaluation, new governance	X		X	X
4. Carew and Wickson, 2010 [1]	TD research, research planning, supporting, evaluation	X		X	X 2
5. Fam, Palmer, Riedy, and Mitchell, 2017 [2]	TD research, practice for sustainability, reflexivity, learning, governance	X	X	X	X
6. Hansson and Polk, 2018 [118]	TD research, sustainable urban development, evaluation			X	X
7. Hermans, Haarmann, and Dagevos, 2011 [51]	Stakeholder participation, monitoring regional sustainability, evaluation	X		X	X
8. Holzer, Carmon, and Orenstein, 2018 [44]	TD research, socio-ecological systems, methodology, evaluation			X	X
9. Hubeau, Marchand, Coteur, Debruyne, and Van Huylenbroeck, 2018 [36]	TD research, agri-food systems, reflexive assessment, sustainability, transformation	X	X 3	X	X
10. OECD, 2018 [46]	Triangular cooperation, value-added, monitoring, evaluation	X 4		X	X
11. Schuurman, De Marez, and Ballon, 2016 [74]	Living Labs, open innovation, impact, small and medium-sized enterprises		X		X
12. Siebenhüner, 2018 [119]	TD research for sustainability, knowledge integration, co-learning			X	X
13. Stokols, Harvey, Gress, Fuqua, and Phillips, 2005 [120]	TD research, collaboration, evaluation, tobacco use science and prevention	X 5		X	X
14. Taplin and Clark, 2012 [121]	Theory of Change, planning, monitoring, evaluation, outcomes, indicators	X		X 6	X
15. Van Geenhuizen, 2018 [47]	Living Labs, user-centered innovation, boundary spanning, evaluation	X 7	X 8	X	X
16. Veeckman, Schuurman, Leminen, and Westerlund, 2013 [48]	Living Labs, characteristics, outcomes, user-centered innovation, evaluation,	X 9	X		X
17. Walker, Rahman, and Cave, 2001 [122]	Adaptive policies, policy analysis, policymaking, outcomes	X 10			X
18. Walter, Helgenberger, Wiek, and Scholz, 2007 [54]	TD research, evaluation, knowledge integration, co-learning			X	X
19. Vogel, 2012 [123]	Theory of Change, monitoring, evaluation, international development	X		X 11	X

¹ ‘Methods’ in Blackstock et al. (2007) [9]; ² ‘Product’ in Carew and Wickson (2010) [1]; ³ ‘Methods’ in Hubeau et al. (2018) [36]; ⁴ ‘Activities’ in [46]; ⁵ ‘Antecedents’ in Stokols et al. (2005) [120]; ⁶ ‘Interventions’ in Taplin and Clark (2012) [121]; ⁷ ‘Exogenous influences’ and ‘Inputs’ in van Geenhuizen (2018) are included under ‘Context.’ According to van Geenhuizen (2018), inputs include the motivation and capabilities of actors; sets of learning tools and models; specific expertise; financial budgets and other resources, as well as the real-life environment. Van Geenhuizen (2018) describes LL processes that are beyond control of managers of LL as ‘exogenous influences’ [47]; ⁸ ‘Inputs’ in van Geenhuizen (2018). van Geenhuizen (2018) lists methods and tools of LL processes as ‘Inputs’ [47]; ⁹ ‘Environment’ in Veeckman et al. (2013) [48]; ¹⁰ ‘Stage setting’ in Walker et al. (2001) [122], which includes important objectives, constraints and policy options; ¹¹ ‘Activities’ in Vogel (2012). [123].

). The framework consists of the following components:

• Four dimensions: context, approach, process and outcomes;
• 44 criteria with related literature references;
• Guiding questions for each criterion.

To capture the plurality of values and perspectives of the actors involved, we followed the four principles [43] when selecting criteria:

• relevance, social significance and applicability;
• credibility, integration and reflexivity, added to traditional criteria of scientific rigor;
• legitimacy, inclusion and fair representation of stakeholder interests;
• effectiveness, that is, actual or potential contribution to problem solving and social change.

In order to harness the potential of the framework, we suggest translating the criteria into questions that make sense in the specific context. The questions provided in the tables in Section 3.2 are indicative. The final selection of the criteria, including the number of criteria a team intends to use to monitor and assess team collaboration and the formulation of questions depend on the precise goals and needs of a particular team or project.

The framework can also be used to track change over time, for example over the course of a project. The guiding questions can therefore be used as a reference point, but they need to be adapted to the particular project stage. For example, questions relevant at the beginning of a project can be selected for a baseline survey. During later project stages (e.g., with progress and final surveys), emphasis can shift towards outputs and impact.

Collecting data at individual and team levels is crucial as a team view might not always reflect the individuals’ opinion [49]. Distinguishing between both levels also provides interesting entry points to a deeper analysis, for example by examining divergences in views. More importantly, questions posed to the team might trigger valuable team level discussions and can therefore be applied to teams or individuals or to both.

Table 1. Overview table of the four dimensions of the TD co-learning framework with key criteria and indications.

Dimension	Key Issues
Context	Represents the setting in which TD collaboration is taking place
	Organizational structure, resources and infrastructure, Boundary setting Real-world context Number and diversity of actors Level of openness Early involvement of key actors, engaged community
Approach	Defines the broad research approach taken and related methodological aspects
	Joint learning and complementary knowledge Use of participatory methods and co-creation Use of action-orientated approach Use of systems approach Reflexivity and monitoring, feedback loops and refinement
Process	Encompasses the way the cooperation is implemented, organized and managed
	Common vision, genuine inclusion, common language, effective communication Ownership and trust, appreciation and respect Competences, knowledge integration, co-learning and co-creation Leadership, roles and decision-making Management and conflict resolution
Outcomes	Subsumes intended and unintended outputs, effects, outcomes and impacts
	Relevance, effectiveness, unintended effects and efficiency Dissemination, networking and mobilization of additional support Transformative learning, capacity-building Satisfaction of core constituencies Impact, comparability and transferability of findings, legacy

provides a summary overview of the key issues covered with the framework.

3.2. The Four Broad Dimensions in Detail

3.2.1. Context

The setting in which a research process occurs comprises institutional, political, socioeconomic, environmental, historical and cultural factors [9,36,50,51]. Available resources and infrastructure, the degrees of freedom in research and so forth are all context-dependent. Hermans et al. (2011) emphasize that context factors might differ significantly from region to region, and that as a result, the same research process may yield different results [51]. Context also influences the other three dimensions in the framework: the approaches and methods used are tailored to project- and location-specific aims; TD processes change as well and all of this has a significant influence on outcomes and how they are being perceived and assessed (

Table 2. Context—Criteria, related literature and guiding questions.

Criteria	Related Literature	Guiding Questions (Indicative)
Organizational structure, resources and infrastructure	Technical infrastructure [48,52]. Flexibility [37]. Access to resources [9]. Institutional context [36,53]). Historical context [36,54]	To what extent does your team have what it needs to work effectively in terms of skills and expertise/data and information?
		To what extent does your team have enough financial resources/time?
		Does your team have effective connections to decision-makers, government agencies, relevant organizations and your target group/s?
Boundary setting and strategic planning	Boundary setting [2,55]. Strategic planning [37]. Explicit theory of change; Relevant research objectives and design [43]. Opportunity to influence [9]. Lifespan [48,56]	Which actors need to be involved in the work?
		What can best be done with the available time, money and person power (what is central and what is marginal)?
		Has the collaboration changed over time? If yes, how and why?
Real-World context	Real-world context [48]. Societally relevant research problem [43]	To what extent does the work in your team address the challenges in the region perceived as important by relevant stakeholders?
Number and diversity of actors	Scale [48]. Actor-complexity [47]. Building a heterogeneous network [41]. Team diversity [57,58]	How comfortable are you with the size/background/skillset of your team?
		In what way does the diverse background/skillset of your team members affect the way you interact with each other?
Early involvement	Early involvement [47,59,60]	Were you engaged in the project from early on?
		From your experience in (research) projects, to what extent does early engagement make a difference for successful collaboration?
Transparency	Level of openness [47,48,60]. Internal and external transparency [4,9,36,54,61]	Does everyone in your team have access to the results of the joint work and the jointly generated new knowledge?
		Can everyone in your team use the key resources without any limitations?
Engaged community	Community [48,62]. Sense of belonging [57]. Continuous commitment and engagement [20,37,47,59,63,64]. Involvement of participants and sense of urgency [41]. Connectivity between researchers [65]. Supporting face to face interaction [66,67,68,69,70]. Empowerment of practitioners [26]	How are you being informed about advances, changes etc. in the joint work in your team?
		Do you send each other relevant materials, such as interesting papers, studies with breakthrough results?
		How frequent on average do you communicate with your team colleagues?
		Which way of communication with your team colleagues do you find the most effective in between face-to-face meetings? What has proven to work well/not at all for your team?
		Would you prefer to meet your team members in person more often?
		Do your colleagues display high levels of cooperation and mutual support?

).

3.2.2. Approach

Approach is included in the framework as it plays a pivotal role in TD research processes. Methodologies are selected based on the broad approach taken. A key question related to ‘approach’ that is of particular interest in this paper is whether a research project is action-orientated. Ramalingam, Wild, and Buffardi (2019) consider monitoring, evaluation and learning crucial in TD research processes, emphasizing that all three should be continuous [40]. “Mutual readiness to reflect, to listen to each other’s views, interests, experiences” and “reflexive monitoring in action” as a practice-oriented approach are suggested for system innovation projects by van Mierlo et al. (2010) [41]. Others, emphasize the importance of co-reflection and adaptive learning [37] as well as of continuous formative evaluation for TD research in sustainability science [4,40,54] (

Table 3. Approach—Criteria, related literature and guiding questions.

Criteria	Related Literature	Guiding Questions (Indicative)
Use of an action-orientated approach	Action-orientated approach [47,71]	To what extent is an action-orientated approach prioritized in your team?
		How well are your partners able to elaborate strategies that are likely to work in the region?
Joint learning	Joint reflection and learning [41,54]. Reflexivity [2,4,7,18,28,37,40,41]. Adaptive learning [37,40,54]	To what extent are you ready to learn from team members with different backgrounds?
		Do you mind putting ideas of your own up for discussion?
		How important do you find joint learning for your team?
Monitoring and evaluation	Reflexive monitoring in action [41]. Monitoring [72]. Monitoring and learning [40]. Ongoing monitoring and reflexivity [43]. Evaluation [47,48,72,73]. Continuous formative evaluation [4]. Participatory evaluation and learning [9,40,47]	Does your team have mechanisms to monitor progress/results? Which ones?
		Does your team evaluate the progress and impact? If yes, how?
Multi-method	Use of multiple approaches and tools [47,74]	Do you combine different kinds of methods in a carefully considered way to achieve goals?
Use of participatory methods	Participatory approach [9,61]	Do you use participatory methods in your teamwork?
		What methods proved most effective for you personally/your team?
Systems approach	Systems approach [47,71]	How well is your team able to carry out more encompassing analyses?
		Have you defined the boundaries of the system you are examining?
Complementary knowledge	Complementary knowledge [20,46]. Contribution of different actors [43]	To what extent do the complementary strengths of team members help achieving goals?
		To what extent is an effective combination of perspectives, resources and skills of the team members actively promoted in the team?
Feedback and adjustment	Iteration, feedback and refinement [47,75,76,77,78,79,80,81]. Adjustment of activities [41]. Adaptive decision-making and revision [37]	Do you feel that there are enough feedback loops in the team?
		Is the feedback received from your team members sufficiently integrated into the planning of next steps?
		How is the feedback process organized in your team? On what occasion? Who is involved?

).

3.2.3. Process

TD research and cooperation among actors with different backgrounds means multiple perspectives, diverse sets of skills and experiences, differences in the terminology used, which often means also different and sometimes conflicting goals. Co-learning, co-creation and co-production also mean working across disciplinary and sectoral boundaries. Professional and personal relationships can therefore play a major role in TD cooperation. The common ground essential for successful collaboration and trust need to be built and this takes time. It follows that the levels and types of stakeholder involvement, ownership and trust, appreciation and respect, the processes in place, leadership, management, decision-making and so forth are key issues (see for example [36]). In our framework, the process dimension encompasses these issues, and more generally, the way the cooperation is implemented, organized, managed and functioning (

Table 4. Process—Criteria, related literature and guiding questions.

Criteria	Related Literature	Guiding Questions (Indicative)
Common vision and genuine inclusion	Creating a common vision, alignment of priorities/interests [9,46,47,48,63,71,74,82]. Shared identity and values [66,83,84]. Inclusiveness [28,71]. Genuine and explicit inclusion [43].	How well are the team members able to include (reconcile) the views and priorities of all involved in the team? Is there a positive/negative example?
		To what extent does your team try to coordinate plans with others in the project? How is the coordination achieved?
Common language	Common language [26,52,59,63,64]. Clarification for “shared interpretive horizon” [66,84,85,86]	How well do you feel your colleagues understand what you are saying?
		Do you have a common glossary of key terms?
		How well is your team able to express goals in a way that is supported by all involved?
Personal motivation	Passion [87]). Motivation [47,48,61,74]. Motivation and encouragement [54]. Strategic intention [87]. Willingness to learn [9,36,54,61]	How often do you go beyond what is required in the tasks assigned to you?
		How much do you feel your team members are inspired and motivated about the work they are doing?
		How satisfied are you with the contribution of other colleagues in the team?
Ownership and trust	Building ownership and trust [46,47,48,60,74]. Ownership of outcomes [9]. Trust [41,54,66,84,88]. Trust and respect [89]	Are you ready to share products that are still work in progress? Why/why not?
		How satisfied are you with the co-ownership of the results of joint work?
Appreciation and psychological safety	Appreciation and respect [47,48,60,74]. Psychological safety [57,90,91]	To what extent do you think your contributions are valued in the team?
		To what extent does your team provide an environment where different opinions can be voiced?
		To what extent do you feel you can take risks in your team voicing a less popular view?
Effective internal communication	Internal communication [9,36,54,61]. Participatory events [61]. Effective communication [43].	To what extent are you aware of who is doing what in your team?
		How effective are your meetings?
Competences	Knowledge and skills [37,47,87]. Communication skills, team working skills and a broad perspective [92]. Capacity to participate [9,36,54,61]. Adequate competences [43]	Are there any skills that you think are missing in your team to do the job more effectively? If yes, which?
		To what extent do you feel your actual responsibilities in the team correspond well with your knowledge and skills?
		Is there any form of training to keep team members’ skills up-to-date?
Co-learning	Co-learning [20,41,47,54,61,76,78,80,81]. Mutual learning [1,4,33,36,50,54,93,94,95,96,97]. Knowledge exchange [26,37,60,90].	Do you implement dedicated measures to support co-learning? If yes, what kind of measures?
		To what extent do you recognize the value of your own knowledge in co-learning?
		What are the top five most challenging obstacles limiting effective knowledge exchange?
Knowledge integration	Knowledge integration [2,20,44,93]. Knowledge management [98]. Knowledge diversity [57,99]	Are you using particular methods or tools that connect different kinds of knowledge (e.g., when addressing an issue/problem you are working on)? If yes, can you provide examples?
		What processes/tools do you use to keep track of available knowledge?
Co-creation	Co-creation [47,48,52,60,81]	By working together, how well are the team members able to identify new ways to solve problems?
		How do you share the work? How is it decided who is doing what?
Leadership	Leadership [4,9,36,37,52,54,57,77,100] Presence of ‘prime movers’ [41]	Who is leading your team? What are the reasons?
		To what extent does a team leader encourage team members to be creative and look at things differently?
		How effective is the team leader’s communication style?
Actors’ roles	User/stakeholder roles [43,46,61,101]. Flexibility in actor roles [47,102]. (New) roles for research(ers) [98]. Role clarity [57]	Are the roles of different members in your team properly defined?
		How balanced is the distribution of work between different team members/gender-wise?
		Do you feel your role in the team has changed over the course of the project?
Decision-making	Decision-making [52]. Quality of decision-making [9]	Does everyone in the team have the same influence on decisions?
		How comfortable are you with the way decisions are made in the team?
Administration and management	Administration and management [45,72,77] Appropriate project implementation [43]. Harnessing differences [55]	Are the results of the team’s decisions and action points documented?
		How effective is the preparation of team-level decisions?
Conflict resolution	Conflict reduction/mitigation/resolution [4,47,52,54,61,82]. Harnessing differences [55,66,83,84,86,88,103]	Were there diverging views when deciding on key issues? If yes, how did you deal with that?
		Were there diverging views on the research agenda? If yes, how did you deal with that?
		To what extent are you as a team able to work through differences of opinion without damaging relationships?
		Do you use professional facilitators?

).

3.2.4. Outcomes

From the point of view of practice partners and stakeholders involved in TD research projects, expected outcomes are central for their motivation and engagement. Whenever the aim is to jointly make a real difference and to arrive at results that are tangible, meaningful and applicable, practice partners tend to engage. In this respect, Theory of Change is helpful because it starts with the joint articulation of long-term goals, expected outcomes and impacts. Subsequent steps are identifying the conditions for those goals to be met. Walter et al. (2007) refer to “long-term effects representing [the] goals of the TD project.” [54] In our framework, the outcomes dimension comprises intended and unintended outputs, effects, outcomes and impacts (

Table 5. Outcomes—Criteria, related literature and guiding questions.

Criteria	Related Literature	Guiding Questions (Indicative)
Relevance	Relevance [37,43,73,104]	Are the aims you formulated (still) relevant?
		Are the activities consistent with the main aims/intended impacts?
Effectiveness	Effectiveness [63,73,77,104,105,106,107]	To what extent is progress being made in relation to the aims of the project?
		What are the main factors influencing the achievement of objectives?
Efficiency	Efficiency [39,46,47,104]. Professionalism [37]. Cost effectiveness [9]	How efficiently does the team use different resources in achieving goals?
		Are the planned activities/budget lines sufficiently on schedule?
		How productive are the meetings?
Unintended effects	Unintended effects [37,46,47,60,73,77]	Is your research guided by the principles of Responsible Research and Innovation (RRI)?
		Do you have processes/methods in place that aim at spotting negative impacts?
Communication and dissemination	Communication and dissemination [64,66,68,84,86,108]. External communication [36]. Distribution of knowledge [54,61]	Are you regularly communicating and disseminating (e.g., interim) research results?
		How do you check whether you effectively reach your target groups?
Mobilization of additional support and network building	Mobilization of additional resources, networks and institutions [46]. Network building [45,47,54,60]. Network relationships [9,36,37,61]	How good is your team in obtaining support from individuals or organizations in the region to help move things forward?
		What other networks/organizations have joined over time in order to foster upscaling and multiplication??
		What additional resources were mobilized?
Satisfaction of core constituencies and community identification	Satisfaction with collaboration [47,109]. Community identification and ‘‘sense of belonging’’ [54]. Satisfaction of core constituencies including expectations; accountability [9]. Satisfaction and commitment [99]	How satisfied are you with your role in the team/the team’s plans/strategy? What can be done better?
		Are representatives of core constituencies/the local community actively engaged in project activities?
		Does the project have an influence on governance arrangements/decision-making in your region?
Benefit(s) received, usefulness	Value creation/sharing "ecosystem" [48,60]. Personal value [52,110]. Impact of collaboration [109]. Recognized impact [9,36]. Proximal outputs (team members’ learning and professional development) [57]	To what extent has your ability to influence policy/meet the needs of your region increased?
		Have you experienced any drawbacks as a result of the collaboration?
		How does the value of collaboration compare to the drawbacks?
Social learning	Social learning [7,9,37,38,39,66,81]. Individual and transformative learning [36,66]. Emergent knowledge and influence of local knowledge on outcomes [9]	Can you identify some examples on how the teamwork has changed the way you see things and your professional practice?
		Can you give an example of how your personal expertise or experience has influenced discussions and directions taken by your team?
Capacity-building	Capacity-building [9,36,37,38,54,60,61,63,98]	To what extent has your ability to apply scientific concepts to addressing real world problems been improved?
		Can you identify some transdisciplinary methods you learned to use?
Comparability/transferability of findings	Comparability of results across regional/national/international contexts [71]. Transferability of results [43,47,111,112]	To what extent can what you learned be applied in other contexts?
		What are the main factors that influence the transferability of findings?
Impact	Significant outcome [43]. Impact [37,46,47,60,73,77]. Type and degree of impact observed [60]. Recognized impacts [9]	Can you identify some examples of actual changes in policy or practice? (instrumental)
		Can you identify some examples of how the broad understanding of the issues studied has been improved? (conceptual)
		Can you identify some examples of increased willingness to engage in new collaborations? (culture change)
Legacy	Sustainability [46,47]	To what extent do you think will the benefits of the project continue after funding ceases?
		What are the main factors for the benefits of the project to last (e.g., institutionalization of new networks and exchanges)?

).

4. Lessons Learned from a First Application of the Framework

In this section, we discuss some first lessons learned when testing the framework in the ROBUST project (see Section 2.2). Overall, the results of the baseline survey on the functioning of TD collaboration were predominantly positive. They, inter alia, indicated the following:

• The partnership spirit is strong in almost all LL teams. This was manifested for example through the joint preparation of the Research and Innovation Agenda that is central for LL work (39% of respondents reported ‘joint drafting, sharing of preparation’ and 35% of respondents described it as ‘drafted by the research partner with significant feedback from the practice partner’).
• A blend of different kinds of expertise in LLs is viewed very positive for achieving LL goals with 69% of respondents ranking it high.
• 77% of respondents feel that their personal contribution to the teamwork is valued (this is almost equally the case for research and practice partners).
• The three so far received personal benefits reported as the most important featured: development of new valuable relationships (72% of respondents), acquisition of new knowledge (61% of respondents), opportunity to address an important issue (61% of respondents).

As the main purpose of the TD co-learning framework presented in this paper is to continuously enhance research-practice collaboration, the main interest in its use is to identify the main challenges in implementation that need to be addressed. We will therefore in the following focus on the challenges encountered at both LL and project levels, as well as briefly discuss implications. We also explain how we adjusted the use of the framework to better meet the needs of the project. As reflexive components were rather new for most project partners, engaging them in this exercise was a first challenge. One common concern was that we would focus too much on collaboration processes instead of the “much more important” research content-related issues.

In the following we will present the main issues that emerged so far. They were identified based on a combination of the analysis of the baseline survey data, continuous observations during project meetings and records related to relevant email exchanges and Skype calls. The challenge of building trust will not be discussed as it is meanwhile broadly reported and confirmed by many scholars working in TD research.

Most of the challenges we discuss are closely interconnected.

Challenge 1.

Disciplinary and professional lenses can cause underestimation of the complexity of TD collaboration.

The way the partners are seeing TD collaboration is shaped by their disciplinary and professional perspective. By professional perspective we mean epistemological positions in the disciplines they are working in and the specificity of their work, such as established professional norms, routines, processes and methods. For example, colleagues who are used to work with quantitative methods and impact evaluation might not immediately recognize the value of a qualitative or discursive approach; so the method might therefore seem less rigorous or not appropriate. A quote illustrates this view: “I believe that having serious discussions based on the survey done with five participants from a team is not a serious approach and base to work with.” This phenomenon was discussed by Mitchell and Ross [2] (p. 174): “what we take to be true is deeply connected to epistemological positions and these differ radically within and between disciplines. Can truth only be found in large statistically significant results? Or are people’s experiences and stories equally powerful sources of rich and meaningful data? Transdisciplinarity requires articulating between different forms of knowledge–talking across different ways of knowing, different forms of truth.”

In the approach we apply in ROBUST, no one needs to evaluate or is evaluated; instead, we aim at triggering joint discussion of key issues which is crucial in meeting project goals. Furthermore, some professionals working in the disciplines such as spatial planning and economics, do not require much interdisciplinary collaboration. Collaboration with diverse actors is therefore something new for them to embark on. Professional jargon hindering effective communication and mutual understanding is a closely related problem (this is in line with for example [30]). All of the aforementioned demonstrates that the view of partners can be limited and fragmented, which leads to the next issue.

Challenge 2.

Reflexive activities are not needed, ‘not helpful’ and ‘too demanding.’

Some partners consider the investment of time and efforts in monitoring and reflexive activities not worthwhile, important or helpful. In some instances, the big picture and complexity of the project are not recognized. This inevitably means that monitoring and reflexivity are perceived as not needed and are seen instead as a ‘distraction’ from the main (content-related) project activities. A quote from one of the project partners can illustrate this: “Our […] partner did not join the project to participate in the experiment of TD collaboration. They joined to address the real problems they are facing in the region.” Another partner commented: “We consider this exercise demanding for many project participants and teams. [It is not clear] how it can contribute to and support the assessment of project progress.” The implication for project planning and implementation is that monitoring and reflexivity components must not become cumbersome for partners. Their benefits need to be clearly communicated and the related activities need to be carefully planned and efficiently implemented. For example, combining project reporting with monitoring and reflexive elements reduces the ‘burden’ and motivates partners.

Challenge 3.

Psychological barriers and comfort zones.

Another issue that emerged as rather stand-alone is reluctance to openly talk about difficulties and tensions. Interestingly, this concerns both the initiation of open discussions within a specific team and the communication to ‘outsiders,’ for example, a project coordination team or another LL team in the project. At the survey design stage when we were asking the projects partners for feedback, one of the partners remarked: “Many of the questions are oriented at ‘problems’ and challenges encountered in the project. This might raise reluctance among participants to provide a sincere assessment.” In general, both research and practice partners took great care to avoid potentially conflictual discussions—sometimes at the cost of continuing with unresolved issues and in some cases, causing frustration of individual team members. Not willing to reveal problems to ‘outsiders’ is a common psychological phenomenon [113]. People prefer to hide difficulties from others since they do not want to show weaknesses and be judged. The latter was a greater concern for research partners.

Challenge 4.

Institutional frameworks can limit the nature of collaboration.

The institutional framework the partners are part of can greatly affect their room for maneuver in collaboration (see also [53]). For many practice partners, such as regional administrators or spatial planners, the agenda, the processes and the way of working can be rather fixed. Activities and work routines tend to be strictly regulated. Space for participatory activities, co-learning and collaboration is often very limited and for some practice partners this is a challenge. The following quote illustrates this view: “We normally do not interact with the general public. The results of the LL process will feed into the planning procedure; debating the topic now with planners from the municipalities would be deviating and municipal politicians have other agendas. […] This is supposed to feed into a legal procedure and needs a high degree of formality.” In some way this issue overlaps with individual’s perspectives that reflect the particular professional environment. The same applies to research partners who often cannot understand the restrictions and routines in private and public sectors. Getting to know each other’s ‘rules of the game’ is a precondition for good TD collaboration.

Challenge 5.

Matching the interests and competences of partners with requirements.

Reconciling the interests of research and practice partners and their capacity and expertise with requirements at project and LL levels is a major issue. Partners rate the importance of various outputs and outcomes differently. Problems are further aggravated by the evolving agenda of LL teams where the interest of either of the partners might be moving towards the area where the other partner is not very interested in but also not very competent. The original idea in the project planning was to bring complementary competences and skills together when building LL teams. Our experience shows that the supply and demand of competences is not always matched well and sometimes team members are not even aware of the skills and expertise their colleagues possess. Mallaband et al. (2017) come to a similar conclusion contending that sometimes the capacity, skills and expertise of particular team members are not clear for other members and they do not understand what these other members can ‘bring to the table’—“in terms of the methods used, credibility of the data and conclusions and the ‘real world’ impact.” [30] (pp. 12–13) Our own conclusion at this point is that even more attention should be paid during work planning to more precisely matching the interests and competences of partners. However, we acknowledge that even paying closer attention to this aspect does not guarantee success as agendas and relationships are evolving. As one of the project partners in ROBUST noted: “Partner selection is very much dictated by circumstances. When it comes to working with local governments and public authorities, policymakers agree to join the project but at the end of the day, civil servants are those involved; both type of actors are not always aligned and most importantly, the mandate of policymakers tends to be limited, while civil servants remain.”

Challenge 6.

Gap between TD theory and practice.

Our experience so far shows that there is a significant gap between the theory, methodology and guiding principles for TD research ‘on paper’ and the way it is implemented in practice. A statement from one of the project partners illustrates this: “These ideas that LLs [meaning TD research as well] will run on their own is just not true. LL work immediately ceases once researchers stop pushing it. These LLs are just too much effort, a lot of pressure is put onto the researchers’ shoulders to implement them without much results.” In theory, LLs are meant to be a joint effort where all partners share responsibility—which infers a rather different mindset. One of the reasons for the gap between theory and practice is the rapid evolution of TD concepts and methodologies. The problem is aggravated by the continuing predominance of disciplinary perspectives, and on the side of practice partners, the rather entrenched professional routines. The evolving and adaptive nature of TD research confuses many actors about what exactly should be done and how, compared to the rather clear goals and linear processes that actors are more used to within their disciplinary and professional environment (see for example [114]).

The six challenges identified above, are in line with the literature touched upon in the introduction. Even though the challenges we have encountered and illustrated above are so far only based on a first application of the framework, ongoing discussions and email exchanges at LL and project levels, they by and large correspond with those previously identified. The added value of applying the TD co-learning framework in our project was that it allowed to identify issues early and trigger much needed discussions. Consistency with earlier research concerns challenges related to the integration of diverse knowledge, disciplinary perspectives and methods [26,28] (Challenge 1). The gap between how TD research is conceptualized and implemented [26] was also identified as a major issue in ROBUST (Challenge 6). One of the challenges identified in the literature [31] was introducing reflexivity in TD collaboration, and our experience in ROBUST is in line with that (Challenge 2 and Challenge 3).

All six challenges listed above require continuous attention from the project coordination team. In order to motivate all partners to genuinely participate in monitoring and reflexive activities, a stepwise approach was elaborated. Most importantly, we explained better how precisely the project is complex, why a reflexive approach is needed and why co-learning is important. Wherever possible, we referred to relevant literature and experiences. For example, it was helpful to refer to previous research projects that had ‘overlooked’ or ‘neglected’ co-learning processes and a reflexive approach and as a result, performed worse than they could have. We also revised the baseline survey and implemented it in a way that was more inviting and less time-consuming. This was achieved through a significant reduction of the number of questions to the most essential, turning most of them into multiple choice or Likert scale questions and running it via a convenient online survey tool. Overall, the baseline survey allowed each LL team to reflect on how their collaboration is going and what needs to be improved. The project coordination team was in turn able to see a bigger picture of how well TD collaboration in the project is functioning.

5. Conclusions

Given the growing emphasis now placed on TD research methods that deliver sustainability solutions to many of the wicked problems we now face, we discuss in this article why it is important to include reflexive components in this type of research. We ascertained that while the TD research approach and TD research projects are well-intended, thorough and well-structured frameworks to enable reflexivity and show tangible improvements, especially for soft outcomes, are needed.

In this concluding section, we will first briefly reflect on the four dimensions and key issues (and criteria) that have been identified as elemental in TD collaboration. This will be followed by some thoughts on a wider application of the framework.

5.1. Importance of the Four Dimensions and Criteria for Fostering TD Collaboration

In this article, we synthesize and enhance earlier meta-level research and reviews, for example [9,26,38,39,43,48] and many other research-based articles. We are presenting the results of our review in the form of a TD co-learning framework which comprises four dimensions with 44 criteria. We think that the framework will support the monitoring of TD processes and therefore will ultimately enhance TD collaboration and (co-)learning.

The results of a first application of the approach in the ROBUST project shows that the framework covers the key facets of TD collaboration and that all four broad dimensions matter for this. The four dimensions allowed obtaining more complete insights into how well the TD teams are functioning. The key issues in TD collaboration were

• The different disciplinary backgrounds of the project partners, their level of engagement and the perceived relevance of the problems they are jointly addressing (context).
• The perceived usefulness of complementary types of knowledge in addressing these problems, and, closely related, attitudes towards the reflexive methods applied (approach).
• The nature and quality of collaboration in terms of reconciling different interests in teamwork, the functioning of knowledge exchange and co-learning, levels of motivation and mutual appreciation, leadership patterns, team-level management and decision-making (process).
• The progress being made in relation to the aims of the team and project, including whether the aims are still achievable; things that can be improved; and personal benefits so far received—as one of the main motivation factors to stay engaged (outcomes).

5.2. Towards a Wider Application of the Framework

TD collaboration potentially entails an enormous amount of learning among all actors involved across the science-policy-practice interface. The monitoring and reflexive components of the framework can capture this learning. We therefore expect that the framework will be of a much broader relevance and will assist actors involved in the planning and implementation of TD research projects. We also believe that the framework will be valuable for any type of multi-actor, interactive, innovation, transformation and action-oriented research project. We think that the framework can most effectively be used in formative evaluation, including in a participatory manner.

The main strengths of the framework can be summarized as conceptual coherence, flexibility and functionality. The same qualities were also recognized as important for evaluating sustainability projects [115,116]. An important characteristic is that the TD co-learning framework is adaptable to the needs of those who will use it. The questions indicated in the framework are by no means prescriptive. Instead, they are meant to accommodate different needs in a project, for example when designing a survey, structuring an interview, facilitating a team discussion and/or joint reflection exercise, creating a reporting template and so forth.

We identified two main groups of actors that could benefit from using this approach:

(1)

Multi-actor teams that do not have the time or resources to develop an own framework for monitoring and reflexive activities. They can cherry-pick what is needed in their situation depending on project goals, scale, actors involved and so forth from the wider list of criteria. To better understand the meaning of each specific indicator for project work, the related practical questions are juxtaposed. Some questions are suitable for designing a survey with more focus on quantitative data (Likert Scale, Multiple Choice). Other questions can be used for a targeted focus groups or workshops to trigger and guide/facilitate participatory reflection.

(2)

Teams which elaborated a framework but seek to widen, adjust or improve it for their specific context and teams which need immediate practical solutions. The framework can be applied in a wide range of sectors. Applications may include tracking stakeholder engagement or satisfaction surveys. In this case, the list of criteria and questions is meant to inspire new ways of thinking or working.

The next step for us will be to use the framework to derive practical lessons learned from other TD research projects and initiatives in the area of agri-food, sustainable rural development and the knowledge-based bioeconomy.