A Conceptual Framework for the Evaluation of Social Agriculture: An Application to a Project Aimed at the Employability of Young People NEET

Round 1

Reviewer 1 Report

The authors aim to propose an evaluation methodology for social agriculture, and this work is crucial to evaluate the impact and outcomes generated by a specific initiative especially for the involved actors. Therefore, I fully acknowledge the value of publishing this paper. However, some of the following points need to be improved before this paper can be accepted.

1)The title is a bit confusing. As it is unclear whether this conceptual framework for the evaluation applies to social agriculture in general or only to specific social agriculture initiative to promote the employability of young people NEET.

2)In the introduction section, the authors should clearly state what are the main and important challenges need to be solved in the research on the evaluation of social agriculture. Then, the authors need to explain how they will attempt to solve these challenges.

3)The authors showed the evaluation results in Table 4, listed 13 indicators as “Highly performant”, 12 indicators are “Medium” and 11 indicators are “low”, without any explanation of them.  However, it is important to describe the mechanism that causes these differences to understand the effects of social agriculture to promote the employability of young people NEET.

4)It is suggested that the authors need to add a note to briefly describe the evaluation manual produced by the SIMRA project as most of the international readers are difficult to understand it.

Author Response

The authors aim to propose an evaluation methodology for social agriculture, and this work is crucial to evaluate the impact and outcomes generated by a specific initiative especially for the involved actors. Therefore, I fully acknowledge the value of publishing this paper. However, some of the following points need to be improved before this paper can be accepted.

R. We thank the reviewer for the words in appreciation of our work. We made our best efforts to improve our paper, according to the valuable suggestions you and the other reviewers forwarded to us.

The title is a bit confusing. As it is unclear whether this conceptual framework for the evaluation applies to social agriculture in general or only to specific social agriculture initiatives to promote the employability of young people NEET.

R. The paper proposes a conceptual framework meant to perform an evaluation of social agriculture, in terms of achievement of specific political objectives. In our case, we evaluated a project aimed at NEETs, and we wanted to check if the outcomes of the project were consistent with the general policy objectives of social agriculture. 

Accordingly we changed the title as follows:

A Conceptual Framework for the Evaluation of Social Agriculture: an application to a project aimed at the Employability of Young People NEET

In the introduction section, the authors should clearly state what are the main and important challenges need to be solved in the research on the evaluation of social agriculture. Then, the authors need to explain how they will attempt to solve these challenges.

R. In the introduction section, we clarified that SI initiatives need policy support to favor the accumulation of immaterial capital (i.e. social capital). However, we already know that SI creates multiple outcomes, some of which may not be accounted for the initial purpose of the policy. Therefore the challenge of our paper is to disentangle the multiple outcomes, in order to enable policy makers to focus only on outcomes that are compliant with the initial purpose of the policy.

We specified in the text that we created a selection grid for SI indicators, by crossing two variables: the policy objectives of social agriculture, and the selection criteria for projects eligible for public funding. 

In practice, Table 1 was used to pick-up the most suitable composite indicators to address the problem of accounting only the outcomes of SI initiatives, in terms of social agriculture objectives. (see lines 48-67 and 74-91)

The authors showed the evaluation results in Table 4, listed 13 indicators as “Highly performant”, 12 indicators are “Medium”and 11 indicators are “low”, without any explanation of them. However, it is important to describe the mechanism that causes these differences to understand the effects of social agriculture promote the employability of young people NEET.

R. We thank you very much for the valuable comment. In fact, we realized that the rough presentation of results of the first version of the paper was unsatisfactory and was not adequate to respond to the purpose of the study.

For this reason, we expanded section 4.2, by adding a synthetic comment for each composite indicator. We tried to provide an interpretation of results, according to the elements of SI listed in Table 2:

- Trigger

- social needs

- context

- agency

- preparatory actions

(see lines 377-434)

In addition, we created Table 4, to focus on the project performances in terms of the 4 social agriculture objectives specified in the methodology section (column headers of Table 1). (see lines 453-475)

 

It is suggested that the authors need to add a note to briefly describe the evaluation manual produced by the SIMRA project as most of the international readers are difficult to understand it.

R. We introduced a brief introduction of SIMRA evaluation manual, explaining that it provides the analytical framework, the set of questions (tools) and the indicators which are suitable to analyze all aspects of SI. (see lines 155-173)

 

 

 

Author Response File: Author Response.pdf

Reviewer 2 Report

The manuscript focuses on a relatively current and interesting topic related to the possible involvement of young people (NEET) in the concept of social agriculture. In the abstract and in the manuscript, however, I lack the broader scientific paradigm to which this topic follows, ie the necessary identification of a broader theoretical framework in which this concept is gradually being formed.

Partial comments:
================
- in the abstract there are no clear numbers concerning the indicators that the authors evaluated, ie 26 indicators in total, however, in a partial breakdown they discuss 13 positive + 12 moderate + 11 low indicators.

- the introduction section is followed by section 2 "Evaluation of SI initiatives", however it is necessary to add a separate theoretical chapter in which the paradigmatic directions concerning the evaluation of social innovation and social agriculture in the context of sustainable regional and local development would be discussed in more depth.
The role of NEETs in the theoretical concepts of the topic is not reflected at all.

- Table 1 is not easy to read - the text is not aligned correctly. In addition, I generally lack logic in the clarity of this table, which is not clear. E.g. criterion "Capacities" appears 2 times, but with different assignment to objectives - IV) Capacities + XII) Capacities. It is also not clear why some text is highlighted in bold and others in italics.

- Table 2 - also not understandable. Definitions of indicators are not clear how they are actually conceived, measured, data sources, etc. References to the SIMRA manual are not appropriate in the paper, because it is not possible for the manuscript to contain ambiguities with references to other documents in the methodological approach. In addition, I recommend putting the whole table 2 in the appendix and specifying it more in connection with the methodological concept and the realized empirical research.

- Empirical application - I definitely recommend structuring the text more so that there is a clear key message in the individual topics of the manuscript. The text is not deeply conceived, there is no interpretation of partial results.

- Conclusion section - it is necessary to redo the overall, because it is only a very weak summary, not a comprehensive synthesis of knowledge in relation to the theoretical background, key results (structured presented) and defining the main contribution of the manuscript in the context of knowledge gap. There is also a lack of a deeper analysis of the link between the methodology and NEETs.

Author Response

The manuscript focuses on a relatively current and interesting topic related to the possible involvement of young people(NEET) in the concept of social agriculture. In the abstract and in the manuscript, however, I lack the broader scientific paradigm to which this topic follows, i.e. the necessary identification of a broader theoretical framework in which this concept is gradually being formed.

R. We are grateful for the valuable suggestions posed by the reviewer, which we carefully considered to improve our paper.

 Regarding the first comment, we would like to clarify that the aim of our paper was focused on the evaluation of an initiative aimed at favoring NEETs employability in agriculture, within the frame of policy support for social agriculture. 

 In the context of Southern Italy, it is perceived (perceived context) that agriculture is the most relevant sector with highest employability, due to the fact that farmers face enormous difficulties in finding young and professional workers for highly specialized jobs. In specific terms, organic farming is particularly emerging in this area and, therefore, the project aimed at bridging the gap between NEETs and farms. (please, see lines 73-81)

As remarked by another reviewer, the original title of the paper was misleading, and we changed to make it more clear. (see the title)

 We want to demonstrate that the project was consistent with the concept of social agriculture. To this aim, the performance of the project was evaluated with a set of indicators which were referred to the national policy objective of social agriculture.

In fact, policy makers may face some difficulty in understanding whether a project addressed at NEETs may or may not be eligible for public funding within the frame of agricultural policy budget. In our paper, we wanted to demonstrate that a project based on Social Innovation initiative, involving multiple actors, may be consistent with agricultural policy, since: a) it provides training for young NEETs, b) after training, the NEETs may be employed in agriculture, c) the overall project is not focused on NEETs (there is a limited number of NEETs in rural areas), but their inclusion is necessary to promote new types of multi-actor collaborations and to promote organic products at local level.

Partial comments:

================

[II.2]- in the abstract there are no clear numbers concerning the indicators that the authors evaluated, ie 26 indicators in total,however, in a partial breakdown they discuss 13 positive + 12moderate + 11 low indicators.

R. We are grateful for the comment. We revised the paper and we found some duplicates. The total number of indicators is 34. The abstract has extensively been revised.

 

- the introduction section is followed by section 2 "Evaluation of SI initiatives", however it is necessary to add a separate theoretical chapter in which the paradigmatic directions concerning the evaluation of social innovation and social agriculture in the context of sustainable regional and local development would be discussed in more depth.

The role of NEETs in the theoretical concepts of the topic is not reflected at all

R. As already previously mentioned, the aim of our paper is to propose a methodology to adopt the evaluation indicators developed for Social Innovation initiatives. We believe that there are several projects of social agriculture, which have the characteristics of Social Innovation. 

The case we presented, referred to NEETs employability in agriculture, is a particular case where we can adopt the SIMRA evaluation method to disentangle the multiple outcomes of the project.

This evaluation exercise is useful for policy makers who are responsible for project funding, and to be able to check if they are consistent with the objectives of social agriculture.

We emphasized this concept in rows (see lines 55-67)

Regarding your recommendation about introducing a theoretical chapter, we clarified that the purpose of our work is to adapt the evaluation framework already developed by the SIMRA manual for Social Innovation (we added a note in the introduction, lines 155-173).

We consider the initiative of NEETs project, as a particular case of social agriculture.

 

Table 1 is not easy to read - the text is not aligned correctly. In addition, I generally lack logic in the clarity of this table, which is not clear. E.g. criterion "Capacities" appears 2 times, but with different assignment to objectives - IV) Capacities + XII)Capacities. It is also not clear why some text is highlighted in bold and others in italics.

R. We revised Table 1 and we eliminated the duplicates of composite indicators. 

The table is a selection grid aimed at selecting the composite indicators which are used later on.

- Table 2 - also not understandable. Definitions of indicators are not clear how they are actually conceived, measured, datasources, etc. References to the SIMRA manual are not appropriate in the paper, because it is not possible for the manuscript to contain ambiguities with references to other documents in the methodological approach. 

In addition, I recommend putting the whole table 2 in the appendix and specifying it more in connection with the methodological concept and the realized empirical research.

R. We converted Table 2 into Box 1. We revised the description of indicators, by adding the direct question which was made to the respondent. In the description we adopted the same labels of composite indicator and indicator which were used by the SIMRA evaluation manual (mentioned many times in the text), and we also provided the page reference for each composite indicator. 

The connection between Box 1 (the former Table2) and the methodology, is stated in lines 274-277.

- Empirical application - I definitely recommend structuring the text more so that there is a clear key message in the individual topics of the manuscript. The text is not deeply conceived, there is no interpretation of partial results.

R. We provided a brief description of the results of the evaluation.
We also added a final table, to summarize the evaluation of performance, in terms of social agriculture policy objectives (see lines 377-474)

 

- Conclusion section - it is necessary to redo the overall,because it is only a very weak summary, not a comprehensive synthesis of knowledge in relation to the theoretical background,key results (structured presented) and defining the main contribution of the manuscript in the context of knowledge gap.There is also a lack of a deeper analysis of the link between the methodology and NEETs.

R. The conclusion session has been completely revised. A clear connection with the aim of the research has been established, as well as the policy implications.

 

 

Author Response File: Author Response.pdf

Reviewer 3 Report

Dear Authors,

I thank you for sharing this interesting paper. I think that the evaluation manual you have developed within the SIMRA project is very valuable. The paper is well structured and to the point.

I have some recommendations that I think are necessary to reach a good quality of your paper:

• I found the title misleading after reading the content. The paper is a rather descriptive case study that exemplifies the use of the evaluation exercise developed by the SIMRA project for social innovation. Of which social agriculture projects can be an example. In my oppinion you do not discuss the link between social agriculutre and employability of Young people NEET. I suggest you change the title, so that it covers the content in a more consistent way.
• I miss a definition of “social innovation”
• The word “either” seems not correctly used? (instead use ‘both’? (line 68 and line 74)
• Your conclusion is not sufficiently in depth. I miss a discussion about the valuable lessons learned by carrying out this evaluation exercise. What are the strengths and weaknesses of the evaluation exercise based on this study?

• • You talk about difficulties to perform an evaluation exercise at an early stage (L368). At the same time in line 65 you indicate this was an specific choice. What does this mean for the use of the evaluation exercise in the future? Minimum requirements (min. length of project before you do an evaluation, different types of uses/goals of the evaluation exercise (eg in early stage it’s a pre-assesment for promotors of project, in x-stage as measure of success..?)
  • I would also be interested to know if the results of the evaluation exercise were appreciated by the interviewees? Is this information perceived as sufficient by involved policy makers and evaluation agencies to make a decisions to support the initiative? (as you refer to in line 80).

 

Author Response

Dear Authors,

I thank you for sharing this interesting paper. I think that the evaluation manual you have developed within the SIMRA project is very valuable. The paper is well structured and to the point.

I have some recommendations that I think are necessary to reach a good quality of your paper:

R. We thank the reviewer for the words in appreciation of our work. We made our best efforts to improve our paper, according to the valuable suggestions you and the other reviewers forwarded to us.

 

I found the title misleading after reading the content.

The paper is a rather descriptive case study that exemplifies the use of the evaluation exercise developed by the SIMRA project for social innovation. Of which social agriculture projects can be an example. In my opinion you do not discuss the link between social agriculture and employability of Young people NEET. I Suggest you change the title, so that it covers the content in a more consistent way

 

R. The paper proposes a conceptual framework meant to perform an evaluation of social agriculture, in terms of achievement of specific political objectives. In our case, we evaluated a project aimed at NEETs, and we wanted to check if the outcomes of the project were consistent with the general policy objectives of social agriculture. 

Accordingly we changed the title as follows:
A Conceptual Framework for the Evaluation of Social Agriculture: an application to a project aimed at the Employability of Young People NEET

 I miss a definition of “social innovation”

We reported the definition adopted by the SIMRA manual  (see lines 160-162)

The word “either” seems not correctly used? (instead use ‘both’?(line 68 and line 74)

R. We changed according to the suggestion.

 

Your conclusion is not sufficiently in depth. I miss a discussion about the valuable lessons learned by carrying out this evaluation exercise. What are the strengths and weaknesses of the evaluation exercise based on this study?

R. The conclusion session has been completely revised. A clear connection with the aim of the research has been established, as well as strengths and weaknesses.

You talk about difficulties to perform an evaluation exercise at an early stage (L368). At the same time in line 65 you indicate this was a specific choice. What does this mean for the use of the evaluation exercise in the future? Minimum requirements (min.length of project before you do an evaluation, different types of uses/goals of the evaluation exercise (eg in early stage it’s a pre-assessment for promoters of project, in x-stage as measure of success..?)

R. The usefulness of performing an evaluation at an early stage of development was mentioned in lines 96-99 and 128-142.

I would also be interested to know if the results of the evaluation exercise were appreciated by the interviewees? Is this information perceived as sufficient by involved policy makers and evaluation agencies to make a decision to support the initiative? (as you refer to in line 80).

R. Unfortunately, the project has been delayed by the Covid-19 pandemic crises, and we missed the opportunity to maintain a close contact with the project management and the policy makers. We hope to re-establish a close collaboration in the near future.

 

 

 

 

 

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Congratulations. The modifications made are very extensive and have helped to improve the quality of the manuscript. I have no further comments.