Sustainability and Quality of Life in Marginalized Areas: An Impact Evaluation of a Community Center in Santa Fe, Mexico

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The authors have addressed the important topic of imapct evaluation of social programmes. This is a valuable voice, also in terms of the spread of this methodology. They refer mainly to examples from South America Asia, but examples of impact evaluation can also be found in Europe, these evaluations do not always result in a scientific article, much more often simply in an application report, but in this case they also have great research value.

I evaluate the article generally good, but I have some important comments:

- I miss a deeper commentary on the quality of life indexes and its three dimensions. The way the indices are constructed is described in detail. What I miss, however, is a reflection on why exactly these areas were chosen. There are many more dimensions of quality of life in the literature. Eurostat, for example, indicates nine dimensions. Overall, there is a lack of reflection on the complexity of quality of life. I understand that this is not the main topic, but the quality of quality of life measurement has a key impact on impact evaluation results.

- There is no annex with the questionnaire that respondents answered. We know nothing about the questions in the survey. In addition, there is no analysis of the reliability of the set of indicators used to determine the indices for each area, not least the Cronbach's Alpha.

- there is no table with the distribution by socio-demographic characteristics of the experimental and counterfactual group, there is only a very general description.

- measures of goodness of fit  for the regression models are missing.

- the discussion takes the form of a summary of the main results. What is missing is a broader commentary on what emerges from these results, what are the implications for further work. How these results relate to other studies, not necessarily in the context of other impact evaluation studies. I am particularly concerned here with references to analyses of marginalised areas of cities. This is a big research question, not addressed at all in this article.

Author Response

I evaluate the article generally good, but I have some important comments:

Comment 1. - I miss a deeper commentary on the quality of life indexes and its three dimensions. The way the indices are constructed is described in detail. What I miss, however, is a reflection on why exactly these areas were chosen. There are many more dimensions of quality of life in the literature. Eurostat, for example, indicates nine dimensions. Overall, there is a lack of reflection on the complexity of quality of life. I understand that this is not the main topic, but the quality of quality of life measurement has a key impact on impact evaluation results.

Response 1: A reflection on the complexity of quality of life and a description of the selected areas were added.

Lines 108-118 (Section 1: Introduction)

Lines 370-409 (Section 4.2)

Lines 706-714 (Section 6: Conclusions)

Comment 2. - There is no annex with the questionnaire that respondents answered. We know nothing about the questions in the survey. In addition, there is no analysis of the reliability of the set of indicators used to determine the indices for each area, not least the Cronbach's Alpha.

Response 2: Annex 1 was added with the questionnaire and the items for the selected variables. The questionnaire is also added in a separate archive for its extension.

Thanks to this comment, Cronbach's Alpha was applied to each index. As a result of this review, the model was strengthened since the variables initially chosen for Capacity and Skills Development and Technology were not appropriate and showed little or no reliability. In the case of the health index, no changes were made since the initial Cronbach's Alpha was 0.7892575. For the Capacity and Skills Development index, other items were selected, resulting in a Cronbach's Alpha of 0.74684558, and for the case of technology access, more variables were included, resulting in a Cronbach's Alpha of 0.60667394. The change of items in technological development and the addition of items in digital access strengthened the model, yielding positive results, but different from what was initially obtained. With this improvement, the impact index increased, and the R-squared value improved. The changes made with data from the improved model are highlighted in yellow.

Lines 443-456 (Section 4.2)

Lines 591-596, 613-615, 629, 632-633, 636-637 (Section 5)

Line 671 (Section 6: Conclusions)

Comment 3. - there is no table with the distribution by socio-demographic characteristics of the experimental and counterfactual group, there is only a very general description.

Response 3: The context of the Pueblo de Santa Fe, the characteristics of the Community Center, and the sociodemographic characteristics of the inhabitants of the Pueblo de Santa Fe have been included, covering both beneficiaries and non-beneficiaries of the community center.

Lines192-283 (Section 3)

Comment 4. - measures of goodness of fit  for the regression models are missing.

Response 4: The R-squared value has been included in the models. It is worth mentioning that although it is low, the models in this study do not aim to predict future data. An evaluation was conducted to determine which model fits best using machine learning, and the results show that linear regression has as good a fit as Gradient Boosting, Random Forest, Neural Network, and others. The errors for these models show minimal differences. Additionally, in the case of models with a single discrete independent variable, the standard methodology used is linear regression.

Lines 586-596, 598-604, 629

Comment 5. - the discussion takes the form of a summary of the main results. What is missing is a broader commentary on what emerges from these results, what are the implications for further work. How these results relate to other studies, not necessarily in the context of other impact evaluation studies. I am particularly concerned here with references to analyses of marginalised areas of cities. This is a big research question, not addressed at all in this article.

Response 5: A section on marginalized areas has been added, explaining how the Pueblo de Santa Fe is considered part of these areas.

Lines 148-191 (Section 2)

Lines 817-838 (References)

The introduction and conclusions have been updated to include the implications for future work and the relevance of the study's replicability.

Líneas 119-147 (Section 1: Introduction)

Lines 706-720 (Section 6: Conclusions)

Author Response File: Author Response.docx

Reviewer 2 Report

Comments and Suggestions for Authors

Dear authors,

Your manuscript addresses a very important challenge of today’s urban development. The effectiveness of non-profit social programs and the research on impact evaluation in general has been a recurrent topic and a priority issue during the recent decades, explicitly in the context of achieving the sustainable development goals (SDG) 2030. Your research collaboration with the institutions involved in the practical implementation of the social programme could be considered crucial for understanding the motivation and the outcomes of the interventions.

Regretfully the manuscript provides no clarity about the particular context within which the community centre in question operates; no data is presented about socio-demographic and cultural characteristics of the community served and the ongoing transformations in the area. You claim that the current study is aimed at empirical and methodological contribution in the case of a first and only private university applying the impact evaluation methodology to its own social programmes (lines 73-75) to provide feedback on the applicability of the method in other similar situations. It is however, not clear from the manuscript why it is important in terms of research that it is a private university implementing such an evaluation and how the methodology is transferrable to other cases.

Studying the context was mentioned as an activity undertaken, which is important as it is stated later that “in the case of this research both groups share the same sociodemographic and cultural profiles” (lines 152-153)”. Yet, no data is further provided about how the undertaken “reconnaissance of the area” (lines 122-123) has enabled an understanding about the context in which the CCMUP operates and has influenced the contents and structure of the survey questionnaires. Building the baseline is a relevant step, however only vaguely described. It would be also important for understanding the context to explain who the “people owning a membership” are (lines 144-150) and how membership is obtained.

The study period is reported to cover the years 2022 to 2024. It is, however further specified in the text that “the improvements to the CCMUP were applied during the second half of 2022 and until March 2024 (lines 264-265); the time interval covered by the study needs to be more precisely stated when presenting the methodology. The period covers even less than two years and seems too brief to estimate the real effectiveness of the functioning of the community centre. The survey questions are not disclosed, which makes it impossible to estimate their relevance to the aims of the study.

The impact evaluation results concerning all the three studied dimensions are reported as “positive” and indicating “general improvement” in the area (370-373). On this basis you rightly claim that the analyzed intervention has been “successful”. It is, however, difficult to agree that such a result would be enough for estimating the real-life effect of the effort and the resources invested in the activities.  A comprehensive impact evaluation of the functioning of the centre should be able to provide a comparison between expected and achieved results, and to serve as a basis for recommendations on its further improvement.

I would encourage you to further conceptualize your research results in the frame of the phenomenon of marginalized urban areas, to discuss in-depth the ways to evaluate the impact of undertaken interventions and to formulate a sound message on the possibilities for going ahead to address and diminish inequality gaps in contemporary cities.

Comments on the Quality of English Language

The language could be improved to more precisely present authors' claims and messages.

Author Response

Comment 1. Regretfully the manuscript provides no clarity about the particular context within which the community centre in question operates; no data is presented about socio-demographic and cultural characteristics of the community served and the ongoing transformations in the area. You claim that the current study is aimed at empirical and methodological contribution in the case of a first and only private university applying the impact evaluation methodology to its own social programmes (lines 73-75) to provide feedback on the applicability of the method in other similar situations. It is however, not clear from the manuscript why it is important in terms of research that it is a private university implementing such an evaluation and how the methodology is transferrable to other cases.

Comment 1.1: Particular context within which the community centre in question operates; no data is presented about socio-demographic and cultural characteristics of the community served and the ongoing transformations in the area.

Response 1.1: The context of the Pueblo de Santa Fe, the characteristics of the Community Center, and the sociodemographic characteristics of the inhabitants of the Pueblo de Santa Fe have been included, covering both beneficiaries and non-beneficiaries of the community center.

Lines192-283 (Section 3)

Comment 1.2: why it is important in terms of research that it is a private university implementing such an evaluation and how the methodology is transferrable to other cases.

Response 1.2: The relevance of impact evaluation of social interventions in private universities has been included.

Line 141-147 (Section 1: Introduction)

Lines 207-219 (Section 3)

Lines 706-720 (Section 6: Conclusions)

Comment 2. Studying the context was mentioned as an activity undertaken, which is important as it is stated later that “in the case of this research both groups share the same sociodemographic and cultural profiles” (lines 152-153)”. Yet, no data is further provided about how the undertaken “reconnaissance of the area” (lines 122-123) has enabled an understanding about the context in which the CCMUP operates and has influenced the contents and structure of the survey questionnaires. Building the baseline is a relevant step, however only vaguely described. It would be also important for understanding the context to explain who the “people owning a membership” are (lines 144-150) and how membership is obtained.

Comment 2.1: no data is further provided about how the undertaken “reconnaissance of the area” (lines 122-123) has enabled an understanding about the context in which the CCMUP operates and has influenced the contents and structure of the survey questionnaires.

Response 2.1: The context in which CCMUP operates and the general characteristics of the population that provides information for the baselines was included.

Lines 193-219, 243-252 (Section 3)

Comment 2.2: Building the baseline is a relevant step, however only vaguely described.

Response 2.2: The description of the baseline has been included.

Lines 410-427 (Section 4.2)

Comment 3. The study period is reported to cover the years 2022 to 2024. It is, however further specified in the text that “the improvements to the CCMUP were applied during the second half of 2022 and until March 2024 (lines 264-265); the time interval covered by the study needs to be more precisely stated when presenting the methodology. The period covers even less than two years and seems too brief to estimate the real effectiveness of the functioning of the community centre. The survey questions are not disclosed, which makes it impossible to estimate their relevance to the aims of the study.

Response 3: The explanation of the brief period of study was included.

Lines: 102-107

Comment 4. The impact evaluation results concerning all the three studied dimensions are reported as “positive” and indicating “general improvement” in the area (370-373). On this basis you rightly claim that the analyzed intervention has been “successful”. It is, however, difficult to agree that such a result would be enough for estimating the real-life effect of the effort and the resources invested in the activities.  A comprehensive impact evaluation of the functioning of the centre should be able to provide a comparison between expected and achieved results, and to serve as a basis for recommendations on its further improvement.

Response 4. The context in which the community center operates and its functioning, as well as the importance of its presence in the Pueblo de Santa Fe, have been included.

Lines 193-219 (Section 3)

Lines 222-239 (Section 3.1)

Comment 5. I would encourage you to further conceptualize your research results in the frame of the phenomenon of marginalized urban areas, to discuss in-depth the ways to evaluate the impact of undertaken interventions and to formulate a sound message on the possibilities for going ahead to address and diminish inequality gaps in contemporary cities.

Response 5: A section on marginalized areas has been added, explaining how the Pueblo de Santa Fe is considered part of these areas.

Lines 148-191 (Section 2)

Lines 817-838 (References)

The introduction and conclusions have been updated to include the implications for future work and the relevance of the study's replicability.

Líneas 119-147 (Section 1: Introduction)

Lines 706-720 (Section 6: Conclusions)

Observations: The results of the model are different because improvements were made based on the observations provided by one of the reviewers. Annex 1 was added with the questionnaire and the items for the selected variables. The questionnaire is also added in a separate archive for its extension.

Author Response File: Author Response.docx

Reviewer 3 Report

Comments and Suggestions for Authors

Dear Editor and authors, thank you for inviting me to review this work.

The quality of this research is unquestionable, and although it employs a method that is not as widespread as others commonly used in research, the authors have taken care to perfectly justify its use as well as to compare their results with similar research that has employed this methodology.

I can only congratulate the authors, and I value their good work, which is directly linked to the social work carried out at the Community Center MAPFRE-UP in Santa Fe Mexico.

My only recommendation to the authors is that they include a little more information about the questionnaires that formed the index, and about the results of their research in the abstract.

Author Response

Comment. The quality of this research is unquestionable, and although it employs a method that is not as widespread as others commonly used in research, the authors have taken care to perfectly justify its use as well as to compare their results with similar research that has employed this methodology. I can only congratulate the authors, and I value their good work, which is directly linked to the social work carried out at the Community Center MAPFRE-UP in Santa Fe Mexico. My only recommendation to the authors is that they include a little more information about the questionnaires that formed the index, and about the results of their research in the abstract.

Response: Annex 1 was added with the questionnaire and the items for the selected variables.

Observations: The results of the model are different because improvements were made based on the observations provided by one of the reviewers. Annex 1 was added with the questionnaire and the items for the selected variables. The questionnaire is also added in a separate archive for its extension.

Author Response File: Author Response.docx

Reviewer 4 Report

Comments and Suggestions for Authors

You can improve the article, if you extend the literature review and present it in a separate part. Is there results of similar researches? You present examples of using the selected method, but what about results and conclusions. Are they confirm your results or not? Give examples, as you point out in Discussion (Line 404). The reader wonders why calculation of the Index (formula 4) and of the DD Index (Formula 5) are the same. I suggest that the name of the third chapter should be Results and Discussion, and that of the forth chapter – Conclusions. Check the text again. There is small technical mistakes (Line 10 - the square bracket, Line 71 - the dash, Line 203 - the dash, Line 252 - the dash, Line 253 - the dash, Line 285 - the dash, Line 291 - the dash, Line 292 - the dash, Line 300 - the dash, Line 301 - the dash, Line 405 - the dash). Keywords: Differences on Differences or Differences in Differences.

Author Response

Comment 1. You can improve the article, if you extend the literature review and present it in a separate part. Is there results of similar researches? You present examples of using the selected method, but what about results and conclusions. Are they confirm your results or not? Give examples, as you point out in Discussion (Line 404).

Response 1: A section on marginalized areas has been added, explaining how the Pueblo de Santa Fe is considered part of these areas.

Lines 148-191 (Section 2)

Lines 816-838 (References)

The introduction and conclusions have been updated to include the implications for future work and the relevance of the study's replicability.

Líneas 119-147 (Section 1: Introduction)

Lines 706-714 (Section 6: Conclusions)

Comment 2. The reader wonders why calculation of the Index (formula 4) and of the DD Index (Formula 5) are the same.

Response 2: Formula 4 describes the relationship between the simple impact index as the dependent variable and membership or non-membership in the community center as the independent variable, while Formula 5 measures the relationship between the difference in the impact index from 2022 to 2024 as the dependent variable, keeping the independent variable constant. What distinguishes one formula from the other is that one represents the simple index and the other represents the difference between indices. Formula 5 eliminates any potential bias due to the passage of time.

Comment 3. I suggest that the name of the third chapter should be Results and Discussion, and that of the forth chapter – Conclusions.

Response 3. Done.

Line 582

Line 662

Comment 4. Check the text again. There is small technical mistakes (Line 10 - the square bracket, Line 71 - the dash, Line 203 - the dash, Line 252 - the dash, Line 253 - the dash, Line 285 - the dash, Line 291 - the dash, Line 292 - the dash, Line 300 - the dash, Line 301 - the dash, Line 405 - the dash). Keywords: Differences on Differences or Differences in Differences.

Response 4. All the marked sections were revised and corrected.

Observations: The results of the model are different because improvements were made based on the observations provided by one of the reviewers. Annex 1 was added with the questionnaire and the items for the selected variables. The questionnaire has also been added to a separate archive for its extension.

 

 

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

I have no further comments.