EASIER: An Evaluation Model for Public–Private Partnerships Contributing to the Sustainable Development Goals

Round  1

Reviewer 1 Report

The paper addresses an interesting and very important research area, investigating how public-private partnerships contribute to sustainable development goals. The authors draw on some prior studies, but a much more critical literature analysis is needed to strengthen the paper’s argument and draw out the (theory) gaps they seek to address. Also, the paper needs to be present much stronger discussion and conclusion sections in order to offer value to the reader. Overall, the manuscript makes some very interesting points and I realize that a lot of work went into this study. Nevertheless, I see room for improvement which will help to enhance clarity, readability, practical and theoretical contributions. The following paragraphs address each section of the paper in more detail and provide suggestions on how to revise the paper.

Major concerns:

Introduction:

While the author(s) establish some links to some extant literature, author(s) need to establish a more coherent framework for the overall paper. That means, the introduction should clearly indicate the need for this paper in relation to extant research studies. The authors do a good job to explain why the paper is relevant for practice, but miss out to clearly draw out the gap(s) they seek to address with regards to extant studies. The authors have established a clear argument with regards to gaps for policy and practice, but further more detailed insights with regards to gaps in extant literature is vital to draw out. Here, the authors need to more clearly link PPPs and key extant studies (e.g. James Barlow, Nigel Caldwell, Ilze Kivleniece) to sustainable development and concepts of sustainability (e.g. Stefan Hoejmose, Johanne Grosvold, Helen Walker; please see some suggestions for further studies in the reference list at the end of my comments).

Conceptual background & Theoretical development:

The authors need to establish some clearer links to extant PPP literature (e.g. please see suggested key references) to guide the reader. This would help to clearly establish the basis of this paper, before delving deeper into social responsibility. The author(s) should clearly draw out the benefits and limitations of PPPs and issues around the management of such long-term relationships to drive social responsibility. This should then be linked to discussions around value for money, risks and particularly sustainable development as key drivers. A much clearer positioning of the current study will help to further guide the reader and draw out gap(s) in extant studies.  This would then help to set up the current conceptual model the authors position.

Methods and Analysis:

Overall, this section is well written, but I see some room for improvement to obtain robust results and strengthen the paper’s argument. The authors should address the following concerns:

# Please clearly tell the reader about your sampling logic with regards to respondents, the PPP projects under investigation and the countries you selected.

# You also need to clearly show the findings/results rather than just ‘tell’ the reader. At the moment, it is impossible for the reader to judge whether you have actually active your research aims/objectives and very limited data are presented. This section needs a complete rework.  

Discussions and Conclusions:

Derived from a conceptual background section which did not clearly draw out the gaps the paper seeks to address, the discussion and conclusion sections do offer very little additional value to the reader as it stands. The authors need to offer more fine-grained results here and discuss what they intended to find out in the introduction section (link to research question; overall aim of the paper). Overall, the authors need to clearly draw out what the theoretical contributions are and how they add to the existing body of knowledge. This section also needs to clear link back to extant studies to offer some clear value to the reader.

Useful references:

Brammer, S. and Walker, H. L. 2011. Sustainable procurement in the public sector: an international comparative study. International Journal of Operations & Production Management 31(4), pp. 452-476.

Roehrich et al. (2014). Are public-private partnerships a healthy option? A systematic literature review. Social Science & Medicine, Vol. 113, pp. 110-119.

Touboulic, A. and Walker, H. L. 2015. Theories in sustainable supply chain management: a structured literature review. International Journal of Physical Distribution & Logistics Management 45(1/2), pp. 16-42.

Zheng, J.; Roehrich, J.K. and Lewis, M.A. (2008). The dynamics of contractual and relational governance: Evidence from long-term public-private procurement arrangements. Journal of Purchasing and Supply Management, Vol. 14 No. 1, pp. 43-54.

Author Response

Point 1: The paper addresses an interesting and very important research area, investigating how public-private partnerships contribute to sustainable development goals. The authors draw on some prior studies, but a much more critical literature analysis is needed to strengthen the paper’s argument and draw out the (theory) gaps they seek to address. Also, the paper needs to be present much stronger discussion and conclusion sections in order to offer value to the reader. Overall, the manuscript makes some very interesting points and I realize that a lot of work went into this study. Nevertheless, I see room for improvement which will help to enhance clarity, readability, practical and theoretical contributions. The following paragraphs address each section of the paper in more detail and provide suggestions on how to revise the paper.

Response 1: We are pleased that you found the topic of our manuscript very important and interesting, and that you think we make relevant points for the Sustainability special issue. Thank you very much for your thoughtful and very constructive comments. We believe that your suggestions, as well as those of other reviewer, have helped us to significantly strengthen the manuscript. We have followed your suggestions in order to enhance the clarity, improve the readability and highlight the practical and theoretical contributions of our manuscript. Below, we provide point-by-point responses to your suggestions. We greatly appreciate your highly detailed feedback, and rest assured that we have done our best to follow each of your recommendations.

Point 2: Introduction: While the author(s) establish some links to some extant literature, author(s) need to establish a more coherent framework for the overall paper. That means, the introduction should clearly indicate the need for this paper in relation to extant research studies. The authors do a good job to explain why the paper is relevant for practice, but miss out to clearly draw out the gap(s) they seek to address with regards to extant studies. The authors have established a clear argument with regards to gaps for policy and practice, but further more detailed insights with regards to gaps in extant literature is vital to draw out. Here, the authors need to more clearly link PPPs and key extant studies (e.g. James Barlow, Nigel Caldwell, Ilze Kivleniece) to sustainable development and concepts of sustainability (e.g. Stefan Hoejmose, Johanne Grosvold, Helen Walker; please see some suggestions for further studies in the reference list at the end of my comments).

Response 2: Thank you for these recommendations. We regret that our prior version of the manuscript was unclear regarding how our work connected with prior literature. In this new version, we have incorporated your suggested references to identify more clearly the gap our work is tapping, as well as how it contributes not only to practice and policy, but also to academia. Specifically, we followed your recommendation and now indicate in the introduction what existing studies have found in regards to advantages and limitations of PPPs (Barlow, Roehrich and Wright (2013) and how prior studies have explored mechanisms to minimize them (Zheng, Roehrich and Lewis (2008)). In addition, we included articles studying governmental contracts linked to issues of sustainability, such as social and environmental criteria (e.g., Brammer and Walker, 2011), among others. Together, they offer a more solid conceptual footing for our paper and a clearer gap identification as they allowed us to 1) highlight the potentialities of PPP in promoting societal progress, 2) acknowledge the need for simpler evaluation models for PPPs, and 3) respond to the call for more conceptual work. We hope that these changes meet your expectations and that we were able to show more clearly our scholarly contributions. Thank you for taking the time to provide specific references at the end of your letter. They have helped us greatly in positioning the paper within the existing literature.

Point 3: Conceptual background & Theoretical development - The authors need to establish some clearer links to extant PPP literature (e.g. please see suggested key references) to guide the reader. This would help to clearly establish the basis of this paper, before delving deeper into social responsibility. The author(s) should clearly draw out the benefits and limitations of PPPs and issues around the management of such long-term relationships to drive social responsibility. This should then be linked to discussions around value for money, risks and particularly sustainable development as key drivers. A much clearer positioning of the current study will help to further guide the reader and draw out gap(s) in extant studies.  This would then help to set up the current conceptual model the authors position.

Response 3: Thank you for highlighting this shortcoming in the prior version of our paper. As previously indicated, we followed your recommendations and we have rewritten large portions of the document to clearly establish the link with prior literature and showing the readers the benefits and limitations of PPPs, which has been described mainly in Section 1 - Introduction. In addition, in Section 2 – Conceptual Model, we have developed more fully the links to extant literature between PPPs and sustainability in general, and to the SDGs in particular, given the focus of the Special Issue. Having these elements as background, we present our conceptual model and later discuss how it can help better assess the contribution of PPPs to sustainable development. Additionally, we followed your suggestion (which was echoed by reviewer 2), and assured that all of the dimensions of our model were justified by prior literature. This new version of the manuscript explores more in detail extant literature for each of the dimensions of our model. Together, we believe that these changes highlight how our EASIER model helps closing, at least partially, the existing gaps in the literature. We hope you concur this new approach.

Point 4: Methods and Analysis - Overall, this section is well written, but I see some room for improvement to obtain robust results and strengthen the paper’s argument. The authors should address the following concerns

Response 4:  We are glad you found our methods section well written. We made an effort to maintain this positive feature in the new version of the manuscript. Below we explain how we have addressed your remaining concerns.

Point 5: Methods and Analysis - Please clearly tell the reader about your sampling logic with regards to respondents, the PPP projects under investigation and the countries you selected.

Response 5:  Thank you for bringing clarity to our manuscript. In retrospect, we realize that we were imprecise in explaining how our cases were selected. In this new version of the manuscript, we explain that, consistent with our qualitative approach, we adopted a purposeful sampling approach (Palinkas et al., 2015). This approach is generally recommended for the identification and selection of information-rich examples, such those in our case. Within the options provided within purposeful sampling, we followed the Criterion-i model to identify and select cases that meet predetermined criterion that we deemed of importance. Specifically, we defined a list of criteria that case studies need to meet or exceed in order to be selected. The set of criteria includes that the case study must be a PPP project where both public and private sectors are clearly identified; that the construction or building phase of the PPP project must be finalised; the case study must afford enough data; and the data and sources of information must be reliable, independent and accessible.

Point 6: Methods and Analysis - You also need to clearly show the findings/results rather than just ‘tell’ the reader. At the moment, it is impossible for the reader to judge whether you have actually active your research aims/objectives and very limited data are presented. This section needs a complete rework.  

Response 6:  Thank you for this comment, as it allowed us to make our case study clearer. In this new version, we have included one table (Table 3) that details the results of each of the two evaluators of La Chira case study, both for the pre-test (i.e. economic efficiency) and the proper EASIER evaluation. Moreover, following reviewer 2’s suggestion, we calculated the inter-rater reliability using Cohen’s kappa in order to provide the reader greater confidence about our results. We have added a new section 4.1 with the results of the inter-rater reliability using Cohen’s kappa, and a number of its variation, including Conger’s kappa, Fleiss' Pi, Gwet's AC and Krippendorff' alpha coefficient. As indicated, in the new version of our manuscript, these coefficients indicate a strong level of agreement of consensus between the two evaluators. Still, we are careful in making definitive interpretations from them and invite future research to invest additional efforts in validating the model.

Point 7: Discussions and Conclusions - Derived from a conceptual background section which did not clearly draw out the gaps the paper seeks to address, the discussion and conclusion sections do offer very little additional value to the reader as it stands. The authors need to offer more fine-grained results here and discuss what they intended to find out in the introduction section (link to research question; overall aim of the paper). Overall, the authors need to clearly draw out what the theoretical contributions are and how they add to the existing body of knowledge. This section also needs to clear link back to extant studies to offer some clear value to the reader.

Response 7:  Thank you for this suggestion. We have revamped the entire discussion section, improving the clarity of the arguments. We have structured it more clearly identifying the implications of our work for academia, policymaking, and practice. Moreover, we have enhanced our discussion about our limitations and explored in greater depth future avenues for research. In doing so, we included both relevant and recent literature to better frame our results within the existing literature. In addition, we related our study to prior literature, such as those suggested by you. Perhaps more importantly, we highlighted how our work provides additional nuances about the importance of evaluating SDGs from the PPPs’ perspective.

Point 8: Useful references.

Response 8:  We sincerely appreciate you taking the time to suggest specific references. We have included them in the new version of the manuscript.

In closing, thank you very much for your very thoughtful and constructive comments. We have done our best to address them all fully, and we hope that you find that the new version of the manuscript has been substantially improved. We sincerely feel, and we hope you agree, that the revised version offers a solid contribution to the literature and to the Sustainability Special Issue about Evaluating SDGs.

References

Barlow, J.; Roehrich, J.; Wright, S. Europe Sees Mixed Results From Public-Private Partnerships For Building And Managing Health Care Facilities And Services. Health Aff. 2013, 32, 146–154; doi: 10.1377/hlthaff.2011.1223.

Brammer, S.; Walker, H. Sustainable procurement in the public sector: An international comparative study. Int. J. Oper. Prod. Manag. 2011, 31, 452–476; doi: 10.1108/01443571111119551.

Zheng, J.; Roehrich, J.K.; Lewis, M.A. The dynamics of contractual and relational governance: Evidence from long-term public-private procurement arrangements. J. Purch. Supply Manag. 2008, 14, 43–54; doi: 10.1016/j.pursup.2008.01.004.

Palinkas, L.A.; Horwitz, S.M.; Green, C.A.; Wisdom, J.P.; Duan, N.; Hoagwood, K. Purposeful Sampling for Qualitative Data Collection and Analysis in Mixed Method Implementation Research. Adm. Policy Ment. Health 2015, 42, 533–44; doi: 10.1007/s10488-013-0528-y.

Author Response File: Author Response.docx

Reviewer 2 Report

Overall, this is an interesting study that seeks to propose or offer a conceptual model (EASIER) that accounts for six dimensions relevant for social, envioronmental and economic progress. The authors have doen an excellent job at articulating and formulating the EASIER conceptual model, and the individual six dimensions are well decsribed and underpinned by the relevant literature. However, the major and main shortcoming of this EASIER model is that it appears to omit a stage within the development, namely that of actually demonstarting that the proposed six dimesnions are indeed reliable and can measure the intended items. Rather than rush to ‘empirically’ validating the model using case stuydies, more attention should have been paid to the presentation of the individual results of the questionnaire as reported on Page 10, Line 380. For instance, was any kind of exploratory factor analysis (EFA ) of confirmatory factor analysis (CFA) used to confirm that the allocation of the individual items to the six dimensions was correct? If not, what is the justification of assigning these items to the six dimesions apriori? Why wasn’t Structural Equation Modelling (SEM) used for the development of this model?

What was the criteria used for the selection of the eight experts on PPP and sustainability (see Page 10, Line 377)? How can, the entire methodology section is neither supported by any references to justify the approaches as undertaken? What kindly sampling approach was used for adminestring the final questionnaire? What was the actual response rate?  Was cronbach alpha coefficient used for assessing the reliability of the individual deimensions as well as the overall scale?

The authors should either include the missing stages as articulated above or focus the entire paper on just the conceptual issues with a future recommendations of verfication and validation of the proposed model.

Author Response

Point 1: Overall, this is an interesting study that seeks to propose or offer a conceptual model (EASIER) that accounts for six dimensions relevant for social, envioronmental and economic progress. The authors have done an excellent job at articulating and formulating the EASIER conceptual model, and the individual six dimensions are well decsribed and underpinned by the relevant literature. However, the major and main shortcoming of this EASIER model is that it appears to omit a stage within the development, namely that of actually demonstarting that the proposed six dimesnions are indeed reliable and can measure the intended items. Rather than rush to ‘empirically’ validating the model using case stuydies, more attention should have been paid to the presentation of the individual results of the questionnaire as reported on Page 10, Line 380. For instance, was any kind of exploratory factor analysis (EFA ) of confirmatory factor analysis (CFA) used to confirm that the allocation of the individual items to the six dimensions was correct? If not, what is the justification of assigning these items to the six dimesions apriori? Why wasn’t Structural Equation Modelling (SEM) used for the development of this model?

Response 1: Thank you very much for this positive comment and for seeing our efforts in articulating our EASIER model. We found your comments to be very constructive, and we took great care to answer each one to the best of our ability. We believe that your suggestions, as well as those of the other reviewer, have helped us to strengthen the manuscript significantly. 

Below, we provide point-by-point responses to your suggestions. We greatly appreciate your highly detailed feedback, and we have done our best to follow each of your recommendations.

Regarding your comment about whether the proposed six dimensions are reliable, two comments are in order. First, in order to make the SDGs more accessible to organizations, there have been recent attempts to SDGs simplify them. For instance, the OECD attempted to organize SDGs according to its Well-Being framework, recognizing overlaps but also important differences (OECD, 2017). Griggs et al. (2014) suggested a six-area framework, which includes thriving lives and livelihoods, sustainable food security, sustainable water security, universal clean energy, sustainable ecosystems, and governance. Similarly, Elder, Bengtsson and Akenji (2016) proposed to streamline the SDGs in six main groups: social objectives, resources, economy, environment, education, and governance. Together, we believe that these studies highlight the need to make the SDGs more “palatable” to different stakeholders through a simplified version of them. We have added these references in Section 2 – Conceptual Model, and added them at the end of this response letter for your reference.

The second aspect refers to the extent to which our six dimensions are valid ones. In order to assure this validity, we contrasted the general model and the specific questions with a panel of experts. The use of a panel of experts is often recommended to maximize the likelihood of content-valid, well-constructed data collection instruments (Rubio et al., 2003). Moreover, we conducted a pilot (PPP project of VLT Carioca), which allowed us to readjust and improve the questionnaire. This new questionnaire was again validated with the experts’ panel in an iterative process. In the version of the paper, we explain these aspects more clearly.

Additionally, following your suggestion, we also ground the dimensions of the model on existing literature (see pages 6 to 10). It should be noted that we also have written a full practical guide that includes a scholarly justification for each question. This is a 100-page document that for the sake of parsimony, we are not including. However, we reference it in the new version of the paper and we could make it available to you for review purposes if you deem it necessarily.

We appreciate your suggestion of exploratory factor analysis (EFA), confirmatory factor analysis (CFA) and Structural Equation Modelling (SEM). However, it should be noted that given the qualitative nature of our work, these techniques could not be applied, as they are useful for quantitative studies. Having said this, and in response to your concern, we calculated the inter-rater reliability using Cohen’s kappa (and a number of its variation, including Conger’s kappa, Fleiss' Pi, Gwet's AC and Krippendorff' alpha coefficient) in order to provide the reader greater confidence about our results. We have included a new sub-section, section 4.1 with the results. Overall, we agree with you that subsequent efforts need to be done in order to enhance the validity of our model. We now make clear in the new version of the manuscript that this is could be an interesting avenue for future research and that the methods you proposed can be useful.

Point 2: What was the criteria used for the selection of the eight experts on PPP and sustainability (see Page 10, Line 377)? How can, the entire methodology section is neither supported by any references to justify the approaches as undertaken? What kindly sampling approach was used for administering the final questionnaire? What was the actual response rate?  Was cronbach alpha coefficient used for assessing the reliability of the individual deimensions as well as the overall scale?

The authors should either include the missing stages as articulated above or focus the entire paper on just the conceptual issues with a future recommendations of verfication and validation of the proposed model.

Response 2: Thank you for bringing these points. First, in regards to the panel of experts, in retrospect, we realize that we were imprecise in explaining how our panel of experts were selected. In the new version of the paper, we explain that the panel following very specific criteria in selecting the members of the panel. Firstly, we wanted to assure that the group comprised a balance mix of academics and professionals knowledgeable about PPPs and sustainability. Another criterion was that, in addition to knowledge, they were currently active in those two fields, in order to assure update insights. Finally, because the intended goal was to develop an assessment model, another criteria was that at least one member had to have experience and knowledge about evaluation of public policies. With these criteria in mind, eight individuals were invited to form the panel, five of them holding PhDs in Economics, Management and/or Quantitate Methods. They had strong research background in fields such as PPP, Sustainability, Strategic Management, Urban Development, and Governance. Importantly, they were ascribed to the International Center of Excellence on PPP in the United Nations Economic Committee in Europe (UNECE). This assured us that they were familiar with the work of United Nations, in particular with the SDGs. Two additional members were professional consultants, with more than 15 years of experience combined in the implementation, management, and advisory in public-private partnerships. Lastly, the group of experts was completed with an experienced PhD candidate in Economics whose topics of interest were evaluation of public policies. During several meetings with the panel members and through an iterative process, we were able to define the conceptual framework and the questionnaire.

Second, in regards to the sampling logic in this new version of the manuscript, we explain that, consistent with our qualitative approach, we adopted a purposeful sampling approach (Palinkas et al., 2015). This approach is generally recommended for the identification and selection of information-rich examples, such those in our case. Within the options provided within purposeful sampling, we followed the Criterion-i model to identify and select cases that meet predetermined criterion that we deemed of importance. Specifically, we defined a list of criteria that case studies need to meet or exceed in order to be selected. These criteria for the selection of case studies includes: the case study must be a PPP project where both public and private sectors are clearly identified; the construction or building phase of the PPP project must be finalised; the case study must afford enough data; and the data and sources of information must be reliable, independent and accessible. We have added a new Figure, Figure 3, in order to depict the research methodology followed for the EASIER conceptual model and the EASIER evaluation tool.

Third, in order to improve the methodology section, in this new version of the manuscript, we have included one table (Table 3) that details the results of each of the two evaluators of La Chira case study, both for the pre-test (i.e. economic efficiency) and the proper EASIER evaluation. Moreover, as previously mentioned, we calculated the inter-rater reliability using Cohen’s kappa in order to provide the reader greater confidence about our results. We have added a new section 4.1 with the results of the inter-rater reliability using Cohen’s kappa and a number of its generalizations (e.g. Conger, Fleiss's kappa, Gwet's AC, or Krippendorff' alpha coefficient).

Lastly, in response to your concern, and as previously mentioned, we have justified each dimension with existing literature to enhance the validity of our six measures. Unfortunately, Cronbach alpha cannot be calculated given that our study in qualitative. However, the lack of quantitative data does not invalidate our approach. We considered your suggestion of only focusing on the conceptual framework. However, we believe that the iterative process with the panel of experts together with the application of the questionnaire to a specific case is an initial validation (rather than a definitive one) that could subsequently be enhanced with quantitative approaches.

Thank you very much for your careful review and thoughtful comments. We took each of them very seriously and have done our best to address them fully. We hope that you find this revision to be much improved.  We hope that we were able to make a stronger case for our contributions and that you find our responses and the new version of the manuscript satisfactory and suitable for the special issue of Sustainability about Evaluating SDGs.

References:

Elder, M.; Bengtsson, M.; Akenji, L. An Optimistic Analysis of the Means of Implementation for Sustainable Development Goals: Thinking about Goals as Means. Sustainability 2016, 8, 962.

Griggs, D.; Stafford Smith, M.; Rockström, J.; Öhman, M.C.; Gaffney, O.; Glaser, G.; Kanie, N.; Noble, I.; Steffen, W.; Shyamsundar, P. An integrated framework for sustainable development goals. Ecol. Soc. 2014, 19, art49.

OECD. Measuring Distance to the SDG Targets: An assessment of where OECD countries stand; OECD: Paris, France, 2017. Available online: http://www.oecd.org/sdd/OECD-Measuring-Distance-to-SDG-Targets.pdf (accessed on 12 March 2019).

Palinkas, L.A.; Horwitz, S.M.; Green, C.A.; Wisdom, J.P.; Duan, N.; Hoagwood, K. Purposeful Sampling for Qualitative Data Collection and Analysis in Mixed Method Implementation Research. Adm. Policy Ment. Health 2015, 42, 533–44; doi: 10.1007/s10488-013-0528-y

Rubio, D.M.; Berg-Weger, M.; Tebb, S.; Lee, E.S.; Rauch, S. Objectifying content validity: Conducting a content validity study in social work research. Soc. Work Res. 2003, 27, 94–104.

Author Response File: Author Response.docx

Round  2

Reviewer 1 Report

Many thanks for addressing all my comments from the previous round in so much detail. The revised manuscript reads very well. Please do proofread the manuscript before submission again.

Author Response

Point 1: Many thanks for addressing all my comments from the previous round in so much detail. The revised manuscript reads very well. Please do proofread the manuscript before submission again.

Response 1: We are pleased that you found the revised version of our manuscript improved and well written. We have carefully proofread the manuscript and edited it accordingly. We hope that the readability of the current version of the manuscript has been substantially improved. Overall, thank you very much for your thoughtful comments and suggestions, as they have been very useful to significantly improve the paper.

Reviewer 2 Report

The reviewer is grateful for the manner in which the authors have accommodated the corrections.

Author Response

Point 1: The reviewer is grateful for the manner in which the authors have accommodated the corrections.

Response 1: We are pleased that you found the revised version of our manuscript improved. As suggested by you and the other reviewer, we have carefully proofread the manuscript and edited it accordingly. We hope that the readability of the current version of the manuscript has been substantially improved. Overall, thank you very much for your thoughtful comments and suggestions, as they have been very useful to significantly improve the paper.