Where Are the Costs? Using an Economic Analysis of Educational Interventions Approach to Improve the Evaluation of a Regional School Improvement Programme
Abstract
:1. Introduction
2. Formative Assessment in Policy and Practice
- Making observations: The teacher needs to explore what the learner does or does not know, and this is typically achieved by listening to learners’ responses, observing the learner on tasks, and/or assessing class or homework tasks.
- Interpretation: The teacher interprets the skill, knowledge, or attitudes of the learners.
- Judgement: Once evidence has been gathered through observation and interpretation, the teacher then makes a judgement on the next course of action to move the learner forward.
- Sharing Learning Expectations: Ensuring the learner knows what they are going to learn and the success criteria to achieve this goal.
- Questioning: Using effective questioning to facilitate learning.
- Feedback: Providing feedback that enhances learning within the moment.
- Self-assessment: Allowing learners to take ownership of, and reflect on, their learning.
- Peer assessment: Providing opportunities for learners to discuss their work with, and to instruct, others.
3. Methods
3.1. Trial Design
3.2. Recruitment
3.3. Study Population
3.4. Outcomes
3.5. Analysis
3.6. Interviews
3.7. Focus Group
3.8. Procedure
3.9. Observations
- Is it clear what the teacher intends the students to learn?
- Does the teacher identify student learning needs?
- Do students understand what criteria will make their work successful?
- Are students chosen at random to answer questions?
- Does the teacher ask questions that make students think?
- Does the teacher give students time to think after asking a question?
- Does the teacher allow time for students to elaborate their responses?
- Is a whole-class response system used?
- Is teaching adjusted after gathering feedback from pupils (data collection)?
- Is there more student talk than teacher talk?
- Are most students involved in answering questions?
- Are students supporting each other’s learning?
- Is there evidence that various forms of teacher feedback advance student learning?
- Do students take responsibility for their own learning?
- Does the teacher provide oral formative feedback?
- Does the teacher find out what the students have learned before they leave the room?
4. Intervention
5. Economic Analysis of Educational Interventions (EAEI)
5.1. Cost-Consequence Analysis
5.2. Rationale for CCA
5.3. Cost Collecting Methodology
5.4. Collating Costs
5.5. Sensitivity Analysis
6. Results
6.1. Learner Outcomes
6.2. Classroom Observations
6.3. The Opinions of Teachers and Learners
6.4. Interviews with Teachers
“…helps me feel I get a greater understanding of my children. And I don’t go home at the end of the week thinking, I don’t think I’ve said five words to that child.”Teacher 6
“And sometimes it can be a little bit of idleness of picking up a pen but sometimes it’s their belief in themselves a lot of the time. And it is, it’s them thinking, “actually, I can do it”. “I think a lot of it is the confidence they have…”Teacher 4
“So, it’s easier. I think the quality of work is easier to mark…I do feel I’ve got extra time.”Teacher 2
“I would say predominantly it’s that the lower achievers it’s had the bigger impact on.”Teacher 5
6.5. Focus Group Interviews with Learners
“You kind of get to know them more, cos like…you just like…you don’t really play with them, cos you like different things, but if you’re discussion partners, you might have to try and get to know them…You might think better of them.”Learner, school L
“It makes the work a bit more straight forward, Cos when you look at the success criteria when you’re working, then it like gives you more to think about it and then more to think about the work.”Learner, school P
“…the teacher will take you out of the lessons and things just to like go over your piece of work and if you’ve done something well, he’ll tell you what you’ve done well and he’ll like highlight it on the success criteria, which is a list of things that you have to do and he’ll highlight it pink and then if you need to do something better, he’ll highlight it green and then he’ll tell you to re-do it and he’ll tell you what to re-do and stuff.”Learner, school O
The teachers discussed how FAIP supported them to help learners better understand the nature of successful outcomes and understand the expectations of quality standards in their work. Additionally, learners described how success criteria helped them complete tasks more successfully.
Teachers identified how using the strategies contained within the FAIP training helped them better understand where the learners were in their learning and provided them with useful, additional information to plan next steps. Teachers also indicated that they were more able to identify which learners needed support and adapt teaching in real time to provide next steps advice and support to learners.
Teachers discussed how feedback strategies supported them to better understand learner progress. Some teachers discussed being able to give immediate feedback to support learners to improve the outcomes they achieve.
Learners identified how the use of a range of self-assessment strategies impacted positively on the learning process and how it helped them engage with, and complete, tasks more successfully.
Teachers identified improved opportunities for learners to discuss their own work to enhance understanding and knowledge. Learners understood what a talk partner was, and how it helped them with their learning. It also enabled them to provide support for other learners. Learners also identified how it improved social relationships in school.
6.6. The Full Economic Cost of FAIP for Tier 2 Teachers
6.7. Sensitivity Analysis
6.7.1. Sensitivity Analysis 1
6.7.2. Sensitivity Analysis 2
6.7.3. Sensitivity Analysis 3
6.7.4. Sensitivity Analysis 4
7. Discussion
8. Limitations
9. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Gorard, S.; See, B.H.; Siddiqui, N. What is the evidence on the best way to get evidence into use in education? Rev. Educ. 2020, 8, 570–610. [Google Scholar] [CrossRef]
- OECD. Value for Money in School Education: Smart Investments, Quality Outcomes, Equal Opportunities; OECD Publishing: Paris, France, 2022. [Google Scholar] [CrossRef]
- Pegram, J.; Watkins, R.C.; Hoerger, M.; Hughes, J.C. Assessing the range and evidence-base of interventions in a cluster of schools. Rev. Educ. 2022, 10, e3336. [Google Scholar] [CrossRef]
- Slavin, R.E. How evidence-based reform will transform research and practice in education. Educ. Psychol. 2020, 55, 21–31. [Google Scholar] [CrossRef]
- Levin, H. Waiting for Godot: Cost-effectiveness analysis in education. New Dir. Eval. 2001, 2001, 55–68. [Google Scholar] [CrossRef]
- Owen, K.L.; Watkins, R.C.; Hughes, J.C. From evidence-informed to evidence-based: An evidence building framework for education. Rev. Educ. 2022, 10, e3342. [Google Scholar] [CrossRef]
- Hummel-Rossi, B.; Ashdown, J. The state of cost-benefit and cost-effectiveness analyses in education. Rev. Educ. Res. 2002, 72, 1–30. [Google Scholar] [CrossRef]
- Machin, S.; Mcnally, S.; Wyness, G. Educational Research Educational attainment across the UK nations: Performance, inequality and evidence Educational attainment across the UK nations: Performance, inequality and evidence. Educ. Res. 2013, 55, 139–164. [Google Scholar] [CrossRef]
- Kraft, M.A. Interpreting Effect Sizes of Education Interventions. Educ. Res. 2020, 49, 241–253. [Google Scholar] [CrossRef]
- EEF. Cost Evaluation Guidance for EEF Evaluations. 2019. Available online: https://d2tic4wvo1iusb.cloudfront.net/documents/evaluation/evaluation-design/Cost_Evaluation_Guidance_2019.12.11.pdf (accessed on 9 January 2024).
- CBCSE. Centre for Benefit Cost Studies in Education. 2015. Available online: https://www.cbcse.org/costout (accessed on 8 January 2024).
- Hinde, S.; Walker, S.M.; Lortie-Forgues, H. Applying the Three Core Concepts of Economic Evaluation in Health to Education in the UK; Centre for Health Economics, University of York: York, UK, 2019. [Google Scholar]
- Quinn, B.; Van Mondfrans, A.; Worthen, B.R. Cost-Effectiveness of Two Math Programs as Moderated by Pupil SES. Educ. Eval. Policy Anal. 1984, 6, 39–52. [Google Scholar] [CrossRef]
- Hollands, F.; Bowden, A.B.; Belfield, C.; Levin, H.M.; Cheng, H.; Shand, R.; Pan, Y.; Hanisch-Cerda, B. Cost-Effectiveness Analysis in Practice: Interventions to Improve High School Completion. Educ. Eval. Policy Anal. 2014, 36, 307–326. [Google Scholar] [CrossRef]
- Buxton, M.J. Economic evaluation and decision making in the UK. Pharmacoeconomics 2006, 24, 1133–1142. [Google Scholar] [CrossRef]
- Corbacho, B.; Pinto-Prades, J.L. Health economic decision-making: A comparison between UK and Spain. Br. Med. Bull. 2012, 103, 5–20. [Google Scholar] [CrossRef]
- Black, P.; Wiliam, D. Assessment and classroom learning. Assess. Educ. Princ. Policy Pract. 1998, 5, 7–74. [Google Scholar] [CrossRef]
- Black, P.; Wiliam, D. Classroom assessment and pedagogy. Assess. Educ. Princ. Policy Pract. 2018, 25, 551–575. [Google Scholar] [CrossRef]
- Clarke, S. Outstanding Formative Assessment: Culture and Practice; Hachette: London, UK, 2014. [Google Scholar]
- EEF. Teacher Feedback to Improve Pupil Learning. 2021. Available online: https://d2tic4wvo1iusb.cloudfront.net/production/eef-guidance-reports/feedback/Teacher_Feedback_to_Improve_Pupil_Learning.pdf?v=1713861365 (accessed on 10 January 2024).
- Bennett, R.E. Formative assessment: A critical review. Assess. Educ. Princ. Policy Pract. 2011, 18, 5–25. [Google Scholar] [CrossRef]
- Black, P.; Wiliam, D. Developing the theory of formative assessment. Educ. Assess., Eval. Account. (Former. J. Pers. Eval. Educ.) 2009, 21, 5–31. [Google Scholar] [CrossRef]
- OECD; CERI. Formative Assessment: Improving Learning in Secondary Classrooms; CERI/OECD: Paris, France, 2005. [Google Scholar]
- Wiliam, D. Embedded Formative Assessment: Strategies for Classroom Assessment That Drives Student Engagement and Learning; Solution Tree: Bloomington, IN, USA, 2017. [Google Scholar]
- James, M. Embedding formative assessment in classroom practice. In Life in Schools and Classrooms; Springer: Singapore, 2017; pp. 509–525. [Google Scholar]
- Wiliam, D. Research into practice. In Getting Evidence into Education: Evaluating the Routes to Policy and Practice; Routledge: London, UK, 2020. [Google Scholar]
- Donaldson, G. Successful Futures Independent Review of Curriculum and Assessment Arrangements in Wales Cardiff: Welsh Government. 2015. Available online: https://gov.wales/sites/default/files/publications/2018-03/successful-futures.pdf (accessed on 9 January 2024).
- Allal, L. Involving primary school students in the co-construction of formative assessment in support of writing. Assess. Educ. Princ. Policy Pract. 2021, 28, 584–601. [Google Scholar] [CrossRef]
- Anders, J.; Foliano, F.; Bursnall, M.; Dorsett, R.; Hudson, N.; Runge, J.; Speckesser, S. The effect of embedding formative assessment on pupil attainment. J. Res. Educ. Eff. 2022, 15, 748–779. [Google Scholar] [CrossRef]
- Cosi, A.; Voltas, N.; Lázaro-Cantabrana, J.L.; Morales, P.; Calvo, M.; Molina, S.; Quiroga, M.Á. Formative assessment at university through digital technology tools. Profr. Rev. Curric. Y Form. Profr. 2020, 24, 164–183. [Google Scholar] [CrossRef]
- Cisterna, D.; Gotwals, A.W. Enactment of ongoing formative assessment: Challenges and opportunities for professional development and practice. J. Sci. Teach. Educ. 2018, 29, 200–222. [Google Scholar] [CrossRef]
- Barana, A.; Marchisio Conte, M. Promoting socioeconomic equity through automatic formative assessment. J. Math. Educ. 2023, 15, 227–252. [Google Scholar] [CrossRef]
- Kingston, N.; Nash, B. Formative assessment: A meta-analysis and a call for research. Educ. Meas. Issues Pract. 2011, 30, 28–37. [Google Scholar] [CrossRef]
- James, M. Assessment for learning: Research and policy in the (dis) United Kingdom. In Assessment Reform in Education; Springer: Dordrecht, The Netherlands, 2011; pp. 15–32. [Google Scholar]
- Leahy, S.; Wiliam, D. Embedding Formative Assessment: Practical Techniques for K-12 Classrooms; Hawker Brownlow Education: Cheltenham, UK, 2015. [Google Scholar]
- Campbell, D.T.; Stanley, J.C. Experimental and Quasi-Experimental Designs for Research; Ravenio Books: Charleston, SC, USA, 2015. [Google Scholar]
- Gray, E.; McCambridge, J.; Strang, J. The effectiveness of motivational interviewing delivered by youth workers in reducing drinking, cigarette and cannabis smoking among young people: Quasi-experimental pilot study. Alcohol Alcohol. 2005, 40, 535–539. [Google Scholar] [CrossRef]
- Hakim, C. Research Design: Successful Designs for Social Economics Research; Routledge: London, UK, 2012. [Google Scholar]
- Handley, M.A.; Lyles, C.R.; McCulloch, C.; Cattamanchi, A. Selecting and improving quasi-experimental designs in effectiveness and implementation research. Annu. Rev. Public Health 2018, 39, 5–25. [Google Scholar] [CrossRef]
- Palinkas, L.A.; Horwitz, S.M.; Green, C.A.; Wisdom, J.P.; Duan, N.; Hoagwood, K. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm. Policy Ment. Health Ment. Health Serv. Res. 2015, 42, 533–544. [Google Scholar] [CrossRef]
- Welsh Government. National Reading and Numeracy Personalised Assessments: Administration Handbook. 2019. Available online: https://hwb.gov.wales/curriculum-for-wales/reading-and-numeracy-assessments/personalised-assessments/national-reading-and-numeracy-personalised-assessments-administration-handbook#specific-requirements-for-numeracy-(procedural)-and-numeracy-(reasoning)-personalised-assessments (accessed on 8 February 2024).
- Dunphy, E. Assessing early learning through formative assessment: Key issues and considerations. Ir. Educ. Stud. 2010, 29, 41–56. [Google Scholar] [CrossRef]
- Stevens, K. Valuation of the child health utility 9D index. Pharmacoeconomics 2012, 30, 729–747. [Google Scholar] [CrossRef] [PubMed]
- Boyer, N.; Miller, S.; Connolly, P.; McIntosh, E. Population Health Economic Evaluation of the School-Based Roots of Empathy Program in Northern Ireland. In Proceedings of the 2018 Annual Research Meeting, Washington, DC, USA, 24–26 June 2018; AcademyHealth: Washington, DC, USA, 2018. [Google Scholar]
- White, J.; Connelly, G.; Thompson, L.; Wilson, P. Assessing children’s social and emotional wellbeing at school entry using the strengths and difficulties questionnaire: Professional perspectives. Educ. Res. 2013, 55, 87–98. [Google Scholar] [CrossRef]
- Weintraub, N.; Bar-Haim Erez, A. Quality of life in school (QoLS) questionnaire: Development and validity. Am. J. Occup. Ther. 2009, 63, 724–731. [Google Scholar] [CrossRef]
- Ghotra, S.; McIsaac, J.L.D.; Kirk, S.F.; Kuhle, S. Validation of the “Quality of Life in School” instrument in Canadian elementary school students. PeerJ 2016, 4, e1567. [Google Scholar] [CrossRef]
- Gorard, S.A.C. Rethinking ‘quantitative’methods and the development of new researchers. Rev. Educ. 2015, 3, 72–96. [Google Scholar] [CrossRef]
- Gorard, S.; Gorard, J. What to do instead of significance testing? Calculating the ‘number of counterfactual cases needed to disturb a finding’. Int. J. Soc. Res. Methodol. 2016, 19, 481–490. [Google Scholar] [CrossRef]
- Bell, A. Designing and testing questionnaires for children. J. Res. Nurs. 2007, 12, 461–469. [Google Scholar] [CrossRef]
- Welsh Government. Curriculum for Wales; Welsh Government: Cardiff, UK, 2024. Available online: https://hwb.gov.wales/curriculum-for-wales/ (accessed on 12 February 2024).
- Morris, S.; Devlin, N.; Parkin, D.; Spencer, A. Principles of economic evaluation in health care. In Economic Analysis in Health Care, 2nd ed.; John Wiley & Sons: Chichester, UK, 2012; pp. 232–250. [Google Scholar]
- Charles, J.; Edwards, R.T. A Guide to Health Economics for Those Working in Public Health: A Concise Desktop Handbook; Centre for Health Economics and Medicines Evaluation: Bangor, UK, 2016. [Google Scholar]
- Charles, J.M.; Harrington, D.M.; Davies, M.J.; Edwardson, C.L.; Gorely, T.; Bodicoat, D.H.; Khunti, K.; Sherar, L.B.; Yates, T.; Edwards, R.T. Micro-costing and a cost-consequence analysis of the ‘Girls Active’programme: A cluster randomised controlled trial. PLoS ONE 2019, 14, e0221276. [Google Scholar] [CrossRef] [PubMed]
- Haghparast-Bidgoli, H.; Skordis, J.; Harris-Fry, H.; Krishnan, S.; O’Hearn, M.; Kumar, A.; Pradhan, R.; Mishra, N.K.; Upadhyay, A.; Pradhan, S.; et al. Protocol for the cost-consequence and equity impact analyses of a cluster randomised controlled trial comparing three variants of a nutrition-sensitive agricultural extension intervention to improve maternal and child dietary diversity and nutritional status in rural Odisha, India (UPAVAN trial). Trials 2019, 20, 287. [Google Scholar] [CrossRef] [PubMed]
- Hartfiel, N.; Edwards, R.T. Cost–consequence analysis of public health interventions. In Applied Health Economics for Public Health Practice and Research; Oxford University Press: Oxford, UK, 2019; pp. 233–247. [Google Scholar] [CrossRef]
- Coast, J. Is economic evaluation in touch with society’s health values? BMJ 2004, 329, 1233–1236. [Google Scholar] [CrossRef]
- Rogers, J.C.; Simmons, E.A.; Convery, I.; Weatherall, A. Public perceptions of opportunities for community-based renewable energy projects. Energy Policy 2008, 36, 4217–4226. [Google Scholar] [CrossRef]
- Glouberman, S.; Zimmerman, B. Complicated and Complex Systems: What Would Successful Reform of Medicare Look Like? Commission on the Future of Health Care in Canada: Ottawa, ON, Canada, 2002. [Google Scholar]
- Chapko, M.K.; Liu, C.F.; Perkins, M.; Li, Y.F.; Fortney, J.C.; Maciejewski, M.L. Equivalence of two healthcare costing methods: Bottom-up and top-down. Health Econ. 2009, 18, 1188–1201. [Google Scholar] [CrossRef]
- Scammacca, N.; Swanson, E.; Vaughn, S.; Roberts, G. Cost-Effectiveness of a Grade 8 Intensive Reading and Content Learning Intervention. Sch. Psychol. Rev. 2020, 49, 374–385. [Google Scholar] [CrossRef]
- Harden, J. The True Cost of £ducation. 2019. Available online: https://www.ascl.org.uk/ASCL/media/ASCL/Our%20view/Campaigns/The-True-Cost-of-Education.pdf (accessed on 10 January 2024).
- NASUWT. Directed Time (Wales). 2024. Available online: https://www.nasuwt.org.uk/advice/conditions-of-service/teachers-working-hours/directed-time-wales.html#TeachersWorkingTime (accessed on 25 January 2024).
- Limwattananon, S. Sensitivity analysis for handling uncertainty in an economic evaluation. J. Med. Assoc. Thail. Chotmaihet Thangphaet 2014, 97, S59–S64. [Google Scholar]
- Vaismoradi, M.; Jones, J.; Turunen, H.; Snelgrove, S. Theme development in qualitative content analysis and thematic analysis. J. Nurs. Educ. Pract. 2016, 6, 100–110. [Google Scholar] [CrossRef]
- Braun, V.; Clarke, V. Thematic Analysis; American Psychological Association: Washington, DC, USA, 2012. [Google Scholar]
- Welsh Government. Draft Curriculum for Wales 2022: A Guide to Curriculum for Wales 2022; Welsh Government: Cardiff, UK, 2019. Available online: https://cyfarthfahigh.merthyr.sch.uk/wp-content/uploads/2019/05/A-Guide-to-the-Curriculum-for-Wales-2022.pdf (accessed on 12 February 2024).
- Levin, H.; McEwan, P.; Belfield, C.; Bowden, A. Economic Evaluation in Education: Cost-Effectiveness and Benefit-Cost Analysis; SAGE Publications: London, UK, 2017. [Google Scholar]
- Welsh Government. School Census Results; Welsh Government: Cardiff, UK, 2018. Available online: https://www.gov.wales/sites/default/files/statistics-and-research/2018-12/180725-school-census-results-2018-en.pdf (accessed on 12 August 2024).
- Boström, E.; Palm, T. The effect of a formative assessment practice on student achievement in mathematics. Front. Educ. 2023, 8, 1101192. [Google Scholar] [CrossRef]
- Martin, C.; Mraz, M.; Polly, D. Examining elementary school teachers’ perceptions of and use of formative assessment in mathematics. Int. Electron. J. Elem. Educ. 2022, 14, 417–425. [Google Scholar] [CrossRef]
- Beames, J.R.; Spanos, S.; Roberts, A.; McGillivray, L.; Li, S.; Newby, J.M.; O’Dea, B.; Werner-Seidler, A. Intervention programs targeting the mental health, professional burnout, and/or wellbeing of school teachers: Systematic review and meta-analyses. Educ. Psychol. Rev. 2023, 35, 26. [Google Scholar] [CrossRef]
- Gorard, S.; Siddiqui, N.; See, B.H. What works and what fails? Evidence from seven popular literacy ‘catch-up’schemes for the transition to secondary school in England. Res. Pap. Educ. 2017, 32, 626–648. [Google Scholar] [CrossRef]
- Speckesser, S.; Runge, J.; Foliano, F.; Bursnall, M.; Hudson-Sharp, N.; Rolfe, H.; Anders, J. Embedding Formative Assessment: Evaluation Report and Executive Summary; Education Endowment Foundation: London, UK, 2018. [Google Scholar]
- Newell, R.G.; Pizer, W.A.; Prest, B.C. The Shadow Price of Capital: Accounting for Capital Displacement in Benefit-Cost Analysis. Environ. Energy Policy Econ. 2024, 5, 49–69. [Google Scholar] [CrossRef]
- Curtis, L.A. Unit Costs of Health and Social Care 2013; Personal Social Services Research Unit: Kent, UK, 2013. [Google Scholar]
- Shand, R.; Bowden, A.B. Empirical support for establishing common assumptions in cost research in education. J. Res. Educ. Eff. 2022, 15, 103–129. [Google Scholar] [CrossRef]
- MacKillop, E.; Sheard, S. Quantifying life: Understanding the history of quality-adjusted life-years (QALYs). Soc. Sci. Med. 2018, 211, 359–366. [Google Scholar] [CrossRef] [PubMed]
- Hartz, S.; John, J. Contribution of economic evaluation to decision making in early phases of product development: A methodological and empirical review. Int. J. Technol. Assess. Health Care 2008, 24, 465–472. [Google Scholar] [CrossRef]
- Ramsey, S.D.; Willke, R.J.; Glick, H.; Reed, S.D.; Augustovski, F.; Jonsson, B.; Briggs, A.; Sullivan, S.D. Cost-effectiveness analysis alongside clinical trials II—An ISPOR Good Research Practices Task Force report. Value Health 2015, 18, 161–172. [Google Scholar] [CrossRef]
- Levin, H.M.; Belfield, C. Guiding the development and use of cost-effectiveness analysis in education. J. Res. Educ. Eff. 2015, 8, 400–418. [Google Scholar] [CrossRef]
- McMillan, J.H.; Venable, J.C.; Varier, D. Studies of the effect of formative assessment on student achievement: So much more is needed. Pract. Assess. Res. Eval. 2013, 18, 1–15. [Google Scholar] [CrossRef]
- Gorard, S. A proposal for judging the trustworthiness of research findings. Radic. Stat. 2014, 110, 47–60. [Google Scholar]
- Gorard, S. Judging the relative trustworthiness of research results: How to do it and why it matters. Rev. Educ. 2024, 12, e3448. [Google Scholar] [CrossRef]
- Greenberg, M.T.; Abenavoli, R. Universal interventions: Fully exploring their impacts and potential to produce population-level impacts. J. Res. Educ. Eff. 2017, 10, 40–67. [Google Scholar] [CrossRef]
- Charles, J.M.; Edwards, R.T.; Bywater, T.; Hutchings, J. Micro-costing in public health economics: Steps towards a standardized framework, using the incredible years toddler parenting program as a worked example. Prev. Sci. 2013, 14, 377–389. [Google Scholar] [CrossRef]
- Dolan, P.; Peasgood, T. Estimating the economic and social costs of the fear of crime. Br. J. Criminol. 2007, 47, 121–132. [Google Scholar] [CrossRef]
Schools | Number of Learners of Statutory Age | Language | Local Authority | % eFSM * | |
---|---|---|---|---|---|
Intervention | School L | 82 | Welsh | Anglesey | 8.5 |
School M | 83 | Welsh | Gwynedd | 19.3 | |
School N | 179 | Welsh | Gwynedd | 34.6 | |
School O | 87 | Welsh | Gwynedd | 16.1 | |
School P | 326 | English | Wrexham | 23 | |
School Q | 57 | Welsh | Gwynedd | 8.8 | |
School R | 355 | English | Flintshire | 8.2 | |
School S | 174 | Welsh | Gwynedd | 3.4 | |
School T | 287 | Welsh | Anglesey | 29.3 | |
Control | School A | 110 | Welsh | Gwynedd | 11.8 |
School B | 118 | Welsh | Anglesey | 7.6 | |
School C | 57 | Welsh | Anglesey | 19.3 | |
School D | 308 | English | Flintshire | 22.6 | |
School E | 214 | English | Wrexham | 19.2 | |
School F | 55 | Welsh | Gwynedd | 30.9 | |
School G | 83 | English | Conwy | 19.3 | |
School H | 112 | Welsh | Denbighshire | 17.9 |
Intervention (N = 110) | Control (N = 139) | ||
---|---|---|---|
Age (years) | 8 | 31 | 28 |
9 | 29 | 37 | |
10 | 41 | 47 | |
11 | 9 | 27 |
Teacher | Gender | School | Total Number of Statutory School-Age Learners | Main Language of Instruction | Local Authority | % eFSM * |
---|---|---|---|---|---|---|
Teacher 1 | Male | School L | 82 | Welsh | Anglesey | 8.5 |
Teacher 2 | Male | School M | 83 | Welsh | Gwynedd | 19.3 |
Teacher 3 | Female | School N | 179 | Welsh | Gwynedd | 34.6 |
Teacher 4 | Female | School P | 326 | English | Wrexham | 23 |
Teacher 5 | Female | School Q | 57 | Welsh | Gwynedd | 8.8 |
Teacher 6 | Female | School R | 355 | English | Flintshire | 8.2 |
Teacher 7 | Male | School T | 287 | Welsh | Anglesey | 29.3 |
Tier | Academic Year | Number of Schools |
---|---|---|
Tier 1 | 2017–2018 | 54 teachers from 27 schools were initially selected through a process of application and interview. Training and collaboration, led by GwE and the expert trainer, commenced in October 2017. |
Tier 2 | 2018–2019 | 326 teachers from 193 schools were selected through application to be part of tier 2 training. Training and collaboration, led by GwE and the expert trainer, commenced in September 2018. |
Tier 3 | 2019–2020 | 261 teachers from 140 schools were invited to be part of the tier 3 training. Training and collaboration, led by GwE and the expert trainer, commenced in September 2019. |
Core Principles | Implications for Teachers | Suggested Teaching Strategies |
---|---|---|
Sharing Learning Expectations: Ensuring the learner knows what they are going to learn and the success criteria to achieve this goal. |
|
|
Questioning: Using effective questioning to facilitate learning. |
|
|
Feedback: Providing feedback that enhances learning within the moment. |
|
|
Self-assessment: Allowing learners to take ownership of, and reflect on, their learning. |
|
|
Peer assessment: Providing opportunities for learners to discuss their work with, and to instruct, others. |
|
|
2018–2019 Prices (Mean) | 2020–2021 Prices (Mean) | 2022–2023 Prices (Mean) | |
---|---|---|---|
Teacher cost yearly | GBP 58,544 | GBP 60,947 | GBP 72,233 |
Cost per pupil yearly | GBP 3165 | GBP 3295 | GBP 3904 |
Cost per hour | GBP 46 | GBP 48 | GBP 57 |
Measure | Intervention (n = 109) * | Control (n = 136) * | |||||||
---|---|---|---|---|---|---|---|---|---|
Mean | SD | Gain | Mean | SD | Gain | Difference in Gain Scores | Effect Size | ||
English age-standardised score | Pre-score | 104.36 | 16.26 | −0.51 | 104.25 | 11.77 | −2.36 | +1.85 | +0.12 |
Post-score | 103.85 | 14.01 | 101.89 | 11.17 | |||||
English progress score | Pre-score | 1006.11 | 22.11 | −0.99 | 1006.57 | 17.48 | −3.96 | +2.97 | +0.15 |
Post-score | 1005.12 | 21.85 | 1002.61 | 16.25 | |||||
Welsh age-standardised score | Pre-score | 100.81 | 15.31 | −0.23 | 103.78 | 12.37 | +1.47 | −1.7 | −0.11 |
Post-score | 100.58 | 15.47 | 105.25 | 13.71 | |||||
Welsh Progress score | Pre-score | 1000.55 | 20.81 | +1.22 | 1006.43 | 18.56 | −0.46 | +1.68 | +0.08 |
Post-score | 1001.77 | 21.60 | 1005.97 | 18.86 | |||||
Numeracy age-standardised score | Pre-score | 106.30 | 14.21 | −1.83 | 106.99 | 15.90 | −0.38 | −1.45 | −0.10 |
Post-score | 104.47 | 13.79 | 106.61 | 14.20 | |||||
Numeracy Progress score | Pre-score | 1009.41 | 18.97 | −2.36 | 1009.61 | 20.68 | −0.46 | −1.90 | −0.10 |
Post-score | 1007.05 | 17.70 | 1009.15 | 18.20 |
Measure | Intervention | Control | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Mean | SD | n | Gain | Mean | SD | n | Gain | Difference in Gain Scores | Effect Size | ||
CHU-9D a | Pre-score | 0.89 | 0.10 | 94 | −0.02 | 0.88 | 0.09 | 110 | 0.00 | −0.02 | −0.21 |
Post-score | 0.87 | 0.10 | 0.88 | 0.09 | |||||||
SDQ b | Pre-score | 15.22 | 3.96 | 85 | −0.18 | 15.23 | 4.75 | 92 | +0.76 | −0.94 | −0.22 |
Post-score | 15.04 | 3.95 | 15.99 | 4.25 | |||||||
QoSL c | Pre-score | 3.48 | 0.33 | 69 | −0.12 | 3.28 | 0.47 | 70 | +0.04 | −0.16 | −0.39 |
Post-score | 3.36 | 0.45 | 3.32 | 0.38 |
Costings of FAIP | ||||
---|---|---|---|---|
Cost Inflated to 2022–2023 Prices | ||||
Units | Cost | |||
Training day 1 | GBP 250 per teacher | 342 | GBP 85,500 | |
Training day 2 | GBP 250 per teacher | 303.5 | GBP 75,875 | |
Review meetings 1 | GBP 125 per teacher | 308 | GBP 38,500 | |
Review meeting 2 | GBP 125 per teacher | 257 | GBP 32,125 | |
Tier 1 Showcase event | GBP 125 per teacher | 300 | GBP 37,500 | |
Final showcase a | GBP 0 | 243 | ||
Staff costs | ||||
Project manager (payments per day) | GBP 350 | 70 | GBP 24,500 | |
Presenter and lead advisor (payments per day) | GBP 350 | 25 | GBP 8750 | |
Six regional advisors for eight days | GBP 350 | 8 | GBP 16,800 | |
Five extra staff project members, | GBP 350 | 1.5 | GBP 2625 | |
Tier 1 teachers (lead and host review meetings) | GBP 13,500 | 2 | GBP 27,000 | |
Tier 1 teachers for training days | GBP 5250 | 1 | GBP 5250 | |
Expert trainer | GBP 3000 | 1 | GBP 3000 | |
General support of school improvement advisers with schools (1 day per school) | GBP 350 | 193 | GBP 67,550 | |
Administration days | GBP 103.13 | 50 | GBP 5156.50 | |
Resources | ||||
Venue (2 full days and 2 half days) | GBP 38,189 | 1 | GBP 38,189 | |
Access to expert trainer platform | GBP 250 | 1 | GBP 250 | |
Printing training materials | GBP 1611.73 | 1 | GBP 1611.73 | |
Filming | GBP 1648.00 | 1 | GBP 1648.00 | |
Translation (materials and in person translation on training days) | GBP 5132.93 | 1 | GBP 5132.93 | |
Total | GBP 476,963 | |||
Teacher costs b | ||||
Time (time cancelled out by time saved) | GBP 0.00 | |||
Books | GBP 355.00 | |||
Materials | GBP 0.00 | |||
Total | GBP 355.00 | |||
Intervention cost | Total | GBP 477,318 | GBP 584,818 | |
Number of pupils exposed to the intervention c | 8075 | |||
Cost per pupil | GBP 59.11 | GBP 72.34 |
Sensitivity Analysis 1: Class size and cost per pupil | |||||
Units | Cost per pupil (2018–2019) | Cost per pupil (2022–2023) | |||
20 | 6460 | GBP 73.89 | GBP 90.73 | ||
30 | 9690 | GBP 49.26 | GBP 60.08 | ||
Sensitivity Analysis 2: Out-of-pocket expenses and cost per pupil | |||||
GBP 51 × 323 + GBP 584,818 (programme costs) | GBP 51 | 323 | GBP 74.46 | ||
Sensitivity Analysis 3: Buying out teacher’s time and cost per pupil using BAU | |||||
Units | Cost of supply | BAU cost | |||
Training day 1 | GBP 250 | 342 | GBP 85,500 | GBP 146,202 | |
Training day 2 | GBP 250 | 303.5 | GBP 75,875 | GBP 129,532 | |
Review meetings 1 | GBP 125 | 308 | GBP 38,500 | GBP 65,835 | |
Review meeting 2 | GBP 125 | 257 | GBP 32,125 | GBP 58,781 | |
Tier 1 Showcase event | GBP 125 | 300 | GBP 37,500 | GBP 64,125 | |
Total BAU cost | GBP 464,475 | ||||
Other costs (includes all costs to run the training events and GwE staff) | GBP 207,463 | ||||
Total cost per pupil | GBP 83.21 | ||||
Sensitivity Analysis 4: Opportunity cost of attending the showcase event. | |||||
Cost | Unit | Programme cost | |||
Two hundred forty-three teachers attending the 3 h showcase event using BAU rate (GBP 57 per hour) = GBP 41,553 GBP 584,818 + GBP 41,553 = GBP 626,371/8075 | GBP 171 | 243 | GBP 584,818 | GBP 77.57 | |
Two hundred forty-three teachers attending the 3 h showcase event using GwE half day supply cover rate (GBP 125) = GBP 30,375 GBP 584,818 + GBP 30,375 = GBP 615,193/8075 | GBP 125 | 243 | GBP 584,818 | GBP 76.18 | |
Two hundred forty-three teachers attending the 3 h showcase event and programme costs using BAU rate (GBP 57 per hour) = GBP 41,553 GBP 671,938 + GBP 41,553 = GBP 713,491/8075 | GBP 171 | 243 | GBP 671,938 | GBP 88.36 |
Programme | Effect Size | Cost per Pupil | Inflated to 2022–2023 |
---|---|---|---|
Switch-on | +0.24 | GBP 627 | GBP 802 |
Accelerated Reader | +0.24 | GBP 9 | GBP 12 |
Philosophy for Children (P4C) | +0.12 | GBP 16 | GBP 21 |
Fresh Start | +0.24 | GBP 116 | GBP 148 |
Literacy software | −0.29 | GBP 10 | GBP 13 |
Response to intervention (RTI) | +0.29 | GBP 175 | GBP 224 |
Summer school 2013 | +0.17 | GBP 1370 | GBP 1752 |
Summer school 2012 Year 7 | −0.02 | GBP 1400 | GBP 1791 |
Summer school 2012 Year 6 | −0.14 | GBP 1400 | GBP 1791 |
FAIP a | +0.12 | GBP 59.11 | GBP 72.34 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Tiesteel, E.; Watkins, R.C.; Stringer, C.; Grigorie, A.; Sultana, F.; Hughes, J.C. Where Are the Costs? Using an Economic Analysis of Educational Interventions Approach to Improve the Evaluation of a Regional School Improvement Programme. Educ. Sci. 2024, 14, 957. https://doi.org/10.3390/educsci14090957
Tiesteel E, Watkins RC, Stringer C, Grigorie A, Sultana F, Hughes JC. Where Are the Costs? Using an Economic Analysis of Educational Interventions Approach to Improve the Evaluation of a Regional School Improvement Programme. Education Sciences. 2024; 14(9):957. https://doi.org/10.3390/educsci14090957
Chicago/Turabian StyleTiesteel, Emma, Richard C. Watkins, Carys Stringer, Adina Grigorie, Fatema Sultana, and J. Carl Hughes. 2024. "Where Are the Costs? Using an Economic Analysis of Educational Interventions Approach to Improve the Evaluation of a Regional School Improvement Programme" Education Sciences 14, no. 9: 957. https://doi.org/10.3390/educsci14090957