Unlocking the power of evaluation to increase school effectiveness

Nearly a year ago I joined the Evaluation Associates whānau as an evaluator. I have noticed the tendency for the eyes to glaze over after I tell someone what I do, but there is a lot about evaluation that I find compelling and that I believe can make a significant contribution to improving education outcomes.

Much of my work over the past 20 years has been in the field of evaluation, working within and alongside a range of organisations, government departments, iwi providers, community providers, and schools to foster and enable a culture of continuous improvement. My experience has been that evaluation has huge transformative potential. Evaluation is not merely a research methodology; it encompasses a mindset and a comprehensive toolkit that empowers individuals and organisations to make informed decisions, develop tailored strategies and interventions, and to innovate in pursuit of excellence.

The educational literature consistently indicates that school improvement requires ongoing and robust internal evaluation, and that there is considerable variability across schools in these practices. Hood and Mayo (2018), in their discussion about innovation and improvement in education, advocate for a system that emphasises the need to build capacity within the sector to learn and develop continuously. This view is further supported by the Review of Tomorrow’s Schools in 2019 (Tomorrow’s Schools Independent Taskforce, 2019). The findings and recommendations from this review have led the Educational Review Office to change its approach from an external evaluation model to a more participatory, collaborative approach with increased emphasis on evaluation capacity building and school improvement. This new model, Te Ara Huarau, aims to ensure that through effective internal evaluation processes all schools are on an improvement trajectory (Goodrick, 2022).

As the national rollout of the Te Ara Huarau model progresses, it is highly likely that schools will have an increased focus on developing effective internal evaluation practices. The scope and context for internal evaluation will vary, from a policy review, a strategic initiative, or in relation to an unforeseen issue, and such work will require both school leaders and teachers to draw on a range of evaluative skills. In this blog I share some actionable steps that could be taken to build internal evaluative capability and practices within schools.

1. Learn to value evaluation

Motivation to engage in and further develop evaluation practices is a critical first step. Research on internal evaluation and evaluation capacity building identifies that attitudes and emotional investment from staff are foundationally important. A key finding from a study looking into evaluation capability building in the early learning sector suggests that prior to looking at ‘how to build capability’ it was more important to explore ‘why would we want to?’ (Moretti, 2021). Motivation for evaluation and monitoring should increase if teachers can see the relevance to teaching and learning and improved outcomes. This exploration can also debunk misconceptions related to quality internal evaluation practices i.e. it doesn’t have to be big, complex, overly technical, compliance-focused, or onerous.

2. Develop an evaluation framework

Foundational to a well-embedded culture of evaluation is conceptualising the school as a system and building evaluation and monitoring into the system. Monitoring is generally information gathering that occurs while an activity or intervention is underway to track progress and implementation. It tends to be regular and routined. Evaluation is seen to take a more in-depth look, in order to assess effectiveness or impact and determine overall success and areas for improvement. However, the boundaries between the two can blur, depending on the evaluation methodology.

Schools in the early stages of establishing evaluation capabilities may have an ad-hoc approach, with multiple data-gathering efforts occurring at various levels (board, SLT, classroom, students) without an overarching or coherent structure. A key feature of a systems approach involves creating a framework for Monitoring, Evaluation and Learning (MEL) which can assist schools to map data collection activities across their strategic plan and reporting cycle, focusing on key questions or desired outcomes. It should also ensure data is not collected unless there is a clear intention to use it. This frame will give the board and school leaders a clear picture of what information will be available to demonstrate effectiveness, to inform decision-making, and to tell the stories that are important to them and their community.

3. Build on existing inquiry-based practices in schools

The concept of inquiry-based approaches to school improvement is familiar to most teachers in New Zealand schools. These approaches, which share common skills with evaluation methodologies, can offer a good starting point and provide a structured approach for examining school systems, plans and practices. Potential areas of focus should be grounded in the unique needs, challenges, and goals of each community.

4. Develop a Theory for Change

Another useful way to further advance evaluative thinking in schools is to encourage staff to develop a Theory for Change (also known as a Theory for Improvement, Logic Models, and Programme Theory) to describe a new initiative or programme. A Theory for Change is usually a ‘one-pager’ that succinctly and visually maps out the identified problem, the planned change strategies, system shifts, and the anticipated outcomes and impacts. While it is a useful step in intervention or programme planning and in getting everyone involved on the same page, from an evaluation perspective it can also reveal suitable quality indicators and ensure that data collection focuses on relevant aspects of the initiative's progress and impact.

5. Use Key Evaluation Questions

The development of Key Evaluation Questions (KEQs) ensures the purpose of self-review or evaluation is clear and they can be used to help structure regular reflection and guide reporting. They are big picture, guide the evaluation, and are focused on what you really need to know. It's easy to get lost in the details of monitoring and evaluation if you do not have a clear sense of what you are trying to find out.

The KEQs should be specific enough to help focus the evaluation, but broad enough to be broken down further into more detailed questions to guide data collection. KEQs usually inquire into some, or all, of the following:

• How well designed is the initiative?

• How well implemented is the initiative, and what contributes to that?

• What particular aspects of the initiative are beneficial (or not beneficial)?

• What are the views and experiences of the participants/stakeholders?

• What are the impacts, outcomes, or changes, and to what extent do they meet the expectations/goals of the initiative?

• What works best for whom, under what conditions, and why/how?

Good evaluation practice will incorporate questions that are explicitly evaluative and assist in data being used to determine some sort of value or merit. Such questions may use wording such as how well, and to what extent.  

6. Emphasise the ‘Now what?’

A common way of framing the evaluation process is:

1. ‘What is so?               -  the state of things

2. ‘Why is it so?              -  the factors that contribute

3. ‘So what?’                   -  why it matters/its value

4. ‘Now what?’               -  what happens next

The final step in the evaluation process – “Now what?” – brings the improvement orientation. “Now what?” should involve questions like:

• Are these results good enough?

• What have we learned?

• What needs tweaking?

• Why are we doing this?

While data provides important insights into a situation it does not generate answers or solutions. Asking how this information will inform decision-making and action for change and improvement is important. It is these conversations that will generate the actions that will lead to better outcomes.

To conclude, evaluation is a crucial enabler of continuous improvement and innovation. By seizing the opportunities to increase the capacity and capability of schools in utilising evaluation effectively, we can assist in further growing a culture of learning and progress. Embracing these efforts will undoubtedly contribute to the betterment of education and lead to positive outcomes for learners, educators, and the wider school community.

References

Moretti, E. (2021). Exploring what motivates evaluation capacity building in early learning services: “what are the children getting out of this?”. Evaluation Matters – He Take Tō Te Aromatawai, 7, 71-95. https://doi.org/10.18296/em.0067

Goodrick (2022). Schools evaluation for improvement approach: implementation case studies. Wellington, Education Review Office. https://ero.govt.nz/how-ero-reviews/how-ero-reviews-schoolskura-english-medium/te-ara-huarau-the-new-approach-to-evaluation/feedback-on-the-new-schools-operating-model

Hood, N., and Mayo, S. (2018). The quest for scale: achieving system-wide innovation and improvement in education. Wellington, The Education Hub. https://www.theeducationhub.org.nz/wp-content/uploads/2018/04/The-Quest-for-Scale.pdf

Tomorrow’s Schools Independent Taskforce. (2019). Our schooling futures: stronger together | Whiria ngā kura tūātinitini Final report by the Tomorrow’s Schools Independent Taskforce. Published by the Ministry of Education. Wellington. https://conversation-space.s3-ap-southeast-2.amazonaws.com/Tomorrows+Schools+FINAL+Report_WEB.pdf 

Keen to find out more? Contact Dr Melanie Atkinson today to learn more.

Contact Dr Melanie Atkinson