Evaluation is the thing

Here’s a definition of evaluation from that first blog:

“The essence of all learning, the essence of teaching, the essence of all systematic improvement has, at its core, careful measurement of where things are in relation to where they should be”.

The notion seems an easy one. However, I seldom find leaders in education who find it easy to put into practice – not in the rigorous, concrete way that I envision, at least. School leaders talk to me regularly about the difficulties they encounter in, what I would term, being evaluative. One possible reason for this is that quality evaluation has not traditionally been the culture of how schools do things, so it has not surfaced as a key skill for many educators. Let me give you an example of the difficulties I see schools facing.

I see a lot of fuzziness in the culture of education. Sometimes it is difficult, even, to get a clear definition of what we want our students to learn at school. Our much lauded NZC is strong on big aspirational statements about what we want students to learn but not so strong on how that learning might happen and how we will recognise when they have learnt it. Some schools have made a very good fist of really defining the learning outcomes to the extent where sensible measurement and evaluation can take place. However, many schools express difficulty in doing so, and struggle to define the ‘total package’ of values, competencies, skills and understandings they want their students to leave with. And without a definition of these, it is impossible to evaluate the extent to which they do leave school with them.

So clarity about what we want students to learn is a first step to quality evaluation. But it is not the only step. Here are a few other hazards that schools encounter in their endeavours to be evaluative, and my advice for overcoming them.

Firstly, many schools think of evaluation as a function of the senior management. It is not. A school which is good at evaluation has students, teachers, leaders, parents and board members who are good at evaluation. Do students at your school know how to carefully measure where they are in relation to where they should be? How about the teachers? The leaders? Quality evaluation is dependent on quality information, and schools need quality information from everyone if they are going to plan for improvement that makes a real difference.

Secondly, many schools rush evaluation. They rush their analysis of data and they rush into solutions before deeply understanding the problems. For example, even when respectable tools are used for measuring something like reading progress, there can be a tendency to just plot the results against a normal curve, spot a few kids lower than they should be, note them for ‘targeting’ the next year, write a report to the Ministry and board of trustees and then repeat the exercise the following year. That’s a trivialised description of what schools spend a tremendous amount of time doing. However, my point is that this planning and reporting cycle may not, despite best intentions, be careful evaluation carried out with improvement in mind. A school staff that is good at evaluation studies – really studies – achievement data and other information, and often finds it needs even more information before it can confidently say it understands the problem and can begin to consider solutions. It also tracks information carefully over time. Longitudinal information is often very revelatory.

A third hazard is the tendency to create solutions to problems that are not necessarily valid solutions. I have talked with many schools with plans to ‘fix’ a student achievement problem. They’ve devised new programmes, created new resources, established new timetables, employed new support staff, and discovered at the end of the year that little improvement has occurred despite the enormous effort. A school that is good at evaluation critiques proposed solutions rigorously. It critiques the assumptions that lie behind the proposed solutions. It critiques the evidential base for the proposed solutions. If it is very wise, the school seeks the critique of an external party – Evaluation Associates, for example!

So there are a few things that I’ve been thinking about which may help schools to improve their evaluative capability. This is not all there is to evaluation, but it’s a great start. Leaders, teachers and conference academics talk a lot about innovation, creativity and continuous improvement. All of the talk is hollow unless we evaluate properly to see if our innovations are improving student outcomes, unless our creativity is enabling us to sharpen our picture of what our accomplished young citizens know and can do, and whether it all does continue to improve each year.

If you have any questions about this article

Contact Michael Absolum ONZM