Careful planning and evaluation in schools can help ensure energy is directed where it is most effective, writes Owen Carter of ImpactEd
At ImpactEd, we’ve been extraordinarily lucky to have Challenge Partners’ support as a founding partner for our organisation, helping us to realise our vision of addressing the evaluation deficit. In essence, we help schools to better evaluate the impact of what they do, so that they can more systematically prioritise those things that are making the biggest difference for pupils – including social and emotional skills such as wellbeing and self-efficacy, alongside academic indicators.
One of the early ways in which Challenge Partners shaped our thinking was around the importance of co-design. We might have a good idea for what we wanted to achieve, but would that map onto schools’ priorities? And how would we ensure that the way in which we supported change in schools was genuinely exciting and not that dreaded sound in schools of yet another new thing?
Part of our solution to addressing this was to bring together an early co-design group of 10 Challenge Partners schools who, as we developed ImpactEd, would pilot our approach and provide us with honest feedback to shape what we do. Now, as our work has grown across England and Wales, operating at trust and system level as well as individual schools, we continue to work closely with the schools in this group and the Challenge Partners network more widely to set our direction. So, what are some of the lessons that we have learned from this process?
Schools care about impact. But this tends to be about more than impact on grades alone.
We were clear from the start that for us impact had to take account of academic achievement, but couldn’t be confined to this. One of the early pieces of feedback we received from our co-design process was to include a sharp focus on aspects of practice which are sometimes harder to approach rigorously: development of pupil wellbeing, metacognitive capabilities or pupil self-efficacy, the kinds of competencies that we might look to build through enrichment or character development activities. This was a key factor in our decision to prioritise research into academically validated measures for a broader range of pupil attributes when starting up ImpactEd, a piece of work supported by Nesta.
For some of the Challenge Partners schools we work with, development of this wider range of skills is at the very centre of their mission. Vicky Edwards, from Keyham Lodge School, recently presented at a Senior Partners and Hub Managers meeting on how they have been developing a cross-curriculum programme aimed to develop pupils’ resilience, called ‘My Skills’. (More on this in the Challenge Partners 2018 annual report). As well as embedding routine pupil self-reflection on resilience via ImpactEd into the activities they are running, Keyham have also found these attributes to be significant predictors of attendance and measures of academic achievement.
In general, we’ve found this not to be an either/or conversation. It can be very easy to get caught up in dichotomies between knowledge and skills. But overwhelmingly across our partner schools we’ve found a pragmatism which acknowledges the importance and inseparability of both. For example, with Bengeworth CE Academy, we are now working to assess the impact of the full range of interventions they run – some of which have a social-emotional focus, some of which are academically focused, and many of which are both. There can be particular benefit to looking at a broad range of measures as this can help us to triangulate between different outcomes and try to unpick what is driving any change we might see.
The evidence agenda has to take school-to-school collaboration into account.
There’s probably never been a better time to be interested in research and evaluation in education. The work of the Education Endowment Foundation in particular has massively raised the profile of evidence in education and how we can use it to influence decision-making.
But even in our most research-engaged schools we’ve found that just engaging with the external research alone is rarely enough. Most schools are eager to answer the question not just of what works, but how it works, and who it works for, and in which settings. We’ve found that just the simple act of connecting the dots between schools who are working on and evaluating similar areas can be of immense value.
It’s encouraging to see several features in the education landscape reflecting this trend – for instance, the EEF’s focus on implementation and research schools, the growth and popularity of collaborative networks like Challenge Partners, and a number of trusts using their convening power to more systematically share insights within their schools.
However, we think this needs to go further. Within the Challenge Partners network we are looking to grow further at hub level so that as well as improving insight at school level on what is working and how, we can also do this increasingly between schools, and help schools target more impactful provision. And more broadly, we are looking at avenues for how we can systematically share some of the data and insights generated by the school-led evaluation we are facilitating. My hope is that over time this bottom-up and system-led evidence base can meaningfully interact with and help develop the kind of large-scale evidence that we see generated through research bodies and national trials.
Schools are surprisingly receptive to hearing about what doesn’t work, as well as what does
One of the dangers in talking about the word ‘impact’ in schools is the possibility for misuse. Often, this gets tied up with words like ‘demonstrating’ or ‘showing’. The inclusion of ‘impact’ after ‘intent’ and ‘implementation’ in Ofsted’s new inspection framework isn’t entirely helpful here either.
For the most part though, we’ve been delighted by how little demonstrating evidence to external bodies has featured in schools’ motivation for working with us. Clearly, where schools are investing time, money and energy, there is a need to assess the impact of that investment and ensure that sufficient scrutiny is given. However, the true potential of a meaningful evaluation process is longer term, in how it enables a continuous improvement process, where schools reflect on what is working well, reduce what it isn’t, and embed this way of working over time.
One of the major concerns we had when starting up ImpactEd was that schools would only want to continue to work with us if their evaluations showed positive findings. However, we’ve found this not to be true. Working with several schools in the Herts and Bucks Hub, for example, we’ve seen both projects that vastly exceeded the expectations teachers had of them, and those which (often counter-intuitively) had negative or negligible findings, spanning areas such as embedding retrieval practice, developing curriculum and implementing school-based study leave. In some ways the most productive conversations have come where things haven’t quite worked as expected – this can generate fruitful discussions as to why this was the case, casting further light onto practice.
This process was summed up well by Toby Sutherland, Headteacher of St Clement Danes School in Chorleywood, who noted that the exciting thing about implementing a more developed approach to impact evaluation “was not just about evaluating the impact of specific projects, but actually beginning to embed this way of thinking in our school more broadly”.
Indeed, in some of our longer-established school partnerships we’re seeing a transition to not just evaluating the impact of specific projects or interventions but to a genuine impact-led culture where, before a school starts doing something, they are deliberate and intentional in mapping out what it is expected to achieve, how they will know if they have achieved it, and what they will do as a result. This is only possible in an environment of trust and collaborative professionalism where it is accepted that sometimes things may not work, and this helps set the agenda for future improvement. That this echoes the Challenge Partners ethos around school improvement is no coincidence.
- ImpactEd will be sharing insights from more than 100 impact evaluations conducted across Challenge Partners schools at our 2020 National Conference. If you’d like to find out more about their work, or some of the collaboration opportunities available uniquely to Challenge Partners schools, visit their website or get in touch directly for more information at firstname.lastname@example.org.