Russell Turner is an independent consultant in social evaluation for the public and third sector. He has worked with national organisations, such as the probation service, as well as local level projects to help them both ‘do’ and use evaluation to improve their practice. Visit www.rdtconsultancy.se for more information.

It’s increasingly important for organisations and charities to demonstrate the impact of their services and projects. This can often mean buying in ‘expert’ evaluation. But how do you know whether you’re buying a good quality evaluation? How do you know your money has been well spent?

TIME for a successful evaluation
You don’t need to become a social impact expert overnight to ensure that the evaluation you buy is of decent quality. In my experience there are four key criteria for a successful evaluation, easily remembered with the TIME acronym: Theory, Implementation, Measurement, and Extend & use.

Theory – the analysis needs to outline the theory of the project or intervention. Without some basic understanding of how the intervention is expected to bring about change, you can’t evaluate whether it has done so. For example, a sports project aiming to reduce youth criminality would probably have some underlying theory behind what they do, such as young people through sports involvement build up their self-esteem and develop a positive pro-social identity. An evaluation that just looks at changes in re-offending would not be sensitive to whether the project itself is working.

Implementation – put simply, the analysis needs to determine whether the project got off the ground. Many projects take time to get going and can be beset by all kinds of operational issues, such as funding, staffing and so on: The theory may be great but the practice didn’t (or hasn’t yet) worked out. A good analysis should tease apart theory and implementation to tell you how, where and what you should be developing to get your project effective.

Measurement – Think of ‘measurement’ in the broadest sense with numerical data and human stories having equal place. A good evaluation should use a measure appropriate to your project and its underlying theory, as well as your budget! A successful analysis also needs to use the right measure in the right way. Admittedly, this can require specialist knowledge about analysis design, but often questions like “Why use this measure with just these people (and not those)?” will get you on the road to a successful analysis.

Extend & use – A good evaluation is only as good as its use! If the evaluation cannot be extended to practice, applied to developmental work or used positively in some other way, then it has failed. Admittedly, this sometimes sits outside both the organisation’s and the evaluator’s control. But only sometimes! At the outset, what was the plan for using the analysis?

Three easy things you can do to get a good quality evaluation
I said you don’t need to become an expert overnight to ensure that the evaluation you buy is of decent quality. So what can you do?

1. Get smart from the start – When you commission an evaluation, ask tenderers to address the ‘TIME’ criteria noted above. Get them to be clear about how they will avoid ‘false positives’ – glowing reports based on just your successful cases – and ‘false negatives’, such as using measures not sensitive to your project’s ‘theory’. There are simple solutions to these problems that needn’t be costly. Before issuing Invitation to Tender documents, plan for how you would use the evaluation results; what sort of findings do you really need and when? Don’t be afraid of seeking external advice on your tender documentation; one or two days’ support from a good evaluation specialist upfront could save you a year’s worth of consultancy being misused in the long run.

2. Be a critical friend to your evaluator – A good evaluator knows their stuff and likes a discussion about their approach. Ask them questions about their methods: how are they proving that the project has caused the impact? Why are they using that measure and not another? Sometimes the more basic a question is, the better. I often offer to hold mini evaluation workshops for staff involved in projects I’m evaluating. Not only does it foster their engagement and increase their understanding of evaluation, it keeps the evaluation real and credible, both in my eyes and for them.

3. Peer review at the post – Seek objective advice from others in your industry or by an independent evaluator on an early draft of your report. This can identify potential problems which can often be rectified – and still within the existing contract with your evaluator! This not only saves you time and money, but also potential harm to services. Budget for a couple of extra days of consultancy time for an independent review of the report. Again, it’s a small investment when the actual savings are considered in getting a good quality evaluation that makes a real difference to your project.