The iceberg has melted, long live the evaluation duck

by Kieron Kirkland, Nominet Trust

Getting your head around evaluation and social impact assessment (evaluation’s sexy stage name) can be a complicated process.  People often aren’t clear about why they are engaging in evaluation or what they want from it, this can lead to problems like people measuring the wrong things and not getting the answers or evidence they need (if you need a reminder, see here for my list of the top 10 reasons to evaluate ).

That’s why analogies and models to explain the purposes and process of impact assessment and evaluation are invaluable.  Sometimes people talk about the ‘tip of the iceberg’.  However, I think that’s got a limited value (for starters Icebergs don’t go anywhere, and were partially responsible for Celine Dion singing ‘my heart will go on’).  Others dive straight in with complicated diagrams.  But these can put people off and confuse matters further.

That’s why by far the most useful analogy I’ve thought of so far has been a swimming duck (go with me).  Aside from being much cuter than an iceberg, a swimming duck is useful to illustrate the 2 key roles of evaluation: assessing change, and understanding how that change happens.

Changes ‘Above the line’

 In looking to evaluate anything we ultimately have to show some evidence of movement (how far the duck travels across the surface of the water). Without measuring movement we cannot hope to judge the success of the model.  This is typically the ‘distance travelled’ that we talk about in outcomes measurement.  No matter how small, there has to have been a specific change or movement in a desired direction. For example, people have developed an increased level of skill or reduced social isolation. To assess this we use outcomes measures, baselines, distance travelled questionnaires and so on. This is useful both to validate the model and communicate its value to stakeholders.

Below the line

 But that’s only half the story. If you want to have any chance of continuing to develop your work in an effective manner or to engage in a more meaningful evaluation process, you have to know what’s going on underneath the surface (how the duck is moving).  You have to understand how that change has happened.  Without this you cannot have an informed development of your practice.

That’s where we come to what’s happening below the surface - how has that measured movement been created, what’s stopping or helping it that you can’t immediately see?

Here those outcomes measures won’t help you.  You need different captures that delve deeper.  These may include a range methods and approaches.  They may be informal, such as team discussions to share practice, or more structured such as qualitative interviews with beneficiaries.  Increasingly at Nominet Trust we’re interested in how data can allow us to see what’s hidden in the depths and not visible to the naked eye.  For example, open data sets can offer information to better understand the context (or in duck speak ‘murky depths’) in which you operate (for more information see our open data and Charities State of the Art review).  Our digital storytelling work can allow collection of large sets of unstructured data to surface patterns of behaviour across a large number of participants and settings (more information here). Lastly, there is the potential of big data to surface patterns - which is comparatively underexplored in evaluation of social projects.

So next time you talk about your evaluation or impact assessment, think about the duck.  What measures are above the line which attempt to capture the success of your model for communication?  And what mechanisms have you got below the line to help you understand how that change is happening?

Without both, you’re only seeing half the picture.

Kieron is the Development Research Manager at Nominet Trust. This article was originally published on his blog. You can watch a video of the Evaluation Duck here.