Beyond Measurement: a change from what is so, to so what?

by Ruth Whateley, Social Impact Analysts Association

As the Manager of the Social Impact Analysts Association I regularly ask myself, what difference does engaging in impact measurement and analysis make to society? What is the point of collecting data to evidence and explain the social impact of our projects, programmes and policies if the information is not being used to change or learn something?

This may sound basic, but these are important questions. Many of us know that an evaluation report can end up on the shelf if it’s not fit for purpose or communicated in the right way. Here there is a distinction to be made between understanding what is so or what’s happening as result of our work, and the question of so what or who cares about this information and who will change something as a result? This distinction, made by one of our members, explains some of the rationale behind our SIAA 2013 conference – Beyond Measurement.

Beyond Measurement was hosted last week by ESSEC Business School in France. For me, a highlight was the positive response to the theme ‘Beyond Measurement’. Throughout the day we asked for views on what Beyond Measurement means to delegates and what challenges and opportunities ‘going Beyond Measurement’ could bring to the social impact analysis field. The following outlines some opinions and themes from the day.

Beyond Measurement is acknowledging different purposes

The variety of participants at the conference illustrated the contrast in needs of stakeholders who use or are involved in impact measurement. Clearly an investor will have different data and information requirements to a frontline case worker, as well as different uses for the measurement. But what do these different requirements mean for the development of appropriate, proportionate and useful impact measurement?

Within this frame Kate Ruff, PhD Candidate in Accounting at York University, Canada, hosted a workshop about ‘The Life of a Measure’. Kate drew attention to research from accounting theory, which indicates that people measure for all sorts of reasons and that the measurement, once taken, can be implicated in other uses regardless of its intended purpose.

Accounting systems have been referred to as ‘answer machines’, ‘learning machines’ ‘rationalization machines’ and ‘ammunition machines’. If we generalise this framework to social impact it would look like people use impact measurement in a variety of ways including: 1) to know where to invest for the greatest social return; 2) to learn to how to improve and deliver effective programmes; 3) to allow stakeholders to judge and hold organisations accountable; 4) to build a case for what society should be pursuing.

One insight from Kate’s workshop was that all measurement approaches, no matter what their purpose, can get implicated in any of these uses and others. It is not within the power of the practitioner or analyst to appropriate the measurement. That is in the hands of those who use the measurement. Beyond measurement is therefore about analysts acknowledging the different purposes and uses of impact measurement, and shaping their work and communication accordingly.

Beyond Measurement is improving communications

Along the theme of communication, Nic Bolto from The Difference magazine in Australia said that we need to ‘go further than being constrained by what we may find or what we may determine, to a whole other footing of communicating outcomes and influencing what it is we’re all here for…better social outcomes.’

Nic’s experience comes from starting up The Difference, an annual report card on Australia’s social health, covering a range of issues including child abuse, suicide and reoffending. It is a fantastic example of using a range of communications techniques to relay a complex set of messages about data that is available publicly. Presentation of information, such as through graphics, was seen as a vital component in communicating measurement in an appropriate way.

Similarly, during the plenary mind mapping sessions a popular answer to the question ‘what is beyond measurement?’ was the effective communication of impact and outcomes. One group outlined that in order to improve and evolve projects based on measurement results, we first need to improve communications with all stakeholders.

Beyond Measurement is understanding mindsets

Another recurring theme through the conference was the idea of ‘mindsets’. During plenary sessions it was flagged that an impact-orientated mindset was necessary to embed measurement in organisations. References were made to organisational learning culture, being open to change and sharing mistakes. Examples were also given around Impact Leadership and the important role both CEOs and board members must take in embedding impact measurement in an organisation’s work.

However, the topic of ‘mindset’ was also referred to in a second sense in the ‘A Clash of Mindsets’ workshop co-hosted by Joy Mackeith from Triangle and Neil Reeder from the London School of Economics. This workshop touched on how analysts themselves deploy different measurement techniques, which often reflect different views or mindsets depending on their analytical and engagement goals. For example, some analysts would argue that there are simple mechanistic ways of working out how people act and there is an absolute reality in the world to be discovered. Simply put, this is a ‘Positivist’ paradigm. Whilst others claim that human systems are messy and that there are deep-rooted behavioural systems we cannot fully understand, taking a more ‘Interpretivist’ paradigm. However, different analysts are likely situated along a spectrum between these two paradigms. It is interesting to see how analysts can take into account these different mindsets or paradigms when conducting impact measurement.

Moving Forward

As you can see there were a range of views around what we meant by ‘beyond measurement’ and, therefore, why and how we should attempt to go beyond measurement. Additional suggestions included discussion of common principles of analysis, shared measurement, collective impact, as well as increasing resources and investment into impact measurement. One point all agreed on was that progress has been made over the last ten years, but we still have a long way to go in terms of understanding how we can better embed useful impact measurement and analysis into practice.

So, my take away from the conference is that when anyone is thinking about undertaking an impact measurement or analysis, it is incredibly important to consider not just what is so, but also so what?

In early 2014 SIAA will be publishing a Beyond Measurement report including follow up thought pieces, articles and resources from the conference. SIAA has also relaunched their resources centre at the conference. Go to www.siaassociation.org/resources and send your resources to [email protected].