How Do You Put Impact Measurement at the Heart of What You Do?

by Harry Hoare, Associate, Social Finance, UK

Measurement is important right? You wouldn’t be reading this in a SIAA publication if you didn’t agree. We hear it from policy-makers all the time too. So why have social programmes consistently failed to do it over the last twenty years? Why do governments still make commissioning decisions on a combination of hunches, guesswork and risk aversion?

This article explores three uses of measurement as part of a Social Impact Bond (SIB), an approach that makes measurement king.

Measurement is used:

1. Before the services goes live – to provide more resources to tackle society’s problems

2. During service – to drive better performance

3. After service – to build an evidence base

A SIB (http://www.socialfinance.org.uk/work/sibs) is a collaboration between government, investors and services providers to provide a new service: the investors’ money funds the service and the government pays them back (with a return) if agreed social outcomes are improved e.g. reduction in reoffending, improved employment etc. Clearly the investors care that the service achieves what it set out to, and that these improvements are captured so that they are able to get paid.

How do you persuade investors and government that this is a good deal for both of them?

At the heart of the SIB contract is one or more outcomes metrics – the terms that dictate whether payments are made and how much. In our work we’ve found that these metrics need to balance government’s preference for greater complexity to capture cost savings to the public purse with investors’ desire for more simple metrics that capture benefits to individuals. Investors (or “social investors”) are willing to put their money to use to promote social ends in order to promote positive change in the lives of those using the service, the ex-offenders or the job seeker. It is therefore crucial to monitor how well these services are working to improve their lives based on the different activities in which they take part and the workers who support them. Under a SIB lots of data is collected to enable analysis of which parts of the service are performance and where extra effort is needed.

For example, in Peterborough we saw many short sentence offenders had low level mental health problems that often went undiagnosed or untreated. As a response (and in order to help achieve the goal of reducing reoffending), we introduced mental health support as part of the overall package offered. Good performance can help inform the allocation of resources and identify service failures as soon as they develop.

Measure what is meaningful

In doing all this, we build an evidence base. Lack of evidence is the main barrier that prevents successful interventions going to scale. In order to deliver impact at a national scale, social organisations need a robust evidence base to show their programme works.

Warren Buffett found it ‘extraordinary’ that academics studied ‘what was measurable, rather than what was meaningful.’ Clearly, it is not more measurement; it is measuring what is meaningful and acting on it. As people working to achieve impact – to reduce reoffending, promote employment, get homeless people into stable housing – we are concerned with an essentially practical (not academic) endeavour.

We want to bring the best of academic rigour to the measurement of the meaningful.

Buffet pondered the adage, ‘to a man with a hammer, everything looks like a nail.’ Does there need to be a distinction between what is measureable and what is meaningful? I believe that to go beyond measurement we need to bring these two together. By measuring what is meaningful – and acting upon it – we can make the world a better place!

Further Reading:

Cabinet Office. Centre for Social Impact Bonds Knowledge Box: http://data.gov.uk/sib_knowledge_box/.

Social Finance. (Jan 2013) Technical Guide to Developing Social Impact Bonds.

Social Finance. (Oct 2013) Investing in Social Outcomes: Development Impact Bonds.

---

Harry joined Social Finance in January 2011 as an Analyst. He works on Social Impact Bond development in the fields of drug treatment and criminal justice. Harry holds an MPhil in Political Theory and BA in Politics, Philosophy and Economics from Oriel College, Oxford.

This article was first published in Beyond Measurement: Reflections on SIAA annual conference 2013. This publication provides reflections on the workshops, hotspots and interactive activities that took place at SIAA annual conference on 10th December 2013, and highlights the discussions and the ideas that came out of the day. Click here to view the publication.

Selecting the Appropriate Measurement System

by Simon Anderson, Social Asset Measurements

Socially oriented organizations, whether they are non-profits, foundations or social purpose businesses, seek, now more than ever, to articulate how they create social change. Understanding the change that is created is important as it assists staff in understanding their organization’s performance, where adjustments can be made, and enables them to showcase and promote their work to the public.

In order to develop the required assessment framework, an organization must consider its core goals and objectives. This process involves identifying the organization’s key activities and intended areas of influence. In addition, another important stage is to determine what kind of indicators to track, whether they are input, output, or impact oriented indicators. This is an interesting decision because there is not a “one size fits all” approach as the correct framework depends on several internal factors. The following will discuss this decision making process, drawing from Social Asset Measurement’s experience working with organizations across Canada.

Given the current interest in measuring impact, organizations are often drawn to developing an impact assessment framework. While measuring impact ensures that the organization captures the change it is creating, it should be recognized that in order to develop and implement a credible system, an organization needs to devote significant internal resources to the project. This issue highlights the first consideration when developing a measurement system: organizational capacity. While it would be ideal for all organizations to have dedicated staff for developing and implementing an impact assessment framework, in reality, this is not always the case. Therefore, if an organization recognizes that it does not have the resources for impact assessment, an alternative is to develop a framework that captures the outputs from their programs. While this does not provide information on the efficacy of the program, it may be a better option than pursuing an impact assessment that is implemented without adequate resources.

Another key consideration is the structure of the program in question. In some cases an organization may maintain strong connections with their program participants after the program is complete. If this is the structure, then capturing data on outcomes that materialize in the future is possible. However, in other contexts, organizations may lose contact with program participants, and while the infrastructure can be adjusted to facilitate ongoing connection, until that is achieved, it may be difficult to capture outcomes that that arise after the program is completed. If this is the case, indicators which capture change that occurs within the duration of the program should be selected.

The final element I will speak to is that of program type.  This issue affects whether an impact analysis can be extended to a social return on investment (SROI) analysis (where a financial value is applied to some of the indicators to enable the calculation of a benefit cost ratio). For some programs, such as employment training or education promotion, there are well defined financial proxies that can be used to assign a financial value to the indicators. However, in other situations, such as an after school art program, it is more difficult to identify a financial value. In these cases it is better to focus on the impact rather than force the assessment into a SROI analysis because using poor financial proxies taints the credibility of the assessment at large. To this end, when determining whether a SROI analysis is applicable, the nature of an organization’s programs needs to be considered.

It is important for organizations to capture data on their work to better understand their programming and how they create change. In doing so, it is an imperative to develop the most appropriate system for an organization as this ensures the organization has the capacity and required infrastructure to utilize the framework, and, so potential data collection opportunities are not left unused. What are your thoughts regarding the selection of a measurement framework? What needs to be considered? What have been your experiences? It would be great to hear your comments below!

---

 

Simon is the manager of Western Canada at Social Asset Measurements (SAM) where he works with non-profits and foundations to develop and implement impact measurement frameworks. Prior to joining SAM, he worked with Human Resources and Skills Development Canada where he focused on social finance policy issues.

Simon holds a Master of Public Policy (University of Toronto) and B.A. in International Development and Economics (University of King’s College).

Social impact analysis and measurement is the growing hot topic among professionals involved in civil society organisations, philanthropy and social finance across Europe. However it is important to ask the question, are we all living in a fantasy world where we think social purpose organisations will implement and use impact analysis to improve their work?

In this blog I will give some real life examples of organisations and individuals from across Europe engaging in social impact analysis on a practical level, to counter the view that this is all one big fantasy. These examples are drawn from working in partnership with members of the Social Impact Analysts Association (SIAA) over the past four months to run a series of partner meetings or SIAA Impact Group launches in Estonia, Bulgaria, Hungary, Ireland, UK and Spain. Across these countries, with their differing political, economic and social landscapes, there was a surprising level of engagement and consensus around three burning questions in this field.

The first question was, why should we do impact analysis in the first place?

At each event there was a sense that many of the attendees were already won over by the idea of doing social impact analysis, but that there is still a need to discuss the main drivers for organisational and individual engagement.

During a meeting at Philanthropy Ireland in May, it was claimed that NGOs in Ireland are primarily engaging with impact analysis in order to satisfy funders. A claim similarly held in the UK and backed up by a New Philanthropy Capital study ‘Making an Impact’ (2012) which found ‘52% of charities that have increased their measurement efforts say they did so to meet funders’ requirements’. However the research also suggested that the main benefit for charities was actually using the analysis to improve services.

At events in both Bulgaria and Hungary attendees heard case studies from Dr Andreas Rickert CEO of Phineo in Germany, an organisation carrying out independent research for donors based on analysing the work of NGOs. Andreas outlined Phineo’s due diligence process, including analysis of all organisational dimensions of a non-profit from internal management to monitoring systems. An evaluation of Phineo’s work showed that 90% of organisations who went through the analysis reported they could improve their work as a result.

How then do we ensure that impact analysis is used to improve services and increase social impact?

At the Ministry of Social Affairs in Estonia practical examples, such as the use of dashboards and IT systems to monitor and analyse the progress of an organisation, were discussed at length. Jaan Aps from the Estonian Social Enterprise Network explained that such dashboards can be used to integrate analysis into the everyday work of an organisation. Further examples were discussed by reference to a recent Estonian social impact evaluation handbook (Jaan Aps, 2012) which has provided much needed language and context specific guidance in this area.

Similarly, practical examples were shared at an event hosted by the Foundation for Development of Democratic Rights in Hungary. Jeremy Nicholls Chairman of SIAA and CEO of The SROI Network shared plans for the development of a social value budget by the FRC Group in the UK. Such a budget would be something that could sit alongside the financial budget in an organisation. It would be used by the management team to monitor, analyse, report and improve upon their work in collaboration with their board of trustees. Jeremy also made reference to the SIAA Principles Working Group’s mapping of existing principles of impact analysis and a SIAA Thought Paper providing a framework for ‘how an organisation can approach the practice of social impact analysis’.

Attendees and speakers in Hungary were very encouraged by such case studies, but were aware of the perception in Hungary that this type of analysis is too complex for the current political and economic context. In response to these concerns a number of speakers such as Eva Varga from NESsT explained that organisations can take baby steps, starting with a simple approach such as theory of change to go beyond just measuring outputs.

In Ireland it was emphasised that good work is already being done on a sector-specific level to analyse outcomes in a collective way. For example the Prevention and Early Intervention Network and Children’s Research Network for Ireland and Northern Ireland are some of the increasing number of organisations trying to facilitate sharing of data and analysis on a sector and system level to influence change. The message being that we need to stop constantly reinventing the wheel and look to how we can use shared measurement approaches and existing evidence and data to make better decisions and improve. Similarly in Bulgaria attendees were very interested to hear the mention of a UK programme Inspiring Impact and its blueprint for shared measurement case studies, such as the outcomes measurement framework Insights from the CAADA (co-ordinated action against domestic abuse) network.

Leadership and skills also came up as vital for organisations to use impact analysis. Without management and board members behind this work it isn’t hard to see how staff can lack the motivation and incentive to use impact analysis to review practice. In the UK Inspiring Impact has also been working on launching a Code of Good Impact Practice to encourage Impact Leadership.

The perceived lack of ‘impact leaders’ and the availability of professionals with the right skills and knowledge were apparent when it came to the discussion of funder involvement in impact analysis. The appropriate involvement of funders and alignment of expectations with investees was identified as a key challenge across all countries.

So, how can we educate and engage with funders and government to ensure this is not a top down conversation?

In Estonia one attendee described the “large dark matter between NGOs and funders” when it comes to communication around impact analysis. At each event attendees coming from both NGO and funder perspectives reiterated this frustration, suggesting there is an imbalance between funder requirements and what is useful and realistic for the organisation in question.

An example of a reporting system that is trying to redress the balance is the new German Social Reporting Standard (SRS). The SRS is a framework for self- assessment for NGOS to report on social impact as well as financial information to funders.The SRS was discussed at a number of the events and met with huge interest especially after hearing that the German Ministry of Social Affairs is now debating whether to adopt this as the primary reporting standard for their work with NGOs.

In Spain at an event organised by the Stone Soup Consulting, Philanthropic Intelligence, Spanish Association of Foundations and SIAA, there was mention of useful practical guidance for funders and social investors. Two examples are The Good Investor and A Practical Guide to Measuring and Managing Impact from the European Venture Philanthropy Association. These guides were outlined as practical resources to guide funders around what is appropriate engagement with investees.

In terms of government involvement, in Bulgaria at an event hosted by the Tulip Foundation a presentation from Nadia Shabani from the Bulgarian Centre for Non-profit Law (BCNL) focused on the need for government to understand the benefits of engaging in impact analysis to help make better social policy decisions. After 15 years working on dynamic legislation, Nadia explained that many social policy models in Bulgaria have been adapted from other countries such as France and the UK. However they often don’t produce the desired outcome as additional effort is need to make the policies work within the Bulgaria context. Impact analysis is something that could enable government and NGOs to identify the barriers to successful implementation of policies, for example in the recent reforms in the child care system.

Similar discussions took place in Ireland and the UK, where local and national government is starting to embed more in depth analysis and evaluation approaches. In the UK the passing of the Social Value Act (2012) requiring ‘public authorities to have regard to economic, social and environmental well-being in connection with public services contracts’ clearly indicates a commitment towards better evidence based policy making. Similarly in Ireland the Irish Government Economic and Evaluation Service (IGEES) has been launched to send economists into government departments to ask key questions around the social impact of their work. It is still an open question as to how much influence these changes will make, but is progress nonetheless.

In Hungary, Estonia and Bulgaria there was also considerable discussion around the involvement of the European Commission, as EU funding contributes to a substantial percentage of funding for NGOs in each country. The challenges identified with this type of funding were that it often leads to over planning of programmes and the use of pre-set indicators which are not useful to the NGOs in their various political, social and economic contexts. In addition there are worries that social impact measurement is a new buzz phrase in Brussels, and that any developments in this area won’t be realistic or appropriate to the needs and organisational capacity on the ground.

Although no one had a quick fix to these challenges there was discussion of how organisations like Ashoka are based in various European countries, but are also able to work in Brussels to influence EU level changes around social enterprises. Many members and attendees suggested that as SIAA develops it might be a very useful vehicle through which analysts and frontline practioners can feedback on the best way forward for impact analysis and measurement issues on a European level. Consultation with analysts and a number of SIAA members is already happening through the GECES Social Impact Measurement Sub-group, to help advise on the European Commission’s Social Business Initiative. Consequently it is not an unrealistic goal for SIAA to play a bigger role in representing analysts at this level.

So, is this real life? 

Having covered a number of examples and new developments I would argue yes, this is real life. Although we have a long way to go there are some inspiring case studies emerging across Europe showing that impact analysis can be useful if everyone in an organisation gets behind it. The language emerging is that social impact analysis requires a particular kind of organisational ‘mind-set’ and ethos. Leaders and frontline workers need to be able to engage with analysts and use impact analysis to look at the work of the whole organisation, its strategic objectives and how they fit into the wider system. If done well it can be a real life way for organisations to improve their work in the service of society.

---

SIAA launched the first Austria Impact Group in November 2012. Three new country Impact Groups are under development in Estonia, Bulgaria, Hungary and Canada with plans for groups in Romania and Portugal in late 2013. Further mapping of the needs in Ireland, Spain and the UK are under way. Get in touch via [email protected] for more information.

---

This blog was originally written for Alliance Magazine.

The Association for Research in the Voluntary and Community Sector‘s (ARVAC) annual lecture was held Tuesday 8 May at the Wellcome Trust in London.

---

Aptly enough for an event titled “What counts? Who counts?”, all the speakers at this year’s Association for Research in the Voluntary and Community (ARVAC) lecture agreed that counting and measurement is crucial to proving social impact, but that in and of itself isn’t enough to illustrate it.

Allan Cochrane from the Open University warned that while a quantitative approach to impact offers benefits such as clear measures of success and failure, there is a danger in assuming everything can be counted. There is also the risk of counting for the sake of it, or of focusing on meeting targets instead of improving the quality of the work being done.

Partly for this reason, Allan spoke about the importance of storytelling to supplement figures. In the Q&A after the panel discussion, the Big Lottery Fund’s use of blogs in reporting was referenced as one way of providing context for a process of impact evaluation.

Sioned Churchill from Trust for London said softer approaches to impact measurement were more representative of the diversity and values of the sector. She warned that the influence of big funders and government makes it difficult for charities to push back about how and why they measure impact. It’s important that impact measurement is part of a learning and knowledge-sharing process which aims to improve performance; rather than being exclusively for reporting purposes

Emma Stone from the Joseph Rowntree Foundation pointed out that completely different reports could be produced for the same projects depending on whom carried them out - funder, grantee or external evaluator – and to what end. Whether impact analysis is seen as academic research, an integral part of grant applications or a marketing or organisational strategy tool, the result will vary enormously.

Interestingly, Emma said that JRF’s recent efforts to report on its own impact had made them more aware that the process – which they’d expected from their grantees for some time – could be intrusive and unsettling.

The message was that pairing stories with hard numbers isn’t a zero sum game; in fact, telling a story alongside hard data can create a far richer and more nuanced image of the impact an organisation makes.