Resources » Topic » Beyond Measurement
Case StudiesEvent ReportsVideos
Books and Guides
L’Avise, l’ESSEC IIES et le Mouves ont publié un petit précis de l’évaluation de l’impact social.
Cette publication est le fruit de travaux collectifs de près de 20 structures rassemblées autour d’une même volonté : proposer des repères à toute entreprise sociale, association ou structure d’utilité sociale qui souhaite enrichir ses connaissances sur le sujet.
Qu’est-ce que l’évaluation de l’impact social ? A quoi et à qui sert-elle ? Comment la mettre en œuvre ? Quels exemples concrets ? Ce petit précis apporte des éléments de réponse et donne des repères indispensables pour évaluer l’impact d’une entreprise sociale.
This resource from Social Finance UK provides information on their experience in developing SIBs. This tool acts as a template for developing approaches to move more resource into prevention work. The report focuses on Children Services although will be useful for other local authority services areas where there is potential for significant social impact.
Social Finance is committed to providing a range of support for those interested in developing SIB proposals. This could range from full engagement through a detailed feasibility study of a particular intervention or issue area to help with specific parts of the SIB development process (see below for further details of this process). We are aiming to provide a set of tools to help minimise the costs of developing these products and we hope that this guide – which is intended to be freely available – is a useful start point.
The Good Analyst authored by Adrian Hornsby, presents a methodology for impact analysis, alongside a set of guidelines for measuring and reporting impact. The book also draws on Adrian and his colleagues’ broader experience of impact research to lay out an overview of where the practice is today, of how we got here, and theory of analysis itself. The Good Analyst incorporates the Investing for Good Guidelines for How to Measure and Report Social Impact and is accompanied by the Dictionary of Indicators.
In Happiness: Lessons from a new science Richard Layard demonstrates the paradox at the heart of our lives: Most people want more income, yet as societies become richer, they do not become happier. Scientific research shows this to be true. There are now sophisticated ways of measuring how happy people are, and evidence shows that on average people have grown no happier in the last fifty years, even as average incomes have more than doubled. This book is considered to be a leading resources in the field of “happiness studies”.
This report from The Good Childhood Inquiry was commissioned by The Children’s Society. The report, authored by Richard Layard and Judy Dunn, was the first independent national inquiry into childhood, started in 2006. Evidence was contributed by over 30,000 people, of which 20,000 were children, from polls, research and focus groups. The report includes recommendations from the panel to parents, teachers, the Government, the media and society in general.
Mission, Inc.: The practitioner’s guide to social enterprise, by Kevin Lynch and Julius Walls Jr, is a practical book that focuses on the day-to-day challenges and opportunities faced by social enterprise practitioners, to help create highly successful businesses.
This book from Barry Schwartz addresses a great paradox of modern life: Why is it that societies where individuals are offered more freedom and choice than ever before have higher rates of depression than ever before? Schwartz argues that the abundance of choice in western society today is making us unhappy.
This book, by William R. Shadish, Donald T. Campbell, and Thomas D. Cook, is a comprehensive text on experimental research. The four main topics are theoretical matters, experimentation, causation and validity.
Daniel Kahneman, Ed Diener and Norbert Schwarz have edited this collection of scientific research to answer the question of well-being, one of the most enduring and elusive subjects of human inquiry. The book is organised in five parts: How can we know who is happy?; Feeling Good or Bad: Pleasures and pains, moods and emotions; Personality and Individual Differences; The Social Context; and Biological Perspectives.
Case Studies
This resource is a French report from a consensus conference on homelessness in France.
In 2012, the National Offender Management Service (NOMS) commissioned NPC as part of a pioneering project to explore how providers can better measure the effectiveness of their work to rehabilitate offenders.
This publication from Nina Mguni and Nicola Bacon at the Young Foundation presents three case studies of the Wellbeing and Resilience Measure (WARM). The report consists of three parts: Defining local wellbeing and resilience, Constructing the Wellbeing and Resilience Measure (WARM), Applying WARM in three case study local areas and Selecting indicators and creating the measurement framework.
Event Reports
SIAA’s 2013 annual conference, Beyond Measurement, took place on December 10th at ESSEC Business School in France. This publication provides reflections on the day and access to further resources.
External Databases and Resources
J-PAL Europe was established in May, 2007 to expand J-PAL advocacy work in Europe and include European researchers in the J-PAL network. The office also manages J-PAL activities in the Middle East and parts of northern and francophone Africa. Based at the Paris School of Economics, J-PAL Europe works to improve the effectiveness of social programs world-wide by supporting researchers working on randomized trials and disseminating their results in order to provide policymakers with reliable information that can make their policies more effective.
J-PAL Europe is a regional office of J-PAL, a focal point for development and poverty research based on randomized trials.
The Agence Nouvelle des Solidarités Actives is a French non-profit organisation that puts into place local, experimental and innovative actions to fight against poverty and exclusion.
The Centre for Social Impact Bonds in the UK Cabinet Office provides information on Social Impact Bonds (SIBs), includes a knowledge box, information on funding, case studies and blogs.
The Education Endowment Foundation (EEF) is an independent grant-making charity dedicated to breaking the link between family income and educational achievement, ensuring that children from all backgrounds can fulfill their potential and make the most of their talents.
The EEF exists to fund, develop and evaluate cost-effective and replicable projects which address educational disadvantage.
Their focus is on supporting projects that show promising evidence of having a measurable impact on attainment or a directly related outcome. We are interested in testing projects’ effectiveness through robust independent evaluations, wherever possible as randomized controlled trials. If they are shown to have an impact, they should be able to be replicated and scaled up to improve outcomes for other disadvantaged pupils.
The Variable and question bank from the UK Data Service provides access and support for an extensive range of key economic and social data, spanning many disciplines and themes. It is an integrated service offering enhanced support for the secondary use of data across the research, learning and teaching communities.
This resource from the Ministry of Justice in the UK provides four rapid evidence assessments reports on intermediate outcomes and reoffending.
The reports are: Intermediate outcomes of arts projects, Intermediate outcomes of family and intimate relationship interventions, Intermediate outcomes of mentoring interventions and Intermediate outcomes of peer relationship interventions.
The Alliance for Useful Evidence is a partnership of The Big Lottery Fund, the Economic and Social Research Council (ESRC) and Nesta. They are an open-access network of more than 1,600 individuals from across government, universities, charities, business and local authorities in the UK and internationally. The website includes blog and publications about research and useful evidence.
The Dartington Social Research Unit is a charity that seeks to improve designing and delivering services for children and their families by promoting the increased use of evidence of what works. Their work spans education, health, social care and criminal justice systems. Their work involves data on children’s needs, information about what works, cost-benefit analysis and how money is spent at the local level. Projects include Investing in Children, A Better Start, Design and Refine and Into One Place.
People want to be happy. But do we know what makes us happy, or how society is best organised to promote happiness?
The Wellbeing Programme was founded in 2003 when Richard Layard gave his public lectures on “Happiness: Has social science a clue?” His book on Happiness then followed. The programme has expanded and now includes three main strands:
- Happiness and Public Policy
- Mental health
- Skills and unemployment
The Wellbeing Programme is also responsible for bringing together the members of the Mental Health Policy Group, which in June 2012 published its report How Mental Illness Loses out in the NHS, the subject of which Richard Layard discussed in his lecture “Mental Health: The New Frontier for the Welfare State”.
In 1996, the Center for the Study and Prevention of Violence (CSPV), at the Institute of Behavioral Science, University of Colorado Boulder, designed and launched a national youth prevention initiative to identify and replicate violence, delinquency and drug prevention programs that have been demonstrated as effective. The Blueprints for Healthy Youth Development project identifies prevention and intervention programs that meet a strict scientific standard of program effectiveness.
This blog by Alnoor Ebrahim, an associate professor in the Social Enterprise Initiative at the Harvard Business School, discusses when it makes sense to measure impacts and when it might be best to stick with outputs.
Geoff Mulgan, director of the Young Foundation, discusses the lack of consensus on what social value is and how to assess it despite the evident enthusiasm from funders, nonprofit executives and policymakers. he argues that when people approach social value as subjective, malleable, and variable, they create better metrics to capture it.
The Family Star Evaluation is a report is an independent evaluation of the Family Star outcomes tool. It seeks to engage parents and children in the work they need to do to change family life and measure and record their progress. The Family Star is highly commended by Charities Evaluation Services (CES).
The paper, by Geoff Dickens, Judy Weleminsky, Yetunde Onifade and Philip Sugarman, is a collaboration between Mental Health Providers Forum (MHPF) and St Andrew’s Healthcare. The study sought to explore the psychometric properties of Recovery Star to inform training and further development.
Specific objectives were to ascertain whether items on the tool measured a single underlying construct relating to recovery (internal consistency); to identify the nature of any underlying factors (factor validity); to identify any item redundancy; and to identify whether Recovery Star detects reported change over time (responsiveness).
The report findings indicate that the Recovery Star has high internal consistency and appears to be measuring an underlying recovery construct – providing further evidence for the use of the tool within recovery focused mental health services. In line with MHPF’s online analysis, it also demonstrates that the tool was capturing changes over time on most Recovery Star domains.
Videos
This is the conference video from SIAA’s annual conference on 10th December 2013, at ESSEC Business School in France.
Working Papers and Research
This report from Social Finance UK and the Centre for Global Development presents the findings from the Development Impact Bond (DIB) Working Group. The report explains how DIBs can enable more impact investment in development, by providing a shared platform for governments, donors, investors, firms and civil society to work together to achieve more.
Do you communicate data and information to stakeholders? These special issues of New Directions for Evaluation edited by Tarek Azzam and Stephanie Evergreen address data visualisation and evaluation. Part 1 introduces recent developments in the quantitative and qualitative data visualisation field and provide a historical perspective on data visualisation, its potential role in evaluation practice, and future directions.
It discusses Quantitative visualization methods such as tree maps, Sparklines, Web-based interactive visualisation, Different types of qualitative data visualisations, and a toolography describing additional data visualisation tools
and software, along with their major strengths and limitations.
Part 2 delivers concrete suggestions for optimally using data visualisation in evaluation, as well as suggestions for best practices in data visualisation design. It focuses on specific quantitative and qualitative data visualisation approaches that include data dashboards, graphic recording, and geographic information systems (GIS).
Readers will get a step-by-step process for designing an effective data dashboard system for programs and organizations, and various suggestions to improve their utility. The next section illustrates the role that graphic recording can play in helping programs and evaluators understand and communicate the mission and impact that an intervention is having in a democratic and culturally competent way. The GIS section provides specific examples of how mapped data can be used to understand program implementation and effectiveness, and the influence that the environment has on these outcomes.
Briefing 47: Barriers to employment from the Centre for Mental Health presents what works for people with mental health problems. Paid work is essential for the wellbeing and financial security for many of us. However, for those with mental health problems who require extra support there are numerous barriers to attaining employment. This report looks at what interventions work as well as where gaps exist in evidence-based interventions and what might be tested to develop that evidence. It includes models such as Individual Placement and Support (IPS) and some provided by the Work Programme and Work Choice.
Eibhlin Ni Ogain, Lucy de Las Casas, Marina Svistak co authored this publication from NPC’s work on Inspiring Impact. The report is about shared measurement, which involves charities working towards similar goals reaching a common understanding of what to measure, and developing the tools to do so. The report discusses the benefits and challenges associated with shared measurement, and through analysis of twenty approaches, examine how it is developed and draw lessons for future initiatives.
The Outcomes Star is a case-management and outcomes-measurement tool developed by Triangle Consulting Social Enterprise in the UK. This guide provides a practical approach to implementing the Outcomes Star that will ensure quality and consistency.
This working paper from Neil Reeder and Andrea Colantonio (LSE) provides an overview of the underlying concepts of impact investing as a form of socially responsible investment. Drawing on relevant literature, this paper casts a critical eye on the roles and responsibilities within measurement, making more explicit the subjective interpretation of social and environmental return (SER) by investors, and the clash of suppositions taken from other older measurement traditions.
The Framework of Outcomes for Young People by Bethia McNeil, Neil Reeder and Julia Rich and the Young Foundation is designed to highlight the fundamental importance of social and emotional capabilities to the achievement of all other outcomes for all young people.
It proposes a model of seven interlinked clusters of social and emotional capabilities that are of value to all young people, supported by a strong evidence base demonstrating their link to outcomes such as educational attainment, employment, and health.
It sets out a matrix of available tools to measure these capabilities, outlining which capabilities each tool covers, and key criteria that might be considered in selecting an appropriate tool – such as cost or the number of users.
It outlines a step by step approach to measuring these capabilities in practice, that is illustrated in four case studies that exemplify how the Framework might be used by providers, commissioners and funders.
The Development of the Outcomes Star by Joy MacKeith was originally published in Housing Care and Support: A journal on policy, research and practice. This paper describes the development process for the Outcomes Stars as a suite of tools which are designed to simultaneously measure and support change when working with vulnerable people as service users. It describes the original process of development of the first Star, in homelessness services in the UK; and subsequent roll out to other client groups and in other countries. The paper indicates the theoretical and philosophical under-pinning of an approach which aims to embody both research and values-based practice in empowerment and respect for the individual.
This working paper is by Alnoor Ebrahim and V. Kasturi Rangan, Social Enterprise Initiative, Harvard Business School.
Leaders of organisations in the social sector are under growing pressure to demonstrate their impacts on pressing societal problems such as global poverty. We review the debates around performance and impact, drawing on three literatures: strategic philanthropy, nonprofit management, and international development. We then develop a contingency framework for measuring results, suggesting that some organisations should measure long-term impacts, while others should focus on shorter-term outputs and outcomes. In closing, we discuss the implications of our analysis for future research on performance management.
This paper from Pietro Micheli and Jean-Francois Manzoni, published in Lone Range Planning, argues that the design of an strategic Performance Management (SPM) system and the definition of its roles are fundamental factors determining its success and impact on business performance.
Local Wellbeing: Can we measure it? by Mandeep Hothi, Marcia Brophy and Nicola Bacon for the Young Foundation, presents proposals for measuring well-being to support local authorities and their partners in the shift to Comprehensive Area Assessment, the new local government performance assessment framework.
This paper by Paul Dolan, Tessa Peasgood, and Mathew White discusses the “economics of happiness”, of which there is increasing interest, and provides a detailed review of literature on this topic. The evidence suggests that poor health, separation, unemployment and lack of social contact are all strongly negatively associated with SWB. However, the review highlights a range of problems in drawing firm conclusions about the causes of SWB; these include some contradictory evidence, concerns over the impact on the findings of potentially unobserved variables and the lack of certainty on the direction of causality. We should be able to address some of these problems as more panel data become available.
John Francis McKernan’s PhD thesis, Truth, Objectivity and Subjectivity in Accounting defends the idea that we can have truth and objectivity in accounting.
