The numbers are telling. While 86.9% of organisations in South Africa’s development sector consider impact measurement a crucial process, a whole 61.7% of them recognise that they don’t do enough of it. Let’s think about why this may be. And while we’re at it, let’s look at some impact-measurement case studies and practical tips on doing it right.
It looks like a contradiction: 86.9% of development organisations think measuring impact is crucial, but 61.7% say they don’t do it enough. This is understandable, to a point, if we consider that people rarely enjoy being scrutinsed – the closer the scrunity the less enjoyable it’s likely to be.
In the case of non-profit organisations, there is also the added fear of losing funding due to an evaluation’s results. Those that are willing to step up and face this fear often find themselves put off by the difficult nature of measuring what is often considered “unmeasurable”, whether due to the scale of a project, the subject matter or the high expectation of required results.
But, if you’re serious about your work, you need to find the means to measure your impact and report it well. It is as crucial to success as the 86.9% think.
Starting the makeover
Today, measuring impact is systematic, targeted and the only way to progressively improve. It has to be. Without the right data and an informed and methodical approach, you won’t be able to learn from the evidence and strengthen your intervention. Even if you’re relatively certain of your impact, evaluations can help you enhance, expand and scale up your model by clarifying what works and what doesn’t. But be prepared to answer some tough questions and accept the numbers for what they are – and what they imply.
Firstly, look at what you’re already doing in terms of impact evaluation. Find out where the project team is currently collecting data, and establish existing points of contact with beneficiaries, stakeholders and partners that can become spaces for collecting more.
Start with an impact mentality, not an activity mentality. Why? Because, “activities reflect busyness,” says Marcus Coetzee, coordinator of the African Social Enterprise Network. So rather keep your focus on how the programme is impacting recipients’ lives. For example, instead of measuring the number of women who are members of a cooperative, measure the income they take home every month and how it affects their regular expenditure.
Allocate responsibilities within your organisation or team. Ensure that one or two people are given the specific task of managing data-collection, analysis and reporting. Although impact measurement is a collaborative process, it will only happen if the director sets the example and buys into the idea.
When an evaluation reveals that you need to do more
When the Tongaat area branch of educational support organisation PROTEC decided to hire an external consultant to build an M&E system for their maths, science and computer supplementary school programme, they were faced with an unexpected result.
The aim of the evaluation was to establish the number of programme youth who went on to graduate from higher education and find jobs in science and engineering. But, in addition to measuring the programme’s impact, the consultant co-developed a Theory of Change with PROTEC’s team (we’ll take a look at the importance of this planning tool shortly in this piece).
After much effort to contact past learners, the study revealed that:
1 Many of the learners, who had promising results in secondary school and matric, were dropping out of university due to non-academic reasons, such as being away from home and lacking support.
2 PROTEC’s programme did not draw sufficient links between short-term and long-term outcomes. Basically, the supplementary support provided up until matric wasn’t enough to ensure that learners made it through their studies and ultimately got a job.
The result of the evaluation? PROTEC reviewed their intervention and created a post-matric programme to support students through university and into the world of work. In this way, they were able to increase the number of learners entering science and engineering fields. They were also able to communicate this fact to their partners and funders – data is even more impactful and useful when shared.
The most crucial step: defining your Theory of Change
If you take a look at your project, you will realise you’re already collecting valuable data that just needs to feed into a model – namely, your Theory of Change – and then be systematically gathered and shared with your stakeholders. A Theory of Change (TOC) will help you to capitalise on your existing data and collect it in a more systematic way. It is a useful tool for developing solutions to complex social problems.
A basic TOC will define the early and intermediate building blocks required to bring about a long-term goal. A more complete TOC will articulate the underlying assumptions necessary for a change to occur and explain how that ultimate change will be achieved by plotting the causal links between activities, outputs, short- and long-term outcomes.
The value of developing a TOC as part of your planning and evaluation is that it creates a commonly understood vision of the programme’s long-term goals – including how they will be reached and what will be used to measure progress. In other words, if you want to measure impact effectively, defining your TOC is the place to start.
SIX STEPS TO A THEORY OF CHANGE
1 Identify the long-term goal.
2 Conduct “backwards mapping” to identify the preconditions necessary to achieve that goal.
3 Define the process or sequence of change anticipated.
4 Identify the interventions that you want to be responsible for, to create these preconditions.
5 Develop indicators for each precondition that will be used to assess the performance of the interventions.
6 Create a diagram and a narrative that capture the various moving parts in your theory.
~ Adapted from Department for International Development (DFID) research, “Review of the Use of Theory of Change in International Development”