If you’re paralysed by the belief that your work is “unmeasurable” or by the fear of not being sufficiently accurate, you will never start the process of understanding your impact.
Choosing what to measure
Also, there is no one-size-fits-all approach to evaluation. Different development projects require different tactics – which is also true for their M&E. For example, two valid approaches to understanding the impact of two different development initiatives could be: 1) tracking the number of people permanently employed as a result of a particular training and 2) understanding the reputation and perceived strategic positioning of a think-tank. And, sometimes, identifying the most useful metric requires some deeper thought.
“Measuring behaviour change requires evidence that it actually does happen,” says Jeremy Nicholls of the SROI Network. “You could use a combination of techniques to achieve this: tracking over time, research that illustrates your activities do result in changes, and some intermediate steps to show progress towards this change. As ever, how much of each technique you use will depend on your audience and the purpose of your analysis.”
It is clear that the measuring process can (and should) be adapted to suit each organisation’s needs. This also applies to the scope and level of accuracy of the assessment. One of the toughest aspects of measurement is defining long-term impact and claiming the appropriate amount of credit for your project. Start small, collect data and seek to understand the direct impact of your work. There’s only one rule: be transparent about what you’re measuring and how.
Case Study
The Institute for Justice and Reconciliation (IJR)
Measuring the reputation and impact of a think-tank
Measuring impact is particularly difficult in cases where it happens over long periods, where it is indirect or the result of collaborative effort, and also where the impact is heavily influenced by external factors like political stability. This is arguably the case for the Institute of Justice and Reconciliation (IJR), an organisation that aims to build democratic, fair and inclusive societies in South Africa and across the continent.
To achieve this, the Institute produces high-quality research that fosters dialogue and provides practical tools for reconciliation at the level of small communities and countries as a whole. But the IJR is constantly reflecting on questions like: how do you measure the contribution to “building democratic, fair and inclusive societies?”; and how do you know you’re achieving your goals in the countries where you operate?
IJR also recognises that measuring their impact involves a multi-pronged approach. So, in addition to tracking outputs like the “number of publications” or “dialogue sessions held”, they’ve started to systematically record impact in the following ways:
1 Through case studies of most significant change over a period of time. These may describe the Institute’s efforts to, for example, change the education curriculum and teachers’ training to teach the history of the different peoples of South Africa.
2 Through the identification and tracking of proxies for impact. An assessment of the strength of IJR’s network of partners and their level of influence will speak to the organisation’s capacity to impact policy, especially in countries other than South Africa.
By systematically recording change over time and identifying key preconditions and proxies for change to occur, IJR is able to direct their efforts where they will be most effective, showcase their successes, and also learn from their challenges and limitations.
Show the right results
As much as engaging with beneficiaries and hearing their feedback is an essential component of an evaluation, it doesn’t answer questions like “who contributed to the change?” or “how sustainable is that change?”.
Once you have the planning in place and your newfound impact mentality intact, it’s time to start measuring and making the most of the results.
When showing results in a report, the most powerful way is to quantify impact in numbers, while telling the human side of the story simultaneously. Both kinds of results have merit and need to be shared.
The merit of numbers (quantitative data): Nowadays it’s all about evidence-based impact, and numbers provide the evidence of an intiative’s quality and reach.
The merit of human stories (qualitative data): In the end, people connect with people, so connect with your donors and partners by sharing stories. To truly communicate a project’s effectiveness you have to demonstrate the need and illustrate tangible impact. This impact is most powerfully communicated through the use of quotes, case studies, photography and video.
And remember…
Whatever the state of your data collection and impact measurement, consider these last words, and keep them top-of-mind…
Impartiality. External evaluations don’t need to happen every year but, when possible, the value of an impartial expert looking into your work is priceless.
Keep it simple. Don’t get bogged down by data and clog up your M&E pipeline. Stick to key questions and indicators.
Be transparent. Discussing your challenges will build trust with your partners and stakeholders. It will also help other organisations and add to your credibility.
There is a danger that our frustrated desire for often unattainable precision distracts us from attainable and accurate estimates of social impact. The problem of social impact measurement is unsolvable precisely, ”but very solvable imprecisely.” – Brian Telstad, Acumen Fund
