One of the great challenges of development work is that you can never really tell whether or not it’s making a difference in the lives of its beneficiaries. Except… you can. Here’s how to find the data you need to quantify the return on social investment – and why you need to do it now.
The director of a foundation in Cape Town places promising students in good (but, for the learners, unaffordable) schools. A volunteer at an NGO in Port Elizabeth facilitates a training workshop for unemployed job-seekers. A socially aware investor in Johannesburg tries to uplift the youth (and make money for investing clients) by running educational impact investment projects.
Yes, they’re all working to make a contribution to society. But how much of a contribution? It’s a question that haunts every well-intended social initiative and innovation there is.
Increasingly, companies and individuals who put money into these kinds of projects want to know their investment is paying off – socially, if not financially. How can this be done? And can this kind of success be measured and quantified? The short answer is, yes. The long answer is going to change the way you look at social investments in general.
Let’s look at a case study…
SAILI is a Cape Town-based NGO that links academically talented students with low-cost, high-quality schools. According to their mission statement, they “identify, develop and support potential in students and schools by guiding talent into the working parts of the state education system.”
The organisation ensures that students with potential have a fair chance at fulfilling that potential and going on to tertiary studies. “We add quality and competitive edge to well-functioning schools by referring our gifted scholars to them,” they add. But, do they really?
SAILI director Sam Christie has asked himself that very question a number of times – and, by using complex calculations and data analysis, he believes he’s found an answer.
One method of measuring the success of SAILI’s placement programme is to ask: How would the students have fared if they’d been placed in non-programme schools not considered strong enough?
“That question is easy,” says Christie. “Apart from astonishingly capable outliers, the chance of a student attaining the same results outside our programme and in a weaker school is so remote as to be insignificant. Simply, most schools do not come anywhere near the performance we get.” Here Christie points to the Cartesian axes (see graph, below), which contrast the test performance by various Western Cape schools (y-axis) with the school fees (x-axis). The curve – and the correlation – is unmistakable.
“Remember, our Maths average was 76%, which is comparable to the top 10 schools in the Western Cape,” Christie continues. “Very few schools will produce a distinction in Maths. While kids might pass Maths elsewhere, the chances of genuine high performance are more or less non-existent. Of course, this does not mean that these kids would not amount to anything in other schools; simply that they would not progress where high performance is a requirement for tertiary study – for instance, in Medicine, Science, Commerce and Engineering.”
So that takes care of the school quality question.
But next, you might have to ask: Would SAILI students not have had access to the schools they’ve been placed in on their own? “This question is harder to answer now than in the past,” Christie concedes. “Because we seek out low-cost schools (lower in cost than in the past and lower than other schemes use), we do run the risk of funding students who might have funded themselves. Take Mondale school in Mitchells Plain, Cape Town: the fees are about R3 000 and neighbouring schools very much less. With these low fees, clearly the school is relatively affordable.” That said, Christie offers the following points as a response:
- Information from SAILI helps link the students and the school. “We inform parents of the value of the school we direct talent into. Many of these students may have supposed they would get better value out of the community than in it” – in other words they undervalue their local schools and aspire to attend other institutions that may not in fact be as good.
- Demand for places in better schools is high. “Schools struggle with non-payment, and primary school assessments are not very accurate. Consequently, many schools prioritise students who seem wealthy over those they suspect might be poor and might not pay fees (they are not compensated for non-payment). Many of our applicants are placed on waiting lists prior to us awarding scholarships. Once awarded, schools in most cases make places available.”
- The threshold for funding is reduced in real terms each year.“We have a lower cap than any other scheme, and emphasis is placed on funding those with the greatest need first.”
Not just about results: How to measure educational returns…
SAILI’s assessment of Cape Town’s high schools is based in part on the relationship between school fees (investment) and test results (return on investment). But when it comes to assessing tertiary education, the formula can become a little more complicated than that.
Payscale.com rates American universities based on a 30-year return on investment (ROI) for Bachelor’s degrees. How? Basically, by looking at how much a graduate from a particular university will end up earning.
Business publication Forbes has developed a far more complex – and, arguably, far better – methodology. Working with the Center for College Affordability and Productivity (CCAP), Forbes/CCAP used five general assessment categories, with several components each (see below).
The result doesn’t necessarily constitute a measurement of return on social investment, but the details of the formula are instructive nonetheless – for instance, the Payscale.com yardstick made up just 15% of Forbes’ overall rating.
Here’s how Forbes and the CCAP weighted their findings:
1. Student satisfaction (27.5%)
Student evaluations from RateMyProfessor.com (17.5%)
Actual freshman-to-sophomore retention rates (5%)
Predicted versus actual freshman-to-sophomore retention rates (5%)
2. Post-graduate success (32.5%)
Listings of alumni in Who’s Who in America (10%)
Salary of alumni from Payscale.com (15%)
American leaders list (7.5%)
3. Student debt (17.5%)
Average federal student loan debt load (10%)
Student loan default rates (5%)
Predicted versus actual percent of students taking federal loans (2.5%)
4. Four-year graduation rate (11.25%)
Actual four-year graduation rate (8.75%)
Predicted versus actual four-year graduation rate (2.5%)
5. Academic success (11.25%)
Student Nationally Competitive Awards (7.5%)
Alumni receiving PhDs (3.75%)
