On 11 March 2015, SIMNA NSW held a workshop ‘Social Benefit Bonds and Practice Innovation’. Facilitated by Doug Taylor, now Director Strategic Engagement for UnitingCare NSW.ACT, Doug opened the session with what sounded like a joke: “What happens when an economist, a social worker and an accountant get together?”
Sally’s measurement story
Answering his question was Sally Cowling, an economist, who, as Director of Research, Innovation and Advocacy for UnitingCare Children, Young People and Families, was responsible for setting up the outcomes measures and analysis in the Newpin Social Benefit Bond.
Everyone who talks about Social Benefit Bonds (or social impact bonds as they’re known internationally), talks about how crucial social impact measurement is. However, this was the first time Sally has been asked to tell the measurement story. “Since the Newpin SBB has been set up, I’ve done a lot of speeches about the bonds, but this is the first time anyone’s asked me to talk about the technical measurement aspects.”
Sally gave an incredibly frank and meaningful account of the learning journey UnitingCare is on. “I want to be brutally honest about what happened and the challenges. About what we’ve learnt as an organisation; what we’ve learnt about accountability; what we’ve learnt about ourselves. And about how we funded a program for a decade without the kind of measurement to support better targeting to those most in need.”
Where the journey began
The journey started with the aim of the program, “we knew we wanted to break intergenerational cycles of abuse and neglect. But we had to think – how do we know when we’ve done that? How do you know when that cycle is broken? So my job was to go through (with ethics approval) every case file from the past three years and work out what outcomes occurred for our members (we don’t call them clients).”
“If you asked Newpin staff, ‘Who is the program for?’ they’d tell you, ‘We turn no person away’. But as an economist I was thinking ‘well this is a serious investment, we need to understand who is receiving these services.’”
Sally took the audience through:
- how Newpin realised that families receiving services fell into distinct groups
- how, when writing their response to the NSW Government’s request for a written proposal, “in the measurement context, we were crystal clear about what we knew and what we were guessing”. For example, we needed access to unpublished data in the next phase to determine how long a young child was likely to spend in foster care in the absence of a Newpin intervention
- how, once they started working with government in the joint development phase, they realised that administrative child protection data couldn’t allow a precise estimate of the ‘counterfactual’ (another ‘guess’ in the proposal). A counterfactual is an estimate of what would have happened to the families involved in the program if they weren’t involved in the program.
- how, in collaboration with NSW Family and Community Services (FACS) they used duration in care before an intervention as the best (and research-based) proxy of the probability of family restoration outcomes. This is being used as a fixed counterfactual for the first 3 years of the bond while they build a ‘live control group’ – an important but resource intensive task.
Sally explained how the focus on measurement for the Newpin Social Benefit Bond has led them to ask questions of other UnitingCare programs. Most of these are funded under standard government grants and “require us to be accountable for ‘bums on seats’ rather than outcomes. If we didn’t know enough about our Newpin target group, and we didn’t specify or measure outcomes for members…what don’t we know about our other programs? What do we know about how well we are and are not doing?”
Wider implications
Across the organisation, as part of the strategic plan, UnitingCare are now implementing a performance improvement project to build one-page reports on programs or program clusters. Generated quarterly, these reports contain:
- some demographic information
- some efficiency measures like caseloads, and
- outcomes measures.
UnitingCare has invested significant money in a client data information system. It can generate a vast number of reports, but the challenge is to build capacity to use the data to improve service quality. Sally said “It doesn’t matter whether the data is on paper or electronic files. If it’s not being looked at and responded to, what is the purpose of collecting it?”
Sally finished off by reminding us that “it’s so important to celebrate measurement, and to focus on what we’re measuring, why we’re measuring it and how that relates to our vision and mission. It’s also important to remember what can’t be measured.
When I walk into a Newpin centre, there’s something about that place and space that ‘feels’ different. It is something both special and intangible. So when you’re writing up results it’s important to reflect on that too. It may help you interpret the ‘quant’ results.”
“Our first Aboriginal Mum who graduated from Newpin herself addressed a parliamentary group and asked them ‘“If you have not been taught how to love and be loved, how can you teach or show it when you have a baby?’”
“The Newpin program recognises and acknowledges the traumatic experiences our members have been through and acknowledges their capacity to make phenomenal change.”
Sally’s presentation was unquestionably remarkable. Her measurement story was one of honesty and meaning. As she finished, she got the kind of applause usually reserved for performances where an encore is an option. We might have to think about what an encore looks like for our SIMNA presenters! Our thanks to Sally and to Doug for their time and support.