By Emma Tomkinson

On Tuesday 15th September SIMNA NSW hosted a workshop about the evaluation of Ability Links NSW, a program to link people with disabilities with mainstream community organisations and services. We heard from Craig Layton from the Department of Families and Community Services with Sara Hudson and Nicki Hutley from Urbis. The event was facilitated by Ric Bless, Metro North Regional Coordinator of Ability Links at UnitingCare.

Our speakers took us through the design of the program, how its evaluation framework was put together and the cost-benefit analysis that became part of it.

Both the delivery and evaluation of Ability Links were very much a partnership. As it was a new way to support people with a disability,the commissioning/tender approach commenced with an extensive pre-market engagement strategy which consisted of state-wide briefings and consultations. This provided the potential service market with an opportunity to begin to engage with the design and requirements of the Ability Links model from the beginning. The evaluation approach was similar. It was set out over three years, with Urbis on board from the beginning.

The initial stage was developing the evaluation framework, which was designed with organisations delivering on the ground. The evaluation framework has been developed and modified over the last two years, with initial reporting involving simply the number of people each provider provided support to and stories of their progress. Sara Hudson stressed that a key part of the evaluation was transparency, with the evaluation website made public http://www.abilitylinksevaluation.com.au/. This was consistent with the principle of learning as the program progressed, particularly as it was a new program without many rules. There was also a real effort to make the evaluation inclusive of all stakeholders, with all the providers getting the reports as they were developed.

The evaluation fieldwork involved 22 site visits, a couple of hundred people interviewed, focus groups with each stakeholder, online surveys with participants and Linkers (the staff of the program), and consultations with the wider community. There were 120 interviews with people with disabilities.

Craig Layton said, ‘The way the program runs…lends itself to an outcomes-focused way of measuring things’, and explained that the usual output performance measures were set aside to focus on outcomes . A set of social outcomes were developed out of the fieldwork. Some providers, particularly Aboriginal providers, can work with people over longer periods of time, so most information is not reported when providers are working with people, but only when outcomes occur. Outcomes are listed in ‘clusters’ e.g. education, then ‘outcomes’ and numbers of outcomes and the proportion of people achieving each of them.

At the end of 2014 the department decided they needed to measure the economic as well as social outcomes of the program, so a cost-benefit analysis was begun.

The first question was, “What is the program trying to do?” with the answers:
• to enable people with disability to do what they want to do with their lives, outside of the specialist disability system
• to help local communities to become more welcoming and inclusive to people with a disability.

The return on investment split into social, economic and environmental value. The Global Value Exchange assisted with the valuation of some outcomes through financial proxies. Nicki Hutley explained to us how understanding the different components is facilitating far more informed decisions about the program’s future.

Our group activity asked the question, “How do you shift cultures towards valuing the measurement of social outcomes, what experiences have you had and advice do you have for others?” Our answers are below:
• Develop capacity of staff
• Produce a safe environment to learn and ‘fail’ (continuous improvement mindset)
• Collect simple and useful data and build in feedback loops
• Use the power of stories
• Get buy in from senior management and frontline staff
• Connect passion with outcomes
• Engage the community

SIMNA NSW is extraordinarily grateful to UnitingCare for hosting us and to Craig, Sara, Nicki and Ric for being so open and generous with your learning.