Anglicare WA was the winner of the 2014 Social Impact Measurement Award for large not-for-profits. Article by Desiree Nangle, Evaluation Project Manager.
Background
Anglicare WA has been on a two and a half year journey of building a culture of evaluation and continuous improvement across the organisation. Driven by our Strategic Plan and our single Beneficiary Performance Indicator (BPI) of moving clients from surviving to thriving, this choice is centred on a strong desire to know and have evidence of the impact our services have.
Before beginning this journey, research into best practice was sought and reviewed. The benefits of creating an organisational culture of evaluation are well documented so Anglicare WA applied for, and won, a Social Innovations Grant from the Department of Communities. This award allowed two years of employment for a fulltime evaluation manager to oversee the implementation of Results Based Accountability (RBA) internally, across approximately 68 services, as well as externally to share learnings with the sector.
Two and a half years later, the journey toward an organisational culture of measurement continues. The process started with introducing individual services to RBA and having them begin implementation. Starting with those services who volunteered for the training, one-by-one, services complete workshops with the evaluation manager to define their outcomes and begin setting up systems to allow for data collection and analysis.
Initial Obstacles
Early roadblocks appeared immediately, mainly stemming from the lack of outcomes measurement within the organisation. Most of our services were found to only be reporting on outputs, as required by funders. Another huge barrier to seamless implementation was not having one overarching database with capacity to capture and analyse the outcome data that would be needed. Having identified these issues and still needing to progress the initiative, solutions for data storage had to be tackled immediately.
Along with data management issues, resource and time management have also been very prevalent throughout this journey. Staff approval and buy-in was identified from the outset as imperative to success. Anglicare WA, aligning with research in change management, took a bottom-up approach, supported by top-down endorsement, with implementation. Starting with the service staff has had huge benefits for both acceptance of measurement and use of results. Allowing staff to define their own outcomes for their services increased their sense of ownership in the process and their accountability for client outcome results.
Early successes
Early successes were evidenced when services began reflecting on their practices and making changes as a result of analysing RBA data. To make data presentation meaningful and less overwhelming, services focus on 3-5 outcomes at a time. This ensures the most important indicators of success are constantly being monitored, staff aren’t drowning in data, and all results can be displayed in a user-friendly, one-page report. It was also found that presenting services with their own data in this one-page, ‘headline measure’ report resulted in abundant gains in staff understanding and buy-in!
While the main objective of this initiative is to build a culture of evaluation allowing for continuous service improvement for our clients, it would be added benefit if service staff support this initiative and find it useful. To practice what we preach and continuously monitor, and therefore improve, our chosen approach to measurement, a meta-evaluation (RBA on RBA) has been collected alongside services’ RBA frameworks.
Results from this measurement, mainly obtained through an all-staff survey that has, so far, been completed four times from December of 2011 through March of 2013, show encouraging results! Staff reporting they understand what RBA is has increased from 53.6% (2011) to 80.9% (2013). Additionally, 66.1% of staff who completed the most recent survey in 2013 (31.4% or 209 staff members) agreed or strongly agreed that implementing RBA within Anglicare WA is a good thing. This latter result was very inspiring giving the time constraints staff are under in that although data measurement takes time, sometimes away from what is identified as their main objectives in client work, they find the results of that work to be useful enough to support the resources required to obtain them.
Moving forward
Moving forward on this road toward impact measurement, Anglicare WA will continue to monitor and work on improving understanding and support of RBA as a tool to measure client outcomes. Recent steps taken to make data and measurement more ‘fun’ include highlighting a service’s “curve of the month” on the intranet and in our staff newsletter as well as posting relevant cartoons on bulletin boards to make data more relatable, fun and familiar.
Anglicare WA’s road to building a culture of evaluation has been long and fraught with learnings but has already led to improvements for our clients! We have also had the opportunity to walk alongside other organisations and mentor them while they undertake implementing RBA within their services. As the community service sector moves in the direction of understanding and rigorously measuring impact, we have learned a lot about what it means to better serve our clients in meaningful ways. RBA is just the beginning of our journey and was chosen for its clear, simple, introductory concepts into impact measurement, but is only meant to be the first step on our evolving quest into evaluation. Alongside RBA, some programs are already using Social Return on Investment (SROI), Most Significant Change (MSC), and Collective Impact (CI) to enhance the RBA work being done.
As of today, 87% of Anglicare WA’s services have begun their RBA process by identifying their outcomes and approximately 40% have been collecting outcomes data for over a year. Our evaluation project manager has also been able to work with over 20 external organisations, independent consultants and government agencies to help them learn and implement RBA within their own areas of service. Additionally, Mark Friedman, the founder of RBA, sponsored by Anglicare WA, has also been to Perth two times (in 2012 and 2013) to meet with Anglicare WA as well as other interested organisations to critically review and guide them in their RBA work.
For further information and examples of work being done by Anglicare WA in this area, please visit our website RBA page at:
http://www.anglicarewa.org.au/resources/results-based-accountability.aspx
Our RBA blog is also available for viewing at: http://www.anglicarewarba.org.au/