“Several things stood out for us in this winning entry. This was a really strong application, particularly around three things: 

1) overcoming challenges and adopting innovative solutions
2) application of the seven principles of social value and
3) using the results for improvement. 

A number of important learnings were also highlighted. Transparency is vitally important in explaining what you have done and the decisions and judgements you have made along the way. As is involving service users from the earliest possible opportunity – a really smart move.” SIMNA Award Judges 2018

Thank you Dr. Johana Susanto
, Research and Outcomes Measurement Manager, for sharing your team’s social impact measurement story with us.

About Us

The Salvation Army Australia is a Christian movement dedicated to sharing the love of Jesus by caring for people, creating faith pathways, building healthy communities and working for justice. Since 2016, the Salvation Army has been building its internal capacity to conduct outcomes measurement (also known as social impact measurement) to support the culture of accountability and learning.

Peer Support

In 2016, the Centre for Social Impact (CSI) partnered with The Salvation Army’s Research Team to develop our first overarching Outcomes Measurement Framework for the organisation. We (The Salvation Army Research team) piloted this framework in 2017, in partnership with frontline services in ACT, NSW and QLD. The aim was to design and implement an outcomes measurement plan and tools for each social service and map the outcomes back to the organisation’s overarching framework, to demonstrate and articulate the collective impact of the organisation. For each individual service, the outcomes would also be mapped to the funding bodies’ outcomes framework.

By March 2018, 21 outcomes measurement pilot projects had been completed across a total of 16 social services. The support from CSI kept us focused and committed to trying to get this right.

Our winning entry – the Moneycare Program!

One of the 16 services included in the pilot study was the Moneycare program. Started in Rockdale (Sydney) in 1990, this is one of the Salvation Army’s national flagship programs. Moneycare supports individuals and families with financial difficulties to address their immediate crisis situation and build their long term financial resilience. It does this through:

1. A free and confidential financial counselling service, and
2. A financial capability work service and other related activities.

This particular pilot project commenced in June 2017. It ran across 39 Moneycare centres in ACT, NSW and QLD. Some 67 Moneycare workers took part.

Starting Out

To ensure that outcomes measurement added value for the Moneycare frontline team and was appropriate for the community members accessing the service, a series of consultation exercises took place. This included but was not limited to: interviews with workers and managers to inform outcomes measurement objectives, design, data collection and reporting to frontline team; a managers’ survey about their team’s perception and readiness to conduct outcomes measurement; and Theory of Change workshops.

Based on their feedback, we established that the specific objectives of the Moneycare outcomes measurement pilot would be:

• To create a systematic evidence-based feedback loop to inform practice and to promote continuous learning and improvement.
• To periodically capture and communicate the differences that the Moneycare service has made in people’s life to Salvation Army staff, leadership, community partners and supporters.
• To inform Government policies in the ‘Financial Wellbeing and Capability’ area and the new ‘Commissioning for Outcomes’ model.
• To share knowledge, strategy and best practice in outcomes measurement with the sector.

The Framework

We chose to adopt the Centre for Social Impact’s ‘Three Ps’ approach for measuring outcomes: understanding Purpose and Process, in order to measure Progress (outcomes) (Muir & Bennett, 2014). Additionally, we incorporated the guiding principles of:

1. co-design (to ensure that the measurement process was practical and useful for the day-to-day service activities),
2. integration in service delivery (to minimise measurement fatigue, to reduce duplication of data collection and to provide opportunity for creation of an evidence-based feedback loop) and
3. holistic impact (to capture the more holistic impact of the program on a person’s life, beyond the changes in their financial situation).

Indicators

Once the outcomes were prioritised and appropriate time points for measuring those outcomes were identified, the Research Team then selected and developed appropriate indicators. We did this using the Centre for Social Impact’s guidelines for indicators assessment and selection (Bennett, Reeve, Muir, Marjolin, & Powell, 2016). Ultimately, a combination of both validated indicators and in-house instruments were used:

• To measure Financial Resilience, the Centre for Social Impact’s Financial Resilience Indicator (Muir et al., 2015) was adapted in-house by the Research team in consultation with the Moneycare team and CSI.
• To measure personal wellbeing outcomes, the research team shortlisted the Personal Wellbeing Index (PWI). It was shortlisted because it is a validated and widely accepted instrument that covers broader aspects of community members’ life that could be impacted by, or are the underlying issues of, financial hardships. It also has been validated across different cultural groups and languages – some of these groups have been accessing Moneycare services.
• To measure Mental Health outcomes the Research team shortlisted Kessler-6. It was shortlisted because like PWI scale, it is a validated and widely accepted instrument and is a short scale that consists of 6-item inventory. This instrument has also been validated and translated to other languages.

Measurement

For the financial counselling service, a baseline outcomes assessment was done at the start of the program using the Financial Resilience indicator, PWI and Kessler-6. An intermediate assessment was done after one month (Kessler-6 assessment only). The endline assessment was done after three months or at exit, whichever occurred earlier. This included the Financial Resilience indicator, Kessler-6 assessment and feedback on Salvation Army and Moneycare workers.

For the financial capability service, the same baseline assessment was completed as above. There was no intermediate assessment as typical engagement tends to be short. Instead, the endline assessment was performed after one month or at exit, whichever occurred earlier. As above, the outcomes were measured using the Financial Resilience indicator, Kessler-6 assessment and feedback on Salvation Army and Moneycare workers.

For both services, the Research Team also tested a follow up assessment 3 months after exit, to better understand the sustainability of the outcomes that have been achieved and if improved financial resilience led to other holistic outcomes (such as education, employment, health, living standard, etc.); and to get community members’ perspective if there were skills they could have learnt from Moneycare to better address their situation after exiting the program. This information would be used to inform service delivery, referrals and collaboration strategy to strengthen community member’s capacity to sustain positive outcomes and to manage future hardships.

Findings

At the end of outcomes data collection period, 572 community members completed the outcomes baseline assessments. From this sample, 213 community members completed both baseline and endline assessment for Financial Resilience indicator, while75 people completed both Kessler-6 baseline and endline assessment. These matching samples were used to analyse progress of community members accessing Moneycare during pilot period. The findings summary is outlined below:

79% of community members reported an improved financial resilience at exit or within 3 months of receiving supports. Areas where more positive outcomes were reported were:

• Debt Management
• Ability to meet living expenses
• Budget
• Proactive behaviour – Seeking professional financial advice in the future
• Know where to find accurate information on personal finances
• Understanding of financial products and services
• Conversations with family about good financial habits or behaviour

67% of community members reported an improvement in their mental health within three months of receiving support.

The key areas within wellbeing and spirituality domain where community members had shown improvements were: satisfaction with their life as a whole, satisfaction with what they are achieving in life, and satisfaction with their standard of living and personal relationships. Further, 84% of respondents agreed or strongly agreed that they were satisfied with their religion and spirituality.

Overall, these findings demonstrated that both services have helped community members in addressing their immediate needs such as their ability to meet daily living expenses or to manage a pressing financial issue. These services have also helped clients in achieving more long-term outcomes such as a better financial knowledge and behaviour, and the overall financial resilience.

Challenges and insights

As you would expect, a number of challenges came our way, some anticipated, some not. Some examples include:

• Initial sense of apprehension in the Frontline Team concerning the objective of measurement initiatives
• Lack of understanding of outcomes monitoring among the frontline team and limited ability to conceptualise how outcomes assessment could be used to inform case work
• Participation rates
• Dropout in the endline assessment
• Low literacy levels of some community members
• Flexibility in completing the assessments
• Length of assessments and time taken for assessment
• Reporting outcomes to diverse funding bodies

A number of important lessons were learnt as well such as:

• Co-design is an important principle in the success of outcomes measurement implementation
• Outcomes measurement should be integrated into service delivery
• Outcomes measurement framework should capture holistic outcomes and not just the service or funding outcomes as this enables stakeholders to understand the total social value created by the service
• Targeted communication and engagement strategy should be developed at the beginning of measurement project to support an effective outcome measurement implementation
• Building capacity of the frontline team to interpret and draw inferences from outcomes measurement findings is an important strategy to maximise benefits of outcomes measurement for the frontline team and community members.

Overall, the Moneycare Outcomes Measurement Pilot has contributed significantly in enhancing the Salvation Army’s knowledge of outcomes measurement. The icing on the cake has been winning the 2018 Changemaker Award courtesy of SIMNA!

Dr. Johana Susanto
, Research and Outcomes Measurement Manager
Policy, Research and Social Justice Department
e: johana.susanto@aue.salvationarmy.org)
February 2019