Setting the scene
Youth Music is a national funder offering grants for music programmes for children and young people in challenging circumstances. Set up in 1999, they believe in the power of music to bring about social and personal development. Youth Music offers funding through three different programmes that vary in length and size of grant; they gave away £9.2 million in 2014-15. Youth Music is an impact champion for the Inspiring Impact programme.
Youth Music have been working on their impact practice since 2000. Carol Reid and Nick Wilsdon, programme director and learning and evaluation manager respectively at Youth Music, co-produced this impact case study with Sarah Menzies from NCVO Charities Evaluation Services.
Youth Music’s journey to evidence the impact of their grant making started in 2000.
2000: First evaluation forms introduced
Once the initial programme was up and running, Youth Music introduced the first version of their grantee evaluation forms, predominantly focusing on accountability.
From the beginning, Youth Music sought advice from a number of external agencies and the sector itself to support their evaluation, running consultation and engagement to inform the programme.
2008: Increasing focus on evaluation and impact
By 2008 Youth Music had grown exponentially. There were ten funding streams and an internal research and evaluation team was created for the first time. The new team reviewed existing processes and helped implement a series of steps to build a more detailed understanding of the impact of their funding, including:
Publishing their first externally-facing impact report, in addition to a series of reports exploring evidence related to the sector.
Establishing an outcomes framework for the first time.
Developing and implementing outcomes training sessions for grantees and Youth Music staff, building on resources and training from
Charities Evaluation Service
- Implementing an evaluation builder that offers framework guidance to Youth Music grantees, building on existing scales but tailored to Youth Music programmes.
Shortly after this, Youth Music streamlined their funding programmes to offer funding ‘modules’ which fully embedded an outcomes approach. There were five outcomes for each module with two core outcomes – improving quality and embedding practice – consistent across all of the programmes. Applicants were asked to work towards four outcomes set by Youth Music, and one which they set themselves.
It was important to Youth Music to use existing outcomes evidence to re-design their funding programme. The new outcomes were developed by reviewing emergent evidence from the programmes, building on the most consistent and common of the outcomes reported to date and in external research. This new outcomes approach then formed the basis of the application process.
2014: Grantees developing in confidence
During 2014, their outcomes approach was bearing fruit. Nick observed that where grantees had developed their own outcomes they had:
…taken ownership of the process and we noticed that those were submitting stronger evaluation reports. They also had a better understanding of the value of this in their own organisational processes.
From this point onwards grantees were no longer required to work towards outcomes set by Youth Music; instead they could define their own outcomes within a newly-established outcomes framework that was developed in response to their findings. In many cases, although the language used by grantholders in their own outcomes was different from that provided by Youth Music, the overall changes organisations were working towards were still well aligned. Moreover, greater ownership of the outcomes often engendered great organisational buy in and understanding in grantees.
2016: New business plan, commitment to improving impact practice
Youth Music published a new business plan in 2016 which included a commitment towards increased use of shared measures, in order to improve impact assessment and evidence impact across the diverse programme. The evaluation builder resources are being updated in light of this.
2016: Becoming an impact champion
Youth Music has become an impact champion as part of the Inspiring Impact programme. They explained that they committed to this because it:
…seemed like a natural fit for what we were doing already and the direction we were going with our grantees, but also strategically as an organisation. It’s about our own impact and supporting others.
Carol and Nick noted that as there are many different approaches to impact, it was useful to have the single Inspiring Impact site to refer grantees to, particularly the impact hub because of the resources it lists.
Improving their approach
Youth Music have made recent changes in three key areas of their impact practice; grantee reporting, developing a new database and sharing learning.
Improving the report template
Until recently, grantee reporting to Youth Music was more open ended, based on the question ‘what progress have you made towards achieving our outcomes.’ Youth Music found that this meant many organisations were reporting on activities rather than outcomes and if they did refer to outcomes, they were often not substantiated by evidence.
In the new reporting template, grantees are presented with their original outcomes and indicators to report against; this means that reporting is much more structured. They are asked to collate their evidence against each indicator and make a judgement on what the data means, and furthermore, what kind of activities are most effective to achieve their outcomes.
Including face-to-face reporting
To strengthen the quality of formal reporting, Youth Music holds regional grant holder gatherings. This includes the senior leadership team as well as other staff. This has allowed a personal face-to-face relationship to develop which encourages more frank conversations.
Offering telephone support
Youth Music also offers telephone support as part of routine monitoring, furthering their move to a more collaborative approach. Grantees are asked about how both their project and its evaluation are going and supported to overcome challenges.
Offering support with analysis
Youth Music also observed that organisations were not always analysing the data they had collected. They have encouraged groups to do this, offering training and support.
Youth Music has moved away from the kind of hierarchical relationship that can develop between grant-makers and grantees. They recognise and train their grants and learning officers to recognise that the grantees have the expertise. Nick stated: ‘We see ourselves as more of a critical friend.’
Having the grants and learning officers behind this initiative has supported the shift; their support was fundamental to creating a shared ethos running through the organisation.
Creating a reflective culture
Youth Music leave space for what they call the ‘ecstatic narrative,’ where grantholders can share the successes of their programme, but make a clear distinction between this and more evaluative thinking, which must refer to evidence of the impact.
Youth Music supports grantees to be more reflective by focusing on the details of the outcomes and indicators and by seeking to understand the fuller experience of how change has happened. For example, by distinguishing those beneficiaries who may have experienced significant change from those for whom only a little or no change has occurred.
Being open about what has not worked
Another challenge is encouraging grantees to talk openly about where their projects have not been as successful. Youth Music has made advancements by creating a ‘more open evaluative way of working,’ creating a culture where grantees have the confidence to report challenges.
Youth Music attributes this culture change to the approach of the grants and learning officers who emphasise the learning element of the work. The officers make it clear that there would be teething problems in any new programme. Youth Music is interested to know what the issues have been, how grantees identified them and what steps have been taken to resolve them.
In addition, Youth Music now gives feedback on the quality of evaluation as well as the funded project itself.
Youth music has recently designed and implemented a new custom-built database, a process taking about 14 months. The database allows powerful reporting functionality and they have pre-built reports. One of the main advantages of this new system is that it will allow them to get both segmented and aggregated data on grantees more quickly. As the data is more readily available so it is more readily used and this has allowed Youth Music to be a more evidence-based funder.
Youth Music produces an annual impact report that is available to the public, detailing how funding has been distributed, grantee profile and outcomes.
As well as regular informal sharing of learning, Youth Music has recently relaunched its charity website and intends to share more of their programme learning on the new site. This will include sharing raw data. This is a new area for development and fits with their values of being open and transparent about their processes and decision-making.
Reflecting on the journey
Presenting evaluation findings
Youth Music observed that some groups are still finding writing up of evaluation findings a challenge. Data analysis has been added to the grantee training, covering qualitative and quantitative approaches. Grantees are also signposted to Inspiring Impact resources including the Impact Hub and Measuring Up, a diagnostic self-assessment toolkit that allows organisations to review and improve their impact practice.
Managing data internally
To deal with the large volume of qualitative data that they collect, Youth Music has devised a coding framework using NVivo; they are now able to analyse all this data. For them, the need for this additional analysis was one of the few downsides of moving away from set outcomes and letting groups design their own.
Resistance from some grantees
One of Youth Music’s challenges is encouraging grantees to recognise that their approach is intended to benefit grantees. As Nick stated, they want grantees to: ‘see that we are not doing it to you; it is beneficial for you.’
There has been some resistance; grantees have asked them ‘when is all this outcomes is stuff going to go away?’ However, Youth Music has observed a change in attitude in the sector and has evidence that the majority of grantees value the process more, as evidence from their stakeholder survey suggests.
Youth Music is a busy organisation and ensuring there is sufficient time to support grantees with their impact needs, can be time consuming. Some internal processes needed to start out as informal, low capacity things but once they are working can then be expanded. For instance, Nick started staff learning sessions that required minimal preparation. The information was then shared at all staff meetings which started out every other month but now happen monthly and are facilitated by all teams in the organisation, with an exciting roster of external speakers. The learning has been written up for internal use and published for external use.
One of Youth Music’s strategic aims is ‘increasing our impact’. The first part of this is improving the quality of grantees’ evidence and building evidence-based decision making. However, they also see improving their own impact practice as an essential feature. As such, they intend to carry out Measuring Up for funders, Inspiring Impact’s recently launched self-diagnostic tool for impact practice.
They will build the results from this into their business plan, focusing attention on the areas they need to develop most around monitoring and evaluating. This self-reflection is an inherent part of their aim of being a strategic learning organisation.
Youth Music also works with their freelance grant assessors to ensure a good flow of evaluation information between the grantees, assessors and the grants team and are focusing on this issue in an upcoming training day for assessors.
To explore longer-term impact, Youth Music also wishes to introduce longitudinal follow up, through a questionnaire distributed by grantees to a sample of their beneficiaries post-intervention. Youth Music would support the grantees to develop systems to allow this, for example through collecting beneficiary contact details that are likely to still be valid in the future.
A sustainable funding model
Youth Music is also keen to develop relationships with other funders around shared funding. This could involve Youth Music supporting a grantee for a specific period and then another funder supporting for a further time period thereby allowing longer-term intervention that may result in deeper impact on communities.
Contributions to Youth Music’s successes
Youth Music shared ten tips for to other grant-giving organisations that wish to focus on impact.
Use a theory of change.
Youth Music found the theory of change process to be very useful in setting out their overall aims and their outcomes. This now sits behind all of their grant-making.
Get buy in from the whole organisation.
Carol and Nick felt that a number of factors had led to an organisational shift, including buy in from the senior leadership team. They were keen to emphasise that the shift towards becoming a learning organisation has been a ‘massive undertaking’ and therefore support from leadership was a prerequisite.
Youth Music felt that the outcomes training had reaped many benefits. By building the skills of grantees, Youth Music simultaneously got buy in from them. Grantees were receptive to the training: it allowed them to meet the team, develop their evaluation plans, meet other grantees and to feel part of a network.
Youth Music were keen to highlight patience as a virtue:
There’s a need to be patient as well in terms of how long that process has taken and to get to where we are now, and to see how long it has taken the sector to be more receptive to that.
They recommend that other organisations build in the necessary time to undertake evaluation as it is ‘fundamental to the work you do.’ They advise: ‘Don’t side-line it. It takes time to do it well.’
Youth Music advised that others thinking of embarking on a similar process should try to make sure they celebrate the successes along the way both internally and externally.
. Ensuring that you have the same messaging throughout all of your processes and engagement with grantees, staff, grantees and any other stakeholders is an essential feature of embedding this change.
Embedding the learning function in grants officers’ roles.
Building this approach into the job role of grants and learning officers has been a key feature to bring about an organisational shift. Carol and Nick felt that the entire organisation has been ‘upskilled … in terms of having an analytical perspective.’
Be open about how the data is used.
Carol and Nick noted that they take the time to let grantees know what happens to the data they present to Youth Music. They explain the process and how the data is compared to Youth Music’s framework and the other work they do, and how it informs future decision making.
Budget for it.
Youth Music advises grantees to be aware that evaluation does have a cost and that they should build it into their funding applications. Larger grants may also need external evaluation. Youth Music fully supports this.
Take small steps to start the process and build from there.
Contact details and further reading
Sarah Menzies, NCVO Charities Evaluation Services
Nick wrote a blog for the ACF website on Changing Impact Practice: From accountability to strategic learning