Inspiring Impact is a programme working with the charity sector to help organisations focus on their impact. It was established in 2012 with a vision to make good impact practice the norm for charities and social enterprises by 2022. Inspiring Impact is a collaborative programme, developed by a consortium of organisations in the charity sector. Inspiring Impact’s partner organisations (ACF, Building Change Trust, CES (now part of NCVO), Evaluation Support Scotland, NCVO, NPC and Substance) include both funders and charities, and each of them apply their own specialist expertise to the programme. Together we aim to bridge the gap between impact theory and practice by delivering tangible free tools for the sector. Inspiring Impact’s programme is centred around five strands of work, each led by different partners, and each with specific resources:
- Impact leadership: helping charities and social enterprises know what good impact practice looks like, and encouraging good impact leadership so organisations can improve.
Resources include the Code of Good Impact Practice and Are you leading for impact?
- Coordinating support: helping charities and social enterprises understand the support available to them, and how best to use it – primarily through Measuring Up!
- Data, tools and systems: developing the infrastructure needed to ensure the sector can review, access and use the best quality data, tools and systems – primarily through the Impact Hub.
- Shared measurement: supporting charities and social enterprises working on similar issues, and towards similar goals, to reach a common understanding of what to measure, and developing the tools to do so. Resources include the Blueprint for shared measurement, The Journey to EmploymenT (JET) framework and the Future of shared measurement guide.
- Funders, commissioners and investors: supporting funders to improve their practices around impact measurement, embed a focus on impact in funding decisions, and build monitoring and evaluation costs into their funding. Resources include the Funders’ principles and drivers of good impact practice.
Inspiring Impact has, over its first three years, achieved significant interest in the sector – with over 187,000 visits to the website and 52,000 interactions with Inspiring Impact resources. This evaluation, at the end of the third year of Inspiring Impact, aims to understand usage, satisfaction and outcomes in relation to the Inspiring Impact website and some of the key resources. These are the Impact Hub and Measuring Up! (both launched July 2014) two of our publications, the Code of Good Impact Practice (published June 2013) and Funders’ principles and drivers of good impact practice (published June 2013). We chose to focus the evaluation on these resources as they are relevant to the widest potential audience and form the core of Inspiring Impact’s offer to date.
We collected data via:
- An online questionnaire. This was sent directly to 2,997 contacts – subscribers to the programme’s newsletter or users of Measuring Up! It was also promoted via social media and in email newsletters
- Semi structured qualitative telephone interviews with ten funders and charities, each lasting 30-60 minutes.
- Google Analytics on the Inspiring Impact website.
Survey data was collected and analysed by CES (part of NCVO) and interviews were carried out and analysed by NPC.
About the survey respondents
The survey was open from 16 March to 3 April 2015 and had a 7% response rate (n=206). Charity, voluntary organisation or social enterprise made up 89% (184) of respondents. Other respondents were from statutory organisations (4%), commercial organisations (3%) or other types of organisation (3%). An example of the survey can be found in the appendix.
About the interviewees
Interviewees were selected from the list of registered Measuring Up! users or because they were known to have used Inspiring Impact resources. We focused on users of Inspiring Impact to ensure we were able to collect data that could help us understand how the resources are currently being used. We approached interviewees via email inviting them to participate, and interviewed them by telephone. Many of the interviewees were known to Inspiring Impact, and whilst interviewees varied in their familiarity with and use of Inspiring Impact, we acknowledge that we may not have captured the full range of views in this limited and selected sample. We are grateful to the following funders and charities for their valuable time and contributions:
- Paul Smith, Age UK Barrow and District
- Alison Penny, Childhood Bereavement Network, part of National Children’s Bureau
- Gina Crane, Esmee Fairbairn Foundation
- Nick Acland, Henry Smith Charity
- Elizabeth Young, Home-Start
- Jenny Wood, Impact Arts
- Harriet Stranks and Michele Lester, Lloyds Bank Foundation
- Carole Coulon, Mayor’s Fund for London
- Kirsty Gillan-Thomas, Paul Hamlyn Foundation
- Judith McComb, Sported
We focused our evaluation on four of the main resource areas, and report on each in turn. Please note, due to rounding, some percentages in graphs may not add up to 100%.
Inspiring Impact’s website (www.inspiringimpact.org) introduces the programme and offers news items, blogs and various tools and resources, including the ones featured in later sections of this report.
Google Analytics show that the Inspiring Impact site had over 40,000 UK users between June 2013 and 10 June 2015 – 79% were from England, 9% from Northern Ireland, 9% from Scotland and 3% from Wales. The site had over 187,000 site visits and these include visitors from across Europe, USA, Russia, Australia, China, Brazil, Canada and India. These figures indicate the programme has good reach and is engaging with a significant proportion of sector. Most survey respondents had visited Inspiring Impact’s website either once or twice (45%) or monthly (25%) during the six months prior to completing the survey. A small number had visited more frequently; 9% had visited weekly and one respondent visited daily. 13% reported not visiting the site at all in the previous six months.
Satisfaction with the website
Survey respondents were generally very satisfied with the quality of the website (see figure 1). 97% agreed or strongly agreed it was useful and 96% said it was well presented. 91% said it was easy to find things, with the same proportion finding it engaging. The strongest negative feedback was on it being easy to find things on the site, with 10% disagreeing or strongly disagreeing with this statement.
User satisfaction with the Inspiring Impact website
63% of survey respondents would recommend the website to others (see figure 2). Some of our interviewees who initially were not aware of all the resources available on the Inspiring Impact website said that now they understood it more, they would definitely recommend it to others (for example funders said they would recommend it to their grantees).
Would you recommend the Inspiring Impact website to others?
Additionally, 56% said they have already signposted others to Inspiring Impact already, with 38% saying they may do in the future.
I ask every new staff member to reread the website and key reports – and have set it as a learning objective for existing staff.
Suggestions for improvement
Survey respondents were asked to make suggestions on how to improve the Inspiring Impact website. These were themed around incorporating more examples and case studies of organisations with good impact practice, and improving the site so it was more visual and interactive, for example with less text and more video, and optimised for access on mobile devices.
The Impact Hub
The Impact Hub is hosted on Inspiring Impact’s website and is a one-stop shop for resources and tools to help organisations improve their impact practice. It pulls together a wide range of resources relevant to improving impact practice, and enables users to search and filter results according to their needs. As well as specific impact and outcomes measures, there is information on diagnostic tools, research reports, public data sets, support and training. The Hub also allows users to upload resources to the site themselves and also rate them, enabling the sector to share resources widely and feedback on how useful they have been.
Usage and satisfaction
Between June 2013 and May 2015, there were 5,928 unique visitors to the Impact Hub. Almost half (46%) of survey respondents had used the Impact Hub. 57% of the survey respondents who had used the Hub were browsing for resources, and 39% used it to look for a specific resource.We asked survey respondents who had used the Hub what they thought about the overall guidance pages. 39% thought they were very helpful and 55% said they were okay. 5% did not consider them helpful. The Hub presents an option for users to open an account which allows access to additional features such as adding, commenting on and rating resources. 48% of survey respondents who had used the Hub had signed up to access these features. The most common reason for not signing up was that respondents were unaware that they had the option to do so.
Survey respondents and interviewees mentioned a variety of tools and resources they had found via the Hub, including evaluation guidance documents, tools, and public data sets. Specific tools or resources mentioned were the Code of good impact practice, Office for National Statistics, data.gov.uk, Journey to EmploymenT framework, Theory of change: the beginning of making a difference, Blueprint for shared measurement, Outcome Stars, and resources on logic modelling and data protection. Users enjoyed the variety of resources available on the Hub and some had accessed it to specifically help implement recommendations from Measuring Up! For those interviewees who had not visited the Hub before, their initial feedback once it was described to them was that this was a very useful resource that helped to fulfil a need in the sector. Some were surprised they were not aware of it as they felt it was relevant to them. Some funders felt they would refer their grantees to it now that they were aware of it, and some did not, as they preferred not to overload grantees with resources. Other funders were already referring their grantees to the Inspiring Impact website, including the Hub.
Suggestions for improvements
We invited survey and interview respondents to suggest ways in which the Hub could be improved. Responses centred around the following themes:
Some suggested that more Northern Ireland specific tools would be useful. Other suggestions included more tools and guidance on qualitative data collection and increasing the overall number of tools.
Some people suggested that the filtering system could be improved and expanded so that it was easier to find tools linked to their particular area of work. In addition, some users found the language difficult to understand, and felt there was too much jargon. Whilst these are defined elsewhere on the Hub, a suggestion was made to have floating text boxes on the search page to help define the category options.
- User reviews and experience
The experiences of others who have used the tools is particularly important to users of the Hub, and respondents felt that feedback and reviews on the listed tools would add value to the Hub and help them to choose appropriate tools. Whilst this option is available, it requires the user to have an account and sign in, perhaps reducing knowledge and use of this facility.
Measuring Up! is a step-by-step self-assessment tool that allows organisations to review and identify ways to improve their impact practice – that is, the way they plan, evidence, communicate and learn from the difference that their work makes.
Usage and satisfaction
Between June 2013 and May 2015, there have been 8,520 unique visitors to the Measuring Up! page, and 1,691 users had registered to use the tool. 45% of survey respondents had used Measuring Up!; of those, three quarters had completed the self-assessment; the remainder had started but not completed it (see figure 3). The main reason for not completing the assessment was lack of time, or because they were considering whether it was appropriate or had decided it was not appropriate.
Have you done the Measuring Up! assessment?
We would like all visitors to use Measuring Up! and the qualitative interviews provide some insights into why the conversion rate from visiting the page to using the tool is not higher. Interviewees who had not used Measuring Up! said it was because they were not aware of it or found it difficult to access, which suggests a lack of awareness amongst our audience. Additionally, Measuring Up! requires users to register and sign in to access the tool, which was found to be a barrier to use. Interviewees felt that more information would be helpful without needing to register and sign in, to help them to understand the tool and how it would work in practice. We consider the specific ideas on how to do this in the suggestions for improvement section below.
Outcomes and actions
Survey respondents who had used Measuring Up! were asked about the extent to which the tool had helped them on a variety of outcomes. Figure 4 reports these results, and we can see that the majority of respondents felt that Measuring Up! had helped them to increase their skills, identify resources, clarify areas of strength and weakness and measure impact better.
Figure 4: To what extent has completing Measuring Up! and getting an action plan helped your organisation to:
Our interviewees provided some interesting examples of Measuring Up! helping to clarify strengths and weaknesses:
- Sported, a membership organisation supporting community sports clubs, found that by using Measuring Up! they were able to identify areas of impact practice where they had most difficulty, and enabled them to focus in on what they needed to change (read more in the box on page 9).
- Impact Arts, a charity using arts to enable social change, said one of the most useful things that came out of the process was seeing how different members of the team perceived the organisation’s current measurement practices. This helped them to identify points where there was a lack of consensus and to address these.
On completion of the Measuring Up! assessment, users obtain an action plan that they can download. The majority of our survey respondents who had completed Measuring Up! found the action plan ‘very helpful’ or ‘helpful’ (79%). Almost two-thirds of survey respondents had carried out some of the actions suggested in their action plan; no one had carried out all of the actions suggested. Just over a third had not yet carried out any actions, but planned to in the future. We asked survey respondents and interviewees about the type of actions they had taken or were planning to take, and the following themes emerged:
- Creating or refining an evaluation plan, such as an evaluation framework or theory of change. Impact Arts for example have developed an evaluation plan, which they have acted on and found very useful.
- Changing or planning to change approach to data collection – for example, identifying new data collection tools, improving baseline data collection processes and increasing the amount of consultation they undertook with beneficiaries.
- Prioritising impact practice organisationally, for example by incorporating it into organisational planning.
The Measuring Up! tool gives a really useful process of what an organisation should be doing in measuring the difference they make, whether just starting the process or ongoing review for each project.‘[We] have conducted an organisation-wide Measuring Up! exercise to determine [an] overall organisational view on how far we are in terms of data gathering and measuring success.[We are] thinking about how to analyse the data we are collecting.
Measuring Up! in practice: an example
Sported is a membership organisation for community sports groups. They have used Measuring Up! for themselves and with their members. They find that with their members, it takes about 1.5 hours to do the first stage of Measuring Up!, and about 40 minutes for each subsequent section. They find the tool easy to use, although some smaller groups can find the detail and language tricky, and so they suggest Inspiring Impact consider a ‘light’ version for those organisations.
Sported found their own Measuring Up! assessment really useful, and it helped them to identify areas of strength and where they needed to focus. They are keen to review their progress by repeating the process later this year.
Suggestions for improvements
We asked all respondents for suggestions on how to improve Measuring Up! and the following themes emerged from the survey and our discussions:
- Improve awareness of the tool and its uses.
It is clear from those who had not heard of Measuring Up! that we need to consider how to increase awareness and understanding of the tool. Once the tool was described, interviewees were very receptive to the idea and some expressed surprise that they had not heard of it as it would be very relevant for them. Some of the funders who were interviewed said they would recommend it to their grantees now they were aware of it.
- Provide specific information about how the tool works in practice, without needing to register and sign in.
Those who were aware of the tool but had not used it felt that if more information about it was available without registering and signing in being necessary, they may be encouraged to use it. Respondents suggested that uptake might be higher if an example action plan was available. Case studies of how the tool worked for organisations of different sizes and from different sectors would also be helpful and help people to decide whether the tool was appropriate for them. Interestingly, Impact Arts said they felt daunted by Measuring Up! initially, but once they started to use it found it easy and useful. This supports the suggestions that case studies and action plan examples being available would be helpful.
- Provide guidance on how long the tool takes to use.
Guidance on how much time it takes to use the tool could allay concerns and encourage use – one interviewee said he had not engaged with Measuring Up! because he did not know whether it would take ‘2 hours or 10 days’.
- Consider a Measuring Up! ‘light’ version.
There was a general feeling amongst interviewees that the language around impact can be difficult to understand. In addition, some survey respondents felt that the process of going through Measuring Up! should be simpler, and that this would be particularly helpful for smaller organisations. Specific suggestions were that different elements of the self-assessment could be marked as ‘essential’ and ‘preferable’, that the language should be simplified and/or a glossary of terms provided.
Inspiring Impact publications
Our survey looked at use of two core Inspiring Impact publications: the Code of Good Impact Practice, which sets out a series of principles for organisations to follow, and Funders’ principles and drivers of good impact practice, which sets out guidance for funders and information on how they can promote good impact practice among organisations and people they support.
Usage and satisfaction
Between June 2013 and May 2015, the Code of Good Impact Practice has been downloaded 2,946 times, and the Funders’ principles and drivers of impact practice 1,559 times. Two-thirds of survey respondents had read in detail or skimmed the Code or its two-page summary and 43% had read in detail or skimmed the funders’ principles. The majority of survey respondents who had read the publications had shared them with others or would consider doing so.
I use the Inspiring Impact publications as learning tools with others, and as a way to initiate conversations with others who are not fully engaged with impact practice.
Judith McComb, Sported
Outcomes and actions: Code of Good Impact Practice
Survey respondents and interviewees were asked if they had learned anything new from reading the Code or its summary version. A variety of improved knowledge was reported, including around data collection, proportionality, embedding impact practice into organisational culture and using evidence to review practice. Respondents have engaged with the core principles outlined in the document, and integrated them into their evaluation plans. Respondents also mentioned they used the publication as a reference tool, for example to provide key definitions of terms, and found it useful to confirm where their strengths in impact practice were.Respondents talked about finding the impact practice cycle and principles particularly helpful, as it had helped to reinforce the importance of all aspects of impact practice, and to do it more strategically:
The Impact Practice Cycle is very useful – it makes more explicit the connection between: planning for impact; doing impact practice; reviewing impact information; and reporting. As an organisation supporting impact practice in the sector for some years it reinforces some of the key messages we have been trying to get across – for example, that impact is not just about tools. We have used it in supporting other organisations to begin to think about their own impact practice in a more joined-up and systematic way. It’s a very useful way of starting the conversations internally at various levels.
The Code of Good Impact Practice is one of those documents that sits on my desk so I can refer back to it.
Jenny Wood, Impact Arts
As a result of reading the Code, 75% said they had either started to review their organisation’s impact practice, or were planning to.Those who had identified their next steps were planning to make improvements in the following areas:
- Prioritisation of impact practice – for example, by looking at impact practice as part of organisational strategic planning, assigning responsibility for impact measurement or planning task groups or discussions with senior managers.
- Data collection.
- Data analysis and reporting.
- Evaluation planning, for example using theory of change or outcomes frameworks.
- Use of data – for example, to improve services or programmes.
Other specific examples of actions were facilitating an internal away day for the senior management team and commissioning an independent evaluation.
Outcomes and actions: Funders’ principles and drivers of good impact practice
Of those who had read the funders’ principles, 31% were planning to make changes to their impact practice and 39% were unsure. Amongst the readers to whom it was applicable (for example those who had grantees), most had either increased their grantee support or were planning to, as a result of reading the resource.
Relevance of Inspiring Impact
There are some additional findings from the evaluation that relate to the Inspiring Impact programme more widely. Some respondents commented that they felt Inspiring Impact was more suitable for smaller organisations, or those earlier on in their impact journey. In addition, some funders felt Inspiring Impact was less relevant to them and more appropriate for charities. This is understandable to a certain extent because Inspiring Impact does have a wide audience and a wide range of resources, but it’s important we consider how to communicate clearly with different groups about what Inspiring Impact can offer them.
Summary and conclusions
Overall, based on this evaluation, those who have engaged with the Inspiring Impact website, resources and tools have:
- found it useful
- improved their knowledge about impact practice, and
- taken positive actions as a result.
- The actions taken are wide ranging and include influencing planning, data collection, analysis, organisational culture and practice, and supporting staff training. These are promising indications that Inspiring Impact is supporting organisations to understand what to measure and how.
The evaluation has also revealed that the following areas could be considered for further focus and development:
- Increasing awareness of Inspiring Impact in the sector, and making it clearer what is available for different audience groups.
- Improving understanding of and access to specific resources, including Measuring Up! and the Impact Hub. This could include using good practice examples and making better use of the additional features, such as the ability to upload and rate resources.
- Clarifying which resources are most appropriate for different types of organisations, their size and their expertise and experience of impact practice, for example by better signposting on the website.
- Encouraging and facilitating the sector to actively contribute to Inspiring Impact. The Inspiring Impact website aims to bring impact tools, resources and information under one roof, to help organisations improve their impact practice. For Inspiring Impact to be even more successful, it relies on the sector itself to contribute, for example by uploading tools to the Impact Hub and reviewing them.
- We are currently developing a network of Impact Champions made up of organisations that themselves are working hard to put good impact practice in place and will support others to do so too through their networks. We expect that they will also encourage the sector to actively engage with tools available.
- Improving access to Measuring Up!, for example by changing the registration requirements, and developing a lighter touch version.
We are currently developing our strategy for the next years of Inspiring Impact, so these areas will be considered by the Inspiring Impact board and developed with the sector where appropriate. We are mindful that we have set ourselves an ambitious aim, and recognise that our best route to reaching more charities and social enterprises over the next seven years is to work with new partners who have excellent reach in the sector. We see this approach as central to our strategy and aim to transform the programme into a UK-wide movement to improve impact practice by working more with partners and networks across the sector. We are continuing to fundraise to develop both the core programme and to pursue further partnership work. This will ensure we can continue on the journey towards our vision: that high quality impact measurement is the norm in the non-profit sector.
If you have any comments or questions please don’t hesitate to get in touch with Rachel, Inspiring Impact programme manager via Rachel.Tait@thinkNPC.org.