Funders guidance - Do

Do

If Plan is about describing the changes that you want to create through your work, and deciding what information to collect, ‘Do’ is about data collection – the process of gathering evidence for how and why your work makes a difference.

This section of Measuring Up! asks you to think about how to collect data, or support others to collect data, in a way that is ethical and robust enough to demonstrate a convincing connection between the goods and services that you provide and the changes that beneficiaries experience as a result of your funding programme.

Do also covers the issue of beneficiary participation. When done properly, involving beneficiaries in collecting data can be empowering for the people involved, and can produce good quality evidence.

It will also be helpful for you to be clear with grantees about the value that you attach to the involvement of end beneficiaries in data collection. At the same time, you will recognise that opportunities for beneficiary engagement will vary according to the resources available, the type of information that is needed, the sort of beneficiaries involved and the benefits and risks to increasing involvement in any particular context.

The guidance in this section includes information on approaches to data collection that is proportionate to your impact measurement goals, your context and the resources you allocate to your own impact practice and that to your grantees.

2.1 We have clear leadership on impact practice, and our grantees and wider stakeholders understand the value we attach to it

This criterion looks at whether or not you as a funder see planning, measuring and making sense of your impact as valuable and important, and integral to your work.

It also concerns how you communicate to a wider audience, including grantees and other funders, the value you attach to impact practice, through the information that you share and make available. Taking a lead in impact practice will have different elements. At several points in your funding cycle you will need to provide information to your grantees on your desired impact.

You will also need to provide information on aspects of impact practice that will help grantees to focus on their own impact and to collect good quality data.

Your grantees’ effectiveness in measuring their outputs and outcomes will depend in large part on the enthusiasm and commitment of the people collecting and using it (their staff and volunteers). Agreeing outputs, outcomes and targets collaboratively and being realistic about their capacity for data collection will help to ensure that this commitment is maintained.

Providing feedback on data that is provided and demonstrating how data is used will also help.

Providing good impact information to your trustees or governing entity based on sound evidence from your grantees can help to keep impact practice a central commitment in your work.

This criterion is fully met if:

People within your project or organisation are clear about their responsibilities for focusing on impact. They communicate the importance of impact measurement to grantees and provide sufficient guidance to grantees to enable them to collect adequate data.

What next?

If you’ve met this criterion in full, you could improve your practice by:

  • Making sure that information about funding programmes, criteria in grant applications and guidance in monitoring meetings all reflect the importance of thinking about and measuring impact
  • Embedding impact practice throughout the grant-making process will lead to a shift in thinking about the value of impact measurement by grantees.
  • Building your own and grantee capacity through increasing knowledge, skills and financial resourcing

The extent to which you can increase the financial resources for impact practice will depend on your own size and capacity as a funder. Even with modest allocated resources, your own shared understanding and skills can be developed. Good practice information shared with grantees can substantially help develop the standard of evidence they produce.

2.2 We have processes in place to follow up when grantees haven’t provided us with adequate information

Whether you are providing unrestricted funding or project and service-specific funding, your monitoring processes should include ways of following up when grantees have not provided you with the necessary and agreed information. This follow up should be timely and allow your grantees to put in place systems to capture information if they are not doing this appropriately. Good communication and collaboration with grantees will help action to be taken to make sure that gaps are not discovered when it is too late.

Information should relate to agreed key outputs and outcomes. Grantees should be encouraged to share decisions about changing planned outputs and also learning about the timescale required to deliver plans and achieve outcomes and targets.

Processes should also check the quality of the information provided. Grantee information should be consistent and delivered on time. Your checking processes should also allow you to assess whether grantees have collected data from beneficiaries in a way that is robust. Providing capacity building in data collection may help your grantees improve the participation of beneficiaries in evaluation and response rates. Your processes should also identify the extent to which data reflects different perspectives and experiences.

Where you are providing small grants or short-term funding you could supplement data collected by your grantees by your own data collection from beneficiaries. This could be done by a beneficiary survey, focus groups or case studies.

This criterion is fully met if:

You check monitoring reports to ensure that information provides good quality evidence of key outputs and outcomes as agreed and there are processes to follow up with grantees on gaps and inconsistencies.

What next?

If you’ve met this criterion in full, you could improve your practice by:

  • Checking that your own data collection and that of your grantees reaches hard-to-reach groups and individuals
  • Including hard-to-reach groups and individuals in your data collection will ensure that your findings reflect a diversity of views. It also means you and your grantees will be able to take steps to make sure that wherever possible groups and people have an equal opportunity to participate, for example by meeting accessibility needs.

2.3 People are supported to collect data accurately and consistently

The information that you collect directly from grantees and the information that you receive from them in monitoring and evaluation reports should be of a good standard. This is necessary in order to provide you with reliable information about the outcomes of specific funded interventions and your wider impact. Data collection tools should be used consistently and appropriately by the people responsible for collecting the data in order to give you the quality and quantity of information that you need.

Ensuring the accuracy of data collected by grantees may present difficulties as you will be one step removed. However, if you provide training, information resources and ad hoc support around data collection to people in your own project or organisation you will be in a better position to offer guidance and support to grantees. Wherever possible and appropriate you may provide additional capacity-building support to your grantees so that their staff and volunteers understand how to collect data properly.

This criterion is fully met if:

Grantees are supported to understand how to apply data collection tools in order to collect information accurately and consistently. People know where to go or who to ask for further support.

What next?

If you have already met this criterion in full, you could improve your practice by:

  • Encouraging those collecting data within your own and your grantee organisations to review data collected for accuracy and consistency, and taking action if necessary
  • Checking that the information gathered is complete and of a sufficient level of quality will help to identify gaps and if this is done within a suitable time frame, it will allow your staff and/or your grantees to implement changes to rectify any data collection issues.

2.4 We encourage the collection of before and after evidence from beneficiaries to see if our funding has made a difference

This criterion looks at the way in which you, your grantees and external evaluators gather information to provide evidence of the changes that result from your funding.

The type and amount of funding you provide, the purpose of the funding and the use that you will make of impact information will affect the standards of evidence that you require. Your priority may be to measure impact in order to learn and reflect on your funded work, to improve your funding programmes and to be accountable to your trustees and other stakeholders for the efficiency and effectiveness of your funding. In this case, it is usually enough to collect ‘before and after’ data which suggests, rather than proves, a logical connection between your funding and the outcomes that beneficiaries experience.

However, if you are funding groups in order to influence policy change or to find out if their way of working/intervention could be replicated successfully by other projects or organisations, the standard of evidence you will need from them will normally be higher. This will involve going to greater lengths to prove and describe the link between the outcomes grantees achieved and the work that they delivered. To gather this sort of information, you will need to encourage the use of more sophisticated ways of collecting data, such as the methods described in the ‘what next?’ tips for practice development below.

In order to understand the extent to which a project or organisation has created change for beneficiaries, there is a need to establish a clear picture of what things were like before and after the intervention. This is usually done by collecting baseline data – data which shows what things were like before the intervention – and then collecting the same information at a later point to show evidence of change. This could be done through a one-off follow up, or a repeated review (to get a more detailed sense of the pace and direction of change). Collecting data in this way is also sometimes referred to as establishing ‘distance travelled‘, since asking the same questions at least twice will show you the amount and type of change that beneficiaries have experienced.

How and when ‘before and after’ data is collected will depend on the chosen tools and an understanding of when beneficiaries achieve different outcomes (see ‘Plan’). If you and your grantees are unable to collect data both before and after the work (for example through a baseline and ‘after’ questionnaire), you can sometimes collect ‘before’ data retrospectively by asking beneficiaries to reflect on how much change they have achieved. However, because this method is more reliant on reflection and memory, it is not always a straightforward way of collecting accurate data.

This criterion is fully met if:

You encourage the collection of evidence that allows the ‘before’ and ‘after’ picture for beneficiaries to be compared and for conclusions to be drawn about the extent to which your funding has created change.

What next?

If you have achieved this criterion in full, you can improve your practice by:

  • Drawing on existing research to compare the changes beneficiaries experience with groups who did not receive such support

Comparing outcome data with research on outcomes for similar groups who did not receive support will give you a more informed picture of the amount of change that is likely to be due to the funded work, and the amount of change that would probably have happened anyway. This will help make a more credible case for the difference the funded work makes.

For example: government research shows that just 35 per cent of homeless young people aged between 16 and 24 maintain a new tenancy when no additional support is provided. However, outcome data could show that for the homeless young people supported by the funded organisation, 65 per cent maintain their tenancy. Using existing data to draw this comparison adds weight to the case that the changes young people experience are down to the funded services.

  • Collecting information from a control group

Funding a control group study will provide the most robust evidence that the outcomes beneficiaries achieve are down to the funded work. This will compare data from your beneficiaries with a group that is the same as your beneficiaries in every way, other than the fact that they have not received support from the funded intervention. This comparison data will give you a detailed, informed sense of the extent to which your work makes a difference.

2.5 We make sure that everyone understands why we are collecting information and how data will be used

This criterion focuses on the processes that you and your grantees have in place for making sure that data is collected ethically. Obtaining informed consent means making sure that beneficiaries have all of the information they need in order to be able to make an informed choice about whether or not to participate.

You will need to make sure that anyone collecting information on your behalf ensures that beneficiaries understand:

  • Why you are collecting information from them
  • How their information will be kept safe and confidential
  • How the information they give you will be used

This will involve explaining to beneficiaries how the findings will be reported and used, so that they can make a choice about whether or not they would like to see what is written about them, or whether they would like to remain anonymous.

This criterion is fully met if:

You and your grantees have processes in place to make sure that everyone who participates understands why information is being collected from them and how the information they share will be used and presented.

What next?

If you have already met this criterion in full, you can improve your practice by:

  • Supporting your grantees to think through any longer-term risks for people participating in impact measurement and providing appropriate support

Obtaining informed consent will help those taking part in the evaluation to think through any immediate risks involved in participating (for example, being identified as someone who uses particular goods and services). However, if your grantees are working with vulnerable people or collecting data on sensitive topics, they will need to think carefully about whether data collection activities might carry a longer-term risk for participants, such as emotional distress. In these cases, they will need to provide information about, or access to, further support in order to meet ethical obligations.

Resources for this section

Choosing data collection tools

Information collection methods: choosing tools for assessing impact is a practical NCVO Charities Evaluation Services handbook aimed at helping to identify and develop ways to collect information on the outcomes of one’s work. It provides a guide through planning the outcome monitoring, selecting appropriate methods, developing tools, preparing and collection of the outcomes information.

Collecting data ethically

Ethical guidelines is a set of published guidelines from Social Research Association that set out what ethical practice looks like in research and evaluation.

Storing data respectfully and legally

The Information Commissioner’s Office is, the UK’s independent authority set up to uphold information rights in the public interest, promoting openness by public bodies and data privacy for individuals. This website contains useful guidance about how to keep personal data in line with the Data Protection Act.

Useful websites

Inspiring Impact’s Impact Hub has a large number of impact and outcome measurement tools and scales.

Evaluation Support Scotland has a series of tools available from their website to support those interested in undertaking self-evaluation.

The Association of Research in the Voluntary and Community Sector (ARVAC) has a handbook on community research that has guidelines and templates to support any community group that wishes to undertake research/evaluation.

Sign-up to the principles

We need your help to put principles into practice