Medium to large voluntary organisations guidance - Do

Do

If Plan is about describing the changes that you want to create through your work, and deciding what information to collect, Do is about data collection – the process of gathering evidence for how and why your work makes a difference.

This section of Measuring Up asks you to think about how to collect data in a way that is ethical, streamlined to your way of working, and robust enough to demonstrate a convincing connection between the goods and services that you provide and the changes that beneficiaries experience.

Do also covers the issue of beneficiary participation. Opportunities for beneficiaries to get involved in data collection will vary from organisation to organisation, depending on the resources available, the type of information you need to collect, the sort of beneficiaries you work with and the benefits and risks to increasing involvement in your particular context. However, when done properly, involving beneficiaries in collecting data can be empowering for the people involved, and produce good quality evidence.

The standard of ‘proof’ you require will influence your approach to data collection. For many organisations, collecting ‘before and after’ data that strongly suggests a link between the work you do and the changes that beneficiaries experience will be evidence enough. However, if your organisation has greater resources to dedicate to impact measurement, or if you are measuring your impact in order to influence policy change or to see if other people should adopt your way of working, you will need to go to greater lengths to prove the link between your work and the changes you observe. The guidance in this section includes information on how to pick an approach to data collection that is proportionate to your resources, impact measurement goals and context.

2.1 We have clear leadership on impact practice, and everyone understands its value to our organisation

This criterion looks whether or not everyone in your organisation sees measuring and making sense of your impact as valuable and important work. It also asks about the extent to which thinking about impact has been embedded in your everyday work, becoming an integral part of how you operate, rather than a separate task.

How effective you are at measuring your impact is dependent in large part on the enthusiasm and commitment of the people collecting and making sense of the evidence (your staff and volunteers), and the people receiving the information (your managers and trustees). Making sure that everyone understands the value of measuring your impact is critical to collecting good quality data.

An important part of this involves having clear leadership around impact, whether this is a person, a team or a working group. This will allow you to make a convincing case for impact measurement, to promote it as a valuable activity within your organisation, and to drive it forwards.

However, effective leadership on impact doesn’t have to take a ‘top-down’ approach. Involving people from all levels of your organisation, staff, volunteers, trustees and even beneficiaries, will help you to spread understanding and enthusiasm for impact measurement throughout your organisation.

This criterion is fully met if:

You have a named person or group responsible for focusing on the organisation’s impact. Everyone in the organisation sees impact measurement as a worthwhile activity.

What next?

If you’ve met this criterion in full, you could improve your practice by…

  • Making sure that thinking about impact is seen as a fundamental part of each role, rather than an afterthought

Including impact in the way peoples’ roles are defined and reviewed will help people to see it as a core part of their everyday work. You might put information about impact measurement in job descriptions, or include discussing impact as a regular item during staff supervision and team meetings.

  • Making impact measurement a regular item in team meetings, supervision and appraisals

Being able to discuss problems with how evidence is collected, as well as celebrate findings and successes, will help to deepen the enthusiasm, involvement and understanding of staff, volunteers and trustees.

2.2 We collect the right amount of evidence, from the right sources, in order to measure our impact effectively

Sometimes, because of limited resources or logistical issues, it might not be possible to collect data from everyone you work with. In these cases you will have to think about collecting information from a sample (a smaller selection from a larger group).

However, decisions around how much data to collect and from whom will be determined by the purpose of your sample. If you want to be able to generalise (draw conclusions about all of your beneficiaries from the data you collect from a sample), you will need to think carefully about your sample size, to make sure that it gives you the right amount of data, and also about the representativeness of your sample – that is, whether or not all groups are included.

However, there may be times when you want collect data from different groups in order to get a more in-depth picture of the amount and type of change that particular beneficiaries experience as a result of your work. For example, if you wanted to understand more about what the barriers and enabling factors were for achieving positive outcomes, you might decide to separate out data from beneficiaries who had achieved excellent outcomes, and beneficiaries who had achieved very poor outcomes. This is called selective sampling. In this case, you will need to make sure that your sample contains within it the right groups in order to tell you what you most want to know, rather than making sure that everyone is equally represented.

There are many different methods of building a good sample, and different approaches to suit the type of data that you want to collect. However, your resources will also be a big consideration when you decide the size and scope of your data collection, both in terms of how much data you can realistically gather, and how much time you can spend on making sense of it during analysis. If you are measuring your impact to learn more about the difference your work makes and to report back to funders and commissioners, consider your available resources and decide on what the largest sample size is that you can feasibly manage. Taking a realistic approach to sample size, and focusing on making sure that the right groups are represented for the purpose of your sample (either to generalise about all beneficiaries, or to gather detailed information about particular groups), will help you to gather useful information that you can use to evidence your impact.

If you are measuring your impact to see if other people could adopt your way of working or to influence national policy, you may well need to gather data from a larger sample, to meet a higher standard of evidence. You will also need to rely on more technically sophisticated methods for setting your sample.

This criterion is fully met if:

The size and scope of your data collection suits the type of data that you want to collect, is in proportion to your available resources, and allows you to draw conclusions about your impact across the different groups you work with.

What next?

If you have met this criterion in full, you could improve your practice by:

  • Checking during data collection that all relevant groups are represented in your data, especially hard to reach groups.

Checking as you go along to make sure that you are gathering data from the right groups will give you time to make adjustments if necessary.

  • Collecting additional data if necessary to make sure that all relevant groups are represented.

If your checking processes reveal a gap, you will need to gather additional data to be sure of having a sample that will give you the information you need.

2.3 We have processes in place to follow up when beneficiaries haven’t provided us with information.

This criterion is about managing response rate – the number of people who decide to participate by giving you information, out of the total number of people that you ask to participate.

Getting enough beneficiaries to participate is crucial to gathering enough data to be able to make decisions about the difference your work is making. The way you plan to collect your evidence will help you with this to an extent – for example, if people find your data collection tools easy to respond to and if your tools are well suited to the people you want to collect data from, they are more likely to give you all of the information you need. However, to encourage people to participate in impact measurement, you will also need to think carefully about collection.

It is important to keep track of who has provided you with information, and to have a plan for following up with people to encourage them to participate if they’ve not done so already, or if they have provided incomplete answers. Whatever strategy you devise will need to be in proportion to your available resources. For example, you might decide to re-contact people about an online survey a maximum of three times. Or, if you wanted to collect data by interviewing people six months after you had finished supporting them, you could take steps to collect contact information from beneficiaries so that you could keep in touch, and provide information about the interviews in local community centres or through service user forums.

This criterion is fully met if:

You have a process in place for following up with people who do not respond or who don’t provide all the information that you need. Your process for following up with people suits the resources you have available.

What next?

If you have already met this criterion in full, you can improve your practice by:

  • Meeting individual accessibility needs wherever possible to make sure that everyone has an equal opportunity to participate

Thinking about how you can apply your data collection tools flexibly will help you to gather responses from groups with different needs. For example, you might be able to carry out your questionnaire over the phone or in person with people with limited literacy, or have an interview in a location that is more accessible.

  • Developing a proactive plan to ensure a good response rate from harder to reach groups.

Harder to reach groups may well be those that you are most interested in getting a good response rate from. Thinking ahead about how to encourage their participation in impact management – for example, by promoting your evaluation through online forums or local community centres, or using interpreters during data collection – will help you to make sure that you can collect the information you need.

2.4 We test out new tools to check for any problems, and we make changes as necessary

This criterion looks at the way your organisation manages the quality of the data you collect, by testing out your data collection tools.

Making sure your tools are fit for purpose involves piloting – road testing them for a short period, and amending them if necessary. It is not unusual for the piloting stage to turn up unexpected issues or to generate poor quality and/or unusable data, so it is very important to conduct a pilot before rolling out a new data collection tool.

Your piloting timeframe should be long enough to tell you if something is wrong with your tool. This means waiting until the people collecting the data have got used to using it, and until they have tried it out in all possible settings, with all the groups that you want to collect data from.

When testing your tools, you need to consider the following key points:

Validity

Does the tool collect the information that you wanted to collect? Or is it gathering information about something you are less interested in?

Reliability

Does it work equally well with all the groups that you want to collect information from? Can it collect information consistently?

Usability

Do the people collecting the information find it easy to use?

Looking through your data you gathered during the pilot will often help you to answer at least some of these questions. However, gathering feedback from the people who collect information is also useful, especially in terms of deciding what to change about your tools.

This criterion is fully met if:

You pilot all your data collection tools fully, and check them for validity, reliability, usability. Amends are made, if necessary, before the tool is rolled out.

What next?

If you have already met this criterion in full, you can improve your practice by:

  • Discussing possible improvements to your tools with beneficiaries, staff and volunteers.

Talking to the people responsible for collecting the data, and for the groups you want to collect data from, will give you useful information about how to change and adapt your tools to improve the quality of the data they collect.

  • Reviewing your tools to make sure they remain relevant over time.

Over time, things can change, such as the balance of skills across your staff team, your information needs, and even your beneficiary groups. Reviewing your tools periodically will help to make sure that they remain effective in your particular context.

2.5 People are supported to collect data accurately and consistently

Your data collection tools will have to be used consistently and appropriately by the people responsible for collecting the data in order to give you the quality and quantity of information that you need. This means providing training, information resources, and ad hoc support around data collection to make sure that everyone understands how to collect data properly.

This criterion is fully met if:

Everyone responsible for data collection understands how to apply data collection tools in order to collect information accurately and consistently. People know where to go or who to ask for further support.

What next?

If you have already met this criterion in full, you could improve your practice by:

  • Reviewing the data you collect for accuracy and consistency, and taking action if necessary

Checking the information you have for completeness and quality will help you to understand if there are any gaps in the quality or consistency of data collection. Some databases will allow you to check data in this way, although alternatively you could pick a sample of data at random to review the way it was collected (this is also known as dip sampling). Reviewing your information in this way will help you to take action where you find inaccurate data.

  • Including data collection in staff reviews and appraisals

Including data collection in team reviews and appraisals will help you to address issues around poor quality or inconsistent collection, and to identify training or support needs.

2.6 We collect before and after evidence from our beneficiaries to see if our work has made a difference

This criterion looks at the way in which you gather information to provide evidence of the changes that result from your work.

Different organisations will require different standards of evidence to demonstrate that changes have occurred, and that these changes result from the goods and services that they provide. If your organisation is interested in measuring its impact for the purposes of learning more about your work, or for reporting back to funders, it is usually enough to provide ‘before and after’ data which suggests, rather than proves, a logical connection between the work that you do and the outcomes that beneficiaries experience.

However, if you are measuring your impact in order to influence policy change or to find out if your way of working could be replicated successfully by other organisations, the standard of evidence you will need will normally be higher. This will involve going to greater lengths to prove and describe the link between the outcomes beneficiaries achieved and the work that you delivered. To gather this sort of information, you will need to use more sophisticated ways of collecting data, such as the methods described in the ‘what next?’ tips for practice development below.

In order to understand the extent to which your organisation has created change for beneficiaries, you need to establish a clear picture of what things were like before and after your work. This is usually done by collecting baseline data – data which shows what things were like before your work – and then collecting the same information at a later point to show evidence of change. This could be done through a one-off follow up, or a repeated review (to get a more detailed sense of the pace and direction of change). Collecting data in this way is also sometimes referred to as establishing ‘distance travelled’, since asking the same questions at least twice will show you the amount and type of change that beneficiaries have experienced.

How and when you collect ‘before and after’ data will depend on your chosen tools and your understanding of when beneficiaries achieve different outcomes (see ‘Plan’). If you are unable to collect data both before and after your work, you can sometimes collect your ‘before’ data retrospectively by asking beneficiaries to reflect on how much change they have achieved, but because this method is more reliant on reflection and memory, it is not always a straightforward way of collecting accurate data.

This criterion is fully met if:

You gather evidence that allows you to compare the ‘before’ and ‘after’ picture for beneficiaries, and which allows you to draw conclusions about the extent to which your work has created change.

What next?

If you have achieved this criterion in full, you can improve your practice by:

  • Drawing on existing research to compare the changes beneficiaries experience with groups who did not receive our support

Comparing your outcome data with research on outcomes for similar groups who did not receive support will give you a more informed picture of the amount of change that is likely to be due to your work, and the amount of change that would probably have happened anyway. This will help you to make a more credible case for the difference your work makes.

For example: government research shows that just 35% of homeless young people aged between16–24 maintain a new tenancy in an area where no additional support is provided. However, your outcome data shows that for the homeless young people that you support, 65% maintain their tenancy. Using existing data to draw this comparison adds weight to the case that the changes young people experience are down to your work.

  • Collecting information from a control group

Collecting your own information from a control group – a group that is the same as your beneficiaries in every way, other than the fact that they have not received your support – provides the most robust evidence that the outcomes beneficiaries achieve are down to your work. Comparing data from your beneficiaries and your control group will give you a detailed, informed sense of the extent to which your work makes a difference.

2.7 The level of involvement that beneficiaries have in our data collection processes suits our ethos, our information needs, and our available resources

This criterion looks at the way your organisation consults and involves beneficiaries in your data collection processes.

Beneficiary involvement in data collection is a continuum, which runs from simple consultation (such as gathering feedback on the way you collect information) through to ownership, where beneficiaries take the lead on planning, data collection, analysis and learning. When deciding the level of involvement that is appropriate for your organisation, you will need to consider the following issues:

Ethos and values

If your organisation has an empowerment focus or a ‘service-user led’ philosophy underpinning its work, actively involving beneficiaries in data collection may be particularly relevant to your context. If properly supported and resourced, involvement may also contribute to what you hope to achieve for the people who use your goods and services.

Data quality

In certain contexts, ex-service users may well be able to gather more honest and detailed information than your staff and volunteers. However, to ensure data quality, you will need to train, develop and supervise beneficiaries as they prepare for and go about data collection, in exactly the same way that you would support staff and volunteers.

Resources

In general terms, the greater the level of involvement, the more resources you will need to support beneficiaries. This could include providing training and support, supervision, IT support, or equipment. You will also need to factor in time for fully de-briefing beneficiaries after their participation, to check that their support needs have been fully met.

Ethical considerations

If your beneficiaries are vulnerable people, you will need to think carefully about whether there is any aspect of their involvement which could potentially increase their vulnerability, endanger their safety and wellbeing, or in any way undo some of the positive changes they have experienced as a result of your work.

This criterion is fully met if:

The role that beneficiaries play in data collection is appropriate to your ethos, the sort of information that you need to collect, and the resources that you are able to dedicate to beneficiary involvement.

This criterion looks at the level of appropriateness, rather than the level of involvement itself. If you have considered all the factors above and decided that there is no appropriate role for beneficiaries in your data collection processes, you will still meet this criterion in full.

What next?

If you have met this criterion in full, you could improve your practice by:

  • Periodically reviewing the role that beneficiaries play in your data collection processes.

This will make sure that you have an up to date picture of the resources, risks and benefits of beneficiary involvement.

  • Including beneficiary participation in the way you assess and manage resources for collecting evidence

Thinking through the resource implications of getting beneficiaries involved in the way you measure your impact, and including this in the way you think about resourcing evidence collection, will help you to put together a short- and longer-term strategy for resourcing beneficiary participation.

2.8 We make sure that everyone understands why we are collecting information and how their data will be used

This criterion focuses on the processes that your organisation has in place for making sure that data is collected ethically. Beneficiaries have the right to opt out of providing information, and a right to be informed about how you plan to store and look after the information that they provide. Obtaining informed consent means making sure that beneficiaries have all of the information they need from you in order to be able to make an informed choice about whether or not to participate.

Your organisation will need to take steps to make sure that beneficiaries understand why you are collecting information from them, as well as how you plan to keep it safe and confidential, and how you will use it. This will involve explaining to beneficiaries how you plan to report on and use your findings, so that they can make a choice about whether or not they would like to see what you write about them, or whether they would like to remain anonymous.

This criterion is fully met if:

You have processes in place to make sure that everyone who participates understands why you are collecting information from them, and how the information they share will be used and presented.

What next?

If you have already met this criterion in full, you can improve your practice by:

  • Routinely giving people information about their rights as a research participant

In addition to obtaining informed consent, routinely providing people with more in-depth information about their rights as a research participant (for example, being able to withdraw at any point and to see whatever information your organisation holds about them) will help to inform and empower the people you are collecting information from.

  • Thinking through any longer-term risks for people participating in impact measurement and providing appropriate support

Obtaining informed consent will help beneficiaries to think through any immediate risks involved in participating (for example, being identified as someone who uses your goods and services). However, if you are working with vulnerable people or collecting data on sensitive topics, you will need to think carefully about whether your data collection activities might carry a longer-term risk for participants, such as emotional distress. In these cases, you will need to provide information about, or access to further support in order to meet your ethical obligations.

Resources for this section

Choosing data collection tools

Charities Evaluation Services, Assessing Change: Developing and Using Outcomes Monitoring Tools (2010)

This guide is a practical handbook aimed at helping to identify and develop ways to collect information on the outcomes of one’s work. It provides a guide through planning the outcome monitoring, selecting appropriate methods, developing tools, preparing and collection of the outcomes information.

Involving beneficiaries in data collection

Voluntary Action Westminster, Involving People – A Practical Guide

This guide contains information about involving people in data collection, and discusses a range of data collection methods that involve greater participation of beneficiaries.

Collecting data ethically

The Social Research Association has published guidelines that set out what ethical practice looks like in research and evaluation.
These can be accessed at http://the-sra.org.uk 

Useful websites

CYFERnet is an online, free, interactive evaluation resource. Users can access brief and informative ‘learning modules’ through the site on a range of issues, including:

  • Sampling
  • Data collection
  • Research methods

There are sample questions to test knowledge, video links to further explanations by evaluation experts and many other useful tools.
The site can be accessed at http://cyfernetsearch.org/

Sign-up to the principles

We need your help to put principles into practice