If Plan is about describing the changes that you want to create through your work, and deciding what information to collect, ‘Do’ is about data collection – the process of gathering evidence for how and why your work makes a difference.
This section of Measuring Up! asks you to think about how to collect data in a way that is ethical, and streamlined to your way of working.
Do also covers the issue of beneficiary participation. Opportunities for beneficiaries to get involved in data collection will vary from organisation to organisation, depending on the resources available, the type of information you need to collect, the sort of beneficiaries you work with and the benefits and risks to increasing involvement in your particular context. However, when done properly, involving beneficiaries in collecting data can be empowering for the people involved, and produce good quality evidence.
The standard of ‘proof’ you require will influence your approach to data collection. For many organisations, collecting ‘before and after’ data that strongly suggests a link between the work you do and the changes that beneficiaries experience will be evidence enough. However, if your organisation has greater resources to dedicate to impact measurement, or if you are measuring your impact in order to influence policy change or to see if other people should adopt your way of working, you will need to go to greater lengths to prove the link between your work and the changes you observe.
The guidance in this section includes information on how to pick an approach to data collection that is proportionate to your resources, impact measurement goals and context.
2.1 We collect proportionate amounts of evidence from our beneficiaries in order to measure our impact effectively
Data should not be collected for data’s sake. Only collect the data you really need. Sometimes, because of limited resources or logistical issues, it might not be desirable or possible to collect data from everyone you work with. In these cases you will have to think about collecting information from a sample (a smaller selection from a larger group).
However, decisions around how much data to collect and from whom will be determined by the purpose of your sample. If you want to be able to generalise (draw conclusions about all of your beneficiaries from the data you collect from a sample), you will need to think carefully about your sample size, to make sure that it gives you the right amount of data, and also about the representativeness of your sample – that is, whether or not all groups are included.
This criterion is fully met if:
The size and scope of your data collection suits the type of data that you want to collect and is in proportion to your available resources.
If you have met this criterion in full, you could improve your practice by:
- Checking during data collection that all relevant groups are represented in your data, especially hard to reach groups
- Checking as you go along to make sure that you are gathering data from the right groups will give you time to make adjustments if necessary.
2.2 Before using any data collection tools we test (pilot) them to make sure they are fit for our purposes
This criterion looks at the way your organisation manages the quality of the data you collect, by testing out your data collection tools. Tools that you could use might include a survey, interview and focus group guides.
Making sure your tools are fit for purpose involves piloting – road testing them for a short period, and amending them if necessary.
Trying out the analysis you plan to use on the data that comes from the tools can also help you to identify any design issues. For example, reading through all the comments from a training feedback form and tallying up the most frequent responses means you can check that the questions you are asking give you the type of information you need.
It is not unusual for the piloting stage to turn up unexpected issues or to generate poor quality and/or unusable data, so it is very important to conduct a pilot before rolling out a new data collection tool.
Your piloting timeframe should be long enough to tell you if something is wrong with your tool. This means waiting until the people collecting the data have got used to using it, and until they have tried it out in different settings, with all the groups that you want to collect data from. The people who are collecting the data will be able to give you the most useful feedback on whether the tool collects the information you want it to and how easy it is to use.
This criterion is fully met if:
You pilot your data collection tools and check with those using the tool whether it needs to be changed. Amendments are made, if necessary, before the tool is rolled out.
- If you have already met this criterion in full, you can improve your practice by:
Considering the reliability and validity of your tools
- Validity – does the tool collect the information that you wanted to collect? Or is it gathering information about something else?
- Reliability – does it work equally well with all the groups that you want to collect information from? Can it collect information consistently?
- Reviewing your tools periodically to ensure this will help to make sure that the data you collect is robust.
2.3 People are supported to collect data and understand what we need to collect and why
Your data collection tools will have to be used consistently and appropriately by the people responsible for collecting the data in order to give you the quality and quantity of information that you need. This might mean accessing training, information resources, and ad hoc support around data collection, making sure that everyone understands how to collect data properly.
It is important to support your team to understand the importance of collecting robust data in a systematic way and the value of this data in terms of organisational learning and accountability to your stakeholders.
This criterion is fully met if:
Everyone responsible for data collection understands how to apply data collection tools in order to collect information accurately and consistently and why your organisation does this. Also, that people know where to go or who to ask for further support.
If you have already met this criterion in full, you could improve your practice by:
- Including data collection in staff and volunteer reviews and appraisals. This will help you to address issues around poor quality or inconsistent collection, and to identify training or support needs.
2.4 We make sure that everyone understands why we are collecting information from them and how their data will be used
This criterion focuses on the processes that your organisation has in place for making sure that data is collected ethically. Beneficiaries have the right to opt out of providing information, and a right to be informed about how you plan to store and look after the information that they provide. Obtaining informed consent means making sure that beneficiaries have all of the information they need from you in order to be able to make an informed choice about whether or not to participate.
Your organisation will need to take steps to make sure that beneficiaries understand why you are collecting information from them, as well as how you plan to keep it safe and confidential, and how you will use it.
This criterion is fully met if:
You have processes in place to make sure that everyone who participates understands why you are collecting information from them, and how the information they share will be used and presented.
If you have already met this criterion in full, you can improve your practice by:
- Enabling people to be involved in the evaluation process anonymously. This will involve explaining to beneficiaries how you plan to report on and use your findings, so that they can make a choice about whether or not they would like to remain anonymous. You can offer the opportunity for them to see what you write about them to give their approval to use their name, request changes or stay anonymous.
2.5 We store and use people’s data safely, respectfully and legally
As part of collecting evidence, you might have to handle sensitive personal information about your beneficiaries. This personal data must be managed in keeping with the Data Protection Act if your organisation is to fulfill its legal responsibilities in terms of keeping data safe and using data appropriately.
Briefly, the main points of the Act state that personal data should be:
- Kept accurate and up to date Used only for the purpose for which it was originally collected
- Not transferred outside the European Economic Area without adequate protection.
- Kept for only as long as is necessary
- Only gathered if strictly needed
- Kept safe from loss, damage and unauthorized access
This criterion is fully met if…
The way your organisation stores and uses the data it collects for impact measurement meets the legal requirements of the Data Protection Act in full.
If you’ve met this criterion in full, you could improve your practice by…
- Developing a written policy on data protection and its relevance to impact measurement
- Developing and sharing a written policy on data protection will create a helpful resource for making sure that everyone in your organisation understands the procedures in place around the legal storage and use of personal data.
- Resources for this section
- Choosing data collection tools. The questions to consider in choosing your data collection methods are outlined on the NCVO Charities Evaluation Services website.
Involving beneficiaries in data collection
Why Bother Involving People in Evaluation is an ESS published workbook to help voluntary organisations to plan why, when and how to involve the people that you work with in evaluation.
Good Practice in User Involvement is a Big Lottery document based on a series of good practice examples identified from the evaluation of BIG’s Research Grants programme. It offers examples about what has worked for other research projects and is designed to help voluntary organisations with engaging users in the planning and delivery of research.
Involve is a think tank that promotes citizenship engagement. It has a series of publications and guidelines to help you to consider how best to engage your beneficiaries.
Collecting data ethically
Research ethics is a set of published guidelines from Social Research Association that set out what ethical practice looks like in research and evaluation.
Storing data respectfully and legally
The Information Commissioner’s Office (ICO) is, the UK’s independent authority set up to uphold information rights in the public interest, promoting openness by public bodies and data privacy for individuals. This website contains useful guidance for organisations about how to keep personal data in line with the Data Protection Act.
Inspiring Impact’s Resource Hub has a large number of impact and outcome measurement tools and scales.
Evaluation Support Scotland has a series of tools to support groups to run self-evaluation.
The Association of Research in the Voluntary and Community Sector has a handbook on community research that has guidelines and templates to support any community group that wishes to run research/ evaluation.