Medium to large voluntary organisations guidance - Assess

Assess

This section covers data analysis – the process of bringing together the evidence that you have collected, and making sense of it in order to understand how much change has happened, for whom, and why.

In addition to covering key steps in analysis, such as comparing groups and making sense of change, this section also explains the questions you will need to ask yourself in order to get an objective picture of your impact overall. This includes thinking about the other factors that could have contributed to your impact, as well as considering how long changes last for, and how much change might have happened independently of your work. It also involves considering whether your own bias or the way you collected the data could have influenced your results.

3.1 We store and use people’s data safely, respectfully and legally

As part of collecting evidence, you will have to handle sensitive personal information about your beneficiaries. This personal data must be managed in keeping with the Data Protection Act if your organisation is to fulfil its legal responsibilities in terms of keeping data safe and using data appropriately.

There is a large amount of useful guidance available on how third sector organisations should handle personal data in a way that meets the Data Protection Act. Briefly, the main points of the Act state that personal data should be:

  • Kept accurate and up to date
  • Used only for the purpose for which it was originally collected
  • Stored only within the European Economic Area (since the Data Protection Act only stands in Europe. This is important if you are storing data on an online platform.)
  • Kept for only as long as is necessary
  • Only gathered if strictly needed
  • Kept safe from loss, damage and unauthorised access

This criterion is met in full if:

The way your organisation stores and uses the data it collects for impact measurement meets the legal requirements of the Data Protection Act in full.

What next?

If you’ve met this criterion in full, you could improve your practice by…

  • Developing a written policy on data protection and its relevance to impact measurement

Developing and sharing a written policy on data protection will create a helpful resource for making sure that everyone in your organisation understands the procedures in place around the legal storage and use of personal data.

  • Routinely sharing your data protection policy with the people you collect information from: explaining your policy to beneficiaries will help them to understand your responsibilities towards them as the data handler, as well as their own data rights as impact measurement participants.

3.2 We have IT systems in place that allow us to input and analyse our data quickly and easily

Making sure that data about your beneficiaries and the work you are doing with them can be stored securely, analysed and reported quickly and easily is a crucial part of ensuring impact measurement is supported by staff and volunteers across the organisation. This is particularly important for organisations that deliver a range of projects and programmes and need to track beneficiaries experience across different services.

There are a growing number of project management systems and outcome reporting platforms that have been developed to support organisations to help manage and measure the impact of their work. These include bespoke databases and IT systems that have been developed for specific types of organisation and online platforms that integrate with third party data collection, management and reporting tools. Organisations should decide which is most appropriate for them and ensure staff and volunteers are supported to use them to manage their work so that data is readily available for the wider purposes of demonstrating impact and contributing to service improvement.

Charities and social enterprises delivering front line services should be prepared to invest around 1% of their operational budgets in their monitoring and evaluation infrastructure. This will ensure that they have the best chance of becoming proficient in collecting and managing data, reflecting on it and using it to inform their practice.

This criterion is fully met if:

Data entry is quick and easy for front line staff, it is secure and can be aggregated and analysed to demonstrate impact and inform service improvement.

What next?

If you’ve met this criterion in full, you could improve your practice by…

  • Storing your data in a way that makes checking for missing or incorrect data quick and easy

Checking for missing or incomplete data is far easier to do once your data is all in one place. This will help you to understand and improve your data quality.

  • Storing your data in a way that means you can automatically generate reports

There are obvious time savings involved with automatic reporting. Automatic reporting will help you to prepare the information for analysis and to work with your data much more easily.

3.3 We compare different types of information to make sense of how and why changes occur for beneficiaries

This criterion focuses on the way in which you bring your evidence together during analysis to explain how and why your work makes a difference.

Explaining why changes occur is vital to understanding your impact. Using your data to explain change requires two steps to analysis. Firstly, you will need to analyse each type of data separately to make sense of it. Secondly, you will need to bring together different pieces of information during analysis to try and explain why changes happen.

This might involve bringing together different types of data about the same beneficiaries. For example, you could compare quantitative data from a questionnaire, which tells you how many people experienced a particular outcome, with qualitative data from interviews where the same people provided in-depth information about how they experienced different outcomes and why they thought those outcomes were achieved.

You may also want to compare different perspectives on change to try and understand why outcomes occur. For example, if you were working with young people, you might collect data from the young people themselves about outcomes, and from their parents and carers as well. Examining the way in which the two groups give different or similar explanations for change will help you to get a more in-depth picture of why outcomes occur.

This criterion is fully met if:

You compare and contrast different types and sources of data, in a way that helps you to understand why beneficiaries experience outcomes as a result of your work.

What next?

If you’ve met this criterion in full, you could improve your practice by…

  • Identifying strong and weak findings by looking at the way in which different data sources support or contradict each other.

Reviewing the way in which different sources of information support or contradict each other will give you a sense of which of your findings are strongest. It will also help you to identify where your data is still inconclusive, and where you might need to collect more information.

  • Exploring your initial findings with key stakeholders during analysis

Exploring contradictory or inconclusive data with your beneficiaries, staff and volunteers during analysis will help you to fill in any gaps and to improve your understanding of your findings. This process of presenting and discussing your findings to build meaning during analysis is called iteration and is a useful tool for adding quality and depth to your findings.

3.4 We check to see if different groups of beneficiaries experience different amounts or different types of change as a result of our work

This criterion looks at the way you compare and contrast outcome data for different groups of beneficiaries during analysis, to find out whether different groups experience different changes as a result of your work.

Trends and patterns in outcome data for the whole of your beneficiary group may hide important differences between different groups. Separating out data for different groups in order to compare and contrast the amount and type of change that they experience – also described as disaggregation – is an important step in your analysis, since it is crucial to understanding whether or not your work is reaching everyone in your target group.

Disaggregation will often throw up important questions and learning points. For example, if particular ethnic groups or age bands are achieving better outcomes from your work, does this indicate a particular strength in the way you deliver your work that your organisation can learn from, or that the services you provide are not relevant or appropriate for everyone in your target group?

This criterion is fully met if:

You compare and contrast outcome data for different types of beneficiary in order to assess whether different groups experience different amounts or types of change.

What next?

If you’ve met this criterion in full, you could improve your practice by…

  • Drawing in different sources of information and different perspectives to explain any differences between groups

Including disaggregation in the way you bring data together to understand change (see 3.3) will add depth to the way you understand your organisation’s impact.

  • Discussing your findings about the differences between groups with beneficiaries, staff and volunteers

Taking unexplained differences between groups back to your staff, volunteers and beneficiaries will help you to collect additional perspectives and information to explore and explain how and why groups experience different outcomes.

3.5 We look carefully at negative and unexpected outcomes, as well as positive outcomes

Considering your findings objectively is a tricky but fundamental part of data analysis. This criterion focuses on whether or not your analysis gives equal weight to positive and less positive findings.

A danger of all research is that, without objectivity, your data can end up reflecting a rosy picture of what you hoped to find, rather than an accurate picture of how things really are. This can mean that any negative or unplanned outcomes are ignored during analysis. However, the most important learning for your organisation may rest with the negative or unplanned outcomes that you discover, so it is important that these outcomes are given due weight during analysis.

Treating negative or unexplained outcomes with the same amount of care and attention as positive outcome data means spending the same amount of time on analysis, and asking the same key questions about why changes occur, who else is involved in creating change, and how experiences differ between groups.

This criterion is fully met if:

Negative and/ or unexpected outcomes are analysed just as carefully as positive and planned ones.

What next?

If you’ve met this criterion in full, you could improve your practice by…

  • Thinking carefully about how the way you collected data might have influenced your findings

Reviewing the way you collected your information will help you to make objective judgements about how important your findings (negative or positive) really are. This includes considering any technical problems during data collection or issues with missing data, as well as reflecting on how your own bias might have influenced the way you interpreted the information you collected.

  • Seeking external review and validation of your findings

Having your processes and findings reviewed externally, by peers or experts, will add another layer of scrutiny and objectivity to your impact measurement.

3.6 We think through the other factors that might have influenced the outcomes that beneficiaries experience

In order to fully understand your impact, your analysis will need to consider the question of attribution – that is, an assessment of how much change was down to your work, and how much was down to the work of others.

Change is complex: in every context, there will always be a number of other agents involved in creating outcomes, including other organisations, other professionals, and even beneficiaries’ families and friends. This means that it will not be possible to come up with a precise figure or percentage of the change that is due to your work alone. However, being able to recognise and describe the role that others play in achieving outcomes for beneficiaries is an important part of data analysis.

Making a sensible estimation of attribution, based on your professional experience and your understanding of your context is a good starting point. You should also consider which other players were involved in creating change, both at the same time as your work, and also in terms of laying the foundation for your work with beneficiaries.

This criterion is fully met if:

You use what you know about the involvement of others in creating change to draw sensible conclusions about the extent to which other actors or circumstances may have supported or held back the achievement of your planned outcomes.

What next?

If you’ve met this criterion in full, you could improve your practice by…

  • Collecting detailed information from beneficiaries about how other people support or undermine change

Asking beneficiaries directly about what proportion of the changes that they experienced was down to your work alone, compared to the influence of other people or factors, will help you to develop a more in-depth understanding of attribution. For example, you might ask people about the percentage of change that was down to your work, and the percentage that was down to any other organisations involved.

  • Considering the way in which broader socio-economic factors support and hold back change

Drawing in wider socio-economic factors which are out of your control as an organisation – for example, widespread unemployment or climate change – will help you to understand your impact in its wider context.

3.7 We think through how much change might have occurred anyway for beneficiaries, without our work

It is possible that some of the changes that beneficiaries experience would have occurred ‘naturally’, that is, independently of your work. This amount of change that would have happened without your intervention is also known as deadweight. It is important to consider deadweight during analysis, since understanding how much change would have occurred anyway will help you to get a more accurate picture of your contribution to change, and the real difference that your work makes.

Thinking about change in this way is sometimes called considering the counterfactual – what would have happened anyway without your intervention, or the changes that would have occurred if your organisation did not exist.

Estimating your deadweight during analysis is not straightforward. For most organisations, considering the following questions should help you to build up a reasonable estimation of your deadweight that will meet the level of ‘proof’ you need to demonstrate your impact.

  1. Would some or all of the change have occurred independently, without your involvement?
  2. Could beneficiaries have achieved exactly the same outcomes elsewhere?

If the answer to these questions is yes, your deadweight will be high. If the answer is no, the chances are that your deadweight will be lower.

If you have the resources available, and if you require a higher standard of evidence around the amount of change that would have happened independently of your work (for example, if you are measuring your impact in order to influence policy change or to see if the way you work could be replicated by other people), then you will need to compare your outcome data against data from a similar group who did not receive goods or services from your organisation.

This can be done in two ways: firstly, by using existing research, where available, about the outcomes that occur for similar groups of beneficiaries without the type of support that your organisation provides; and secondly, by collecting your own data from a control group (a similar group who did not receive support from your organisation). These methods require more time and resources, but will give you a higher standard of evidence to take account of what would have happened anyway.

This criterion is fully met if:

You are able to describe the extent to which positive outcomes might have occurred independently of your work.

What next?

If you’ve met this criterion in full, you could improve your practice by…

  • Comparing your outcome data with existing research looking at outcomes achieved independently by similar groups

Comparing your outcome data with existing research, where available, on the outcomes that similar groups achieve independently of the type of support your organisation offers will give you a more informed sense of what would happen if your organisation did not exist.

For example, if you provided support that aimed to reduce levels of long-term unemployment, you could look at local data to see what proportion of local people found work after six months of unemployment, and compare this with data on the proportion of your beneficiaries who found work after the same period. The difference between the two figures would show you how much change is likely to have occurred without your intervention.

  • Comparing your outcome data with a control group

Collecting data from a control group will help you to establish a more accurate picture about how much change would occur without your work, and to gather a higher quality of evidence to demonstrate how much change would have occurred independently of your work.

3.8 We can describe where outcomes from our work overlap with costly economic, social and environmental issues (for example, climate change or unemployment)

Making a financial case for your work means highlighting the connection between the outcomes you achieve for groups of beneficiaries, and your organisation’s contribution to ‘costly’ societal problems, for example unemployment, environmental damage, and offending. Presenting your outcome data in this way will help you to demonstrate how your work could save money for taxpayers, commissioners, funders and the government. This can be a powerful tool for explaining the value of your work to partners, funders, commissioners, and the public.

Economic evaluation is not a new idea and there are many different approaches to choose from, some of which are more complex and time-consuming than others. For the majority of organisations, simply considering potential cost savings in your analysis and describing how and where your work impacts on costly economic, social and environmental issues – rather than coming up with a specific number for costs saved or avoided – will add depth and value to the way you report your findings.

For example, if the outcomes from work include reduction in landfill waste, you could describe the cost of landfill and highlight your organisation’s contribution to reducing this cost. Equally, if your work results in reduced reoffending or a reduction in serious offences, your analysis could include a summary of the costs involved in court proceedings and custodial prison sentences, and a description of how your work can result in some of these costs being avoided.

However, if your funder or commissioner requires you to demonstrate your impact in financial terms, or if you want to look in greater detail at the potential savings that result from your work, you will need to consider using a more sophisticated methodology to describe the financial value of your work – for example, cost effectiveness analysis or Social Return on Investment (SROI).

This criterion is fully met if:

You are able to describe how your work overlaps with costly economic, social and environmental issues, and where the outcomes you achieve for beneficiaries could potentially contribute to cost savings.

What next?

If you’ve met this criterion in full, you could improve your practice by…

  • Relating your outcomes against the cost of delivery to demonstrate your cost effectiveness.

Relating the outcomes you achieve for beneficiaries to the cost of delivery (your input) will give you information about how much it costs to deliver each outcome. This is also known as your cost effectiveness. Comparing your cost effectiveness with other organisations is a useful way of highlighting any potential savings for funders and commissioners.

  • Describing the overall economic, social and environmental value of your work in financial terms by using monetisation.

Monetisation involves setting a proxy financial value for each outcome achieved – whether economic, environmental or social – in order to be able to describe your overall impact in financial terms. This methodology is used in SROI.

Resources for this section

Storing data respectfully and legally

The Information Commissioner’s Office (ICO) is the UK’s independent authority set up to uphold information rights in the public interest, promoting openness by public bodies and data privacy for individuals. The ICO website contains useful guidance for organisations about how to keep personal data in line with the Data Protection Act.

Using IT systems to input and analyse data quickly and easily

Performance Hub, Using ICT to improve your monitoring and evaluation (2008)

This workbook covers the steps and issues that you will need to consider to develop appropriate computer systems that help you with impact measurement.
The workbook includes a free IT development checklist.

NCVO, A guide to managing ICT in the Voluntary and Community Sector (2007)

This guide covers the main issues that VCOs face in managing ICT, from policies and procedures to keep things running; from how to produce an ICT strategy to putting realistic costs in your funding bids.

Describing where outcomes from our work overlap with costly economic, social and environmental issues

Thinking through the other factors that might have influenced outcomes

Thinking through how much change might have occurred without our work If you are interested in developing your practice further in this area, you could refer to the Cabinet Office guide to SROI (2012) which provides a detailed overview of this methodology.

The guide also provides useful guidance about thinking through the other factors that might have influenced outcomes (attribution) and the changes that could have occurred without your work (deadweight).

Useful websites

NCVO have a section on their website dedicated to effective ICT for third sector organisations. This includes information on budgeting for and buying a system, training and support, and planning for IT. There are also a number of resources to download such as ICT publications, research, and blogs.

CYFERnet is an online, free, interactive evaluation resource. Users can access brief and informative ‘learning modules’ through the site on a range of issues, including:

  • Handling quantitative data (numbers and statistics)
  • Handling qualitative data (narrative and stories)

There are sample questions to test knowledge, video links to further explanations by evaluation experts and many other useful tools.

Sign-up to the principles

We need your help to put principles into practice