Medium to large voluntary organisations guidance - Plan

Plan

Careful, realistic planning lays the foundation for good impact practice. Accordingly, ‘Plan’ is the longest section of Measuring Up, and the questions this section asks you to consider are relevant to all organisations, whatever their size, shape or ambition in terms of being able to plan, evidence, understand, communicate and learn from their impact.

Planning requires you to reflect on how and why you think your work can make a difference. It is also an excellent first step for involving others in thinking about your impact – everyone can get involved in discussing and defining your organisation’s purpose, outcomes and impact.

Your goals for impact measurement will also determine the depth and detail of information that you set out to collect, as well as your plans for collecting it. Organisations that need information to remain accountable to funders and commissioners and to learn more about how to improve their work will require good quality evidence. However, the standard of evidence required for organisations who want to influence policy or to explore the efficacy of a new type of programme or initiative will be much higher. We have explored this issue of proportionality throughout the guidance provided with this section.

1.1 We use local information to provide evidence of the need for our work

A needs assessment sets the scene for the work you do. It will identify the extent and seriousness of existing problems, any services currently available and the needs that are not being met. Once you have clarified the need for your organisation, you can start defining how you plan to make a difference, laying the foundations for good impact practice.

If your organisation has been running for some time, the chances are that you will have conducted a needs assessment in the past. If you are a new organisation, however, you will have to begin from scratch.
To begin with, look for local data and any relevant national statistics and reports to get information on:

  • The area or region your organisation will operate in
  • The state of current service provision
  • The type of person you want to help, and where they are
  • The sort of problems they face, and possible solutions.

This criterion is fully met if:

You can describe the need for your organisation, and evidence that need using available information.

What next?

If you’ve met this criterion in full, you could improve your practice by:

  • Collecting your own information to use alongside existing data
  • Collecting your own information will add depth to your needs assessment and could make it more relevant to your specific context. You could gather this information by speaking to current service users, potential service users, community groups, and other organisations delivering similar work.
  • Working proactively to target ‘harder to reach’ groups

Harder to reach groups may be the very people you want to work with, and so it is important that you consider how to include their views in your needs assessment. You may need to think through people’s accessibility requirements, or provide translation services.

1.2 We can describe who will benefit from our work.

Being able to describe your target group – the main group or groups in society that you work with – is an important step in defining your purpose as an organisation. This means that it’s also an important step in focusing on your impact.

Your target group could be identified by geographical area, age, gender, sexual orientation, ethnicity, or by health status or disability. You may also want to identify socio-economic groups, such as single parents, children excluded from school or homeless people.

You can use the information gathered through your needs assessment to help you to define your target group or groups.

This criterion is fully met if:

You can describe in detail the group or groups that will benefit from your work.

What next?

If you’ve met this criterion in full, you can improve your practice by:

  • Describing different sub-groups within your target group.
  • Unpicking the different groups hidden within your target group will give you a more detailed picture of the different types of people who might benefit from your work.
  • Separating out end, first-line, and indirect beneficiaries of your work
  • Your end beneficiaries are the groups who directly benefit from your work.

Your first-line users are the people who access your services directly, but may not be your actual target group – for example, if your organisation ran parenting workshops to reduce childhood obesity, your first-line users would be parents or carers, but your end beneficiaries would be the children themselves.

Other groups might benefit indirectly from your work, for example other agencies which refer people to you. Breaking down your target group in this way will help you to build up a fuller picture of the types of people who might benefit from your work.

1.3 We have a clear mission statement, setting out our purpose and values.

Your mission statement outlines your organisation’s overall purpose, as well as the core values underpinning your work. Setting out the overall purpose of your organisation will help you to define the biggest, most broad-reaching change that you would like to see as a result of your work. This is also known as your impact – the broad or longer-term effects of your project or organisation’s work. This can include effects on people who are direct users of a project or organisation’s work, effects on those who are not direct users, or effects on a wider field such as government policy.

Defining your organisation’s overall purpose and the changes that you would like to create begins the discussion about how you will measure the difference your work makes. Involving staff, volunteers, trustees, partners and beneficiaries at this stage will add depth and detail to your planning processes, and will help people to feel involved and engaged.

This criterion is fully met if:

Your organisation is guided by a document which clearly sets out your overall purpose as an organisation, and the core values that underpin your work.

What next?

If you’ve met this criterion in full, you can improve your practice by:

  • Making sure that your mission statement and core values are agreed on and recognised by the whole organisation

Involving people in putting your mission statement together, and making sure that the end document is recognisable to everyone within your organisation (including staff, trustees, and volunteers) will make it a more powerful guiding document.

  • Making sure that your mission statement and core values are written into all of your key documents, and that people are clear about how they influence your day-to-day work

Writing your core values and purpose into your key documents will help to embed your mission statement into the way you think about and approach your work. This will help to make sure that your values and ethos are put into practice, bringing your mission statement to life.

1.4 We can describe the positive outcomes that we want to achieve for beneficiaries

Your outcomes are the changes, benefits, learning or other effects that happen as a result of the work you deliver. Your work as an organisation will potentially produce any number of outcomes for different groups, both positive and negative, planned or unexpected.

Outcomes are not the same as outputs, which refer to the activities, services and products provided by an organisation. Your outcomes describe the difference that your work makes, and not the work itself.

For example:

A youth group runs sexual health workshops with local young people. The output is the workshop – the service that they deliver. The outcome is what changes for the young people as a result of participating – an increased understanding of how to protect themselves from sexually transmitted diseases.

This criterion is fully met if:

You have clearly defined positive outcomes that you hope to achieve for different groups.

What next?

If you’ve met this criterion in full, you can improve your practice by:

  • Describing the order in which outcomes happen for beneficiaries

Once you have defined your outcomes for beneficiaries, you may decide to map out your intermediate outcomes – the small changes that occur for beneficiaries in the lead up to a bigger change or an end outcome. For example, an organisation working to support people using alcohol might have abstinence from alcohol as an end outcome, with reducing the amount of alcohol consumed, and increasing the length of time without drinking as intermediate outcomes. Mapping your intermediate outcomes in this way will help you to develop a clearer description of the order in which changes happen for beneficiaries.

  • Developing your outcomes in consultation with potential beneficiaries

Consulting with potential beneficiaries, where possible, will help you to develop an in-depth picture of the sort of outcomes that could result from your work, and how beneficiaries might describe and experience these changes. This will help you to define outcomes that are relevant to your beneficiaries and realistic, given the work you do.

1.5 We can describe how and why our work makes a difference to beneficiaries (our ‘theory of change’)

By describing the link between the work that you do (your outputs) and the changes that it creates for individuals (your outcomes) and for society more widely (your impact), you are setting out how and why your organisation plans to make a difference. This is also known as developing your theory of change – creating a story or narrative about the changes that you want to create through your work.

As well as spelling out how you think your work creates changes for beneficiaries, your theory of change document will guide your decisions about what information you might need to collect in order to evidence the extent to which this narrative is correct.

However you choose to present your theory of change, it should be:

Credible –based on previous experience and insight from different stakeholders, or from research.
Achievable – do you have the necessary resources to make the differences you describe?
Testable – your theory of change document will need to be linked in to your plans for collecting evidence.
Supported – have you involved all the right stakeholders in the process, and do they all agree with the way your organisation is described?

This criterion is fully met if:

You have a document which describes or maps out the connection between the work that you do and the difference that you plan to make for your beneficiaries and for society more widely. This document should be credible, achievable, supported, and linked in to your plans for collecting evidence.

What next?

If you’ve met this criterion in full, you can improve your practice by…

  • Including details of the timeframe in which you expect changes to happen.
  • Mapping timescales for the way in which beneficiaries access your services and experience outcomes will give you a more detailed understanding of when and how your work makes a difference. It can also help you to understand the ‘journey’ that beneficiaries experience when using your services.
  • Including detail of the resources you will need and the people you will need to work with at each stage

Including information about the resources you will need (your inputs) and the partners that you will need to collaborate with for your work to be successful will help you to produce a theory of change document that is realistic and a useful tool for strategic planning.

1.6 We can describe how the outcomes we want to achieve overlap with the difference our partner organisations want to make

Organisations do not operate in a vacuum, and the work of other agencies and services will have an effect on the impact that you achieve. For example, two organisations campaigning for policy reform at the same time, in the same area, are likely to overlap, and local services working with vulnerable people are likely to support some of the same people simultaneously.

If there are other organisations doing similar work to you, you will need to consider the role that they play in your success. This might involve discussing the different outcomes that partner agencies achieve for the same beneficiaries, or thinking about the extent to which partner agencies contribute to delivering shared outcomes – outcomes that both you and your partner agencies would like to achieve through your work.

This criterion is fully met if:

You share an understanding with partner organisations about the way in which your organisation’s outcomes contribute to, or overlap, with the impact that they want to achieve.

What next?

If you have met this criterion in full, you can improve your practice by:

  • Developing agreements and systems that allow you to use third party information from partner agencies.
  • Sharing information that you have collected separately will help you to understand the extent to which your work overlaps. This will give you a more informed understanding of the role of different. organisations in achieving specific outcomes.
  • Working together with partner agencies to develop shared measurement frameworks or plans to measure change over the longer term.

By working in collaboration with partner agencies to develop a plan for impact measurement, you will be able to work as a team to gather information that evidences the role of each agency in achieving outcomes. Working as a team can also help you to measure the more broad-reaching, longer-term changes that come out of your work together – helping you to evidence your collective impact.

1.7 We are clear about the information that we need most urgently, both for reporting to funders and for internal learning

Before you begin collecting evidence, it is important to be clear about which information you need in order to be able to meet your reporting requirements externally to funders and commissioners, and internally to your trustees or board. It will also help you to make sure that you have all the information that you need internally to learn and improve your service. Clarifying this at the planning stage will help you to avoid collecting information that you don’t need, or missing out something vital.

It will also help you to set priorities about the information you most need to collect – this is fundamentally important for building a plan for collecting evidence that matches your available resources. It may not be realistic to measure everything! Focusing on measuring a prioritised number of things well, rather than trying to measure everything, is often a more straightforward and robust approach.

This criterion is fully met if:

You have defined which information you need to in order to be able to report to funders and commissioners, and to your trustees or board.

What next?

If you have met this criterion in full, you can improve your practice by:

  • Regularly reviewing what information you need to learn from your work and make management decisions.

As your work changes and develops over time, you may decide that you need different information in order to learn about and improve your impact. Regularly reviewing your information needs will make sure that you have the right information to help you continuously improve what you do.

  • Thinking ahead about the information you might need to make future bids for service and organisational development.
  • Thinking strategically about the information you might need in the future to develop your work will help you to make informed decisions about the way you grow and fund your organisation.

1.8 We know what information to collect to show which goods and services have been delivered, and to whom

This section covers the way in which your organisation sets indicators – well-defined, easily measurable information, which shows how well your organisation is performing. You will need to set indicators that tell you about your outputs (as well as your outcomes, see 1.9).

Once you have identified your outputs and prioritised those you want to measure (see 1.7), you will need to identify output indicators – pieces of information that will help you to work out whether or not you delivered your work in the way you planned.

Output indicators are usually quantitative – that is, they collect numbers and statistics. For example, if you provided a helpline, your output indicators might be:

  • Number of calls received
  • Number of individual callers
  • Length of calls

You would also want to collect some qualitative information as well – that is, more descriptive information. For example, the helpline could collect information about the types of issues that were raised by callers.
You will also need to set indicators for the type of people accessing your services (for example, ethnic group, sexuality, gender, and age). This will help you to understand whether or not you are reaching your target groups, and whether or not your services are fully accessible.

This criterion is fully met if:

You have set output indicators for a prioritised number of outputs, that include information about the goods and services that you delivered through your work, and the type of people who accessed them.

What next?

If you have met this criterion in full, you can improve your practice by:

  • Setting indicators that measure client satisfaction for all of your outputs.

Collecting information about the level of client satisfaction will help you to understand how people experience the services that you provide, and how you might be able to improve the way you work. (Note that client satisfaction is an output, rather than an outcome indicator, since it collects information on how people felt about the service they received, rather than what changed for them as a result.)

  • Setting indicators that tell you about the way in which services were delivered (for example, how often services run, how long they are provided for, and who drops out).

Collecting information on the way in which services were delivered – rather than simply whether or not they were delivered – will give you more detailed information about whether or not your service is running in the way you planned. For example, it would be useful for the helpline to collect information about how often the same callers phoned the helpline, since this adds detail to our understanding of how people are using the service being offered.

1.9 We know what information to collect to provide evidence of the changes that beneficiaries experience as a result of our work

Once you have identified your outcomes and prioritised those you want to measure, you will need to identify outcome indicators – pieces of information that will help you to work out whether or not your work made the difference that you planned to achieve.

Outcome indicators are the signs that the outcome has happened, or that progress is being made towards it. They can be quantitative (measuring the number of changes that happened) and qualitative (describing people’s perceptions and experiences).

You are likely to find a number of possible indicators with each outcome. As with outcomes themselves, identify and use only the most relevant ones, to keep your evidence collection proportionate and streamlined.
To provide good evidence of change, you will need to select more than one indicator for each outcome. For example, if you wanted to measure the extent to which your networking event had resulted in the outcome of ‘increased partnership working’, you indicators might be:

  •  Number of new collaborations or partnerships

This would give you information about how many new partnerships had occurred.
If you want information about the nature and depth of those new partnerships, a relevant indicator might be:

  • Level of information sharing

This criterion is fully met if:

You have selected a number of priority outcomes and set at least two outcome indicators for each. Your outcome indicators can provide you with the information you need to assess whether and how your organisation has achieved the outcomes that it planned.

What next?

If you have met this criterion in full, you can improve your practice by:

  • Making sure that you have a good mix of qualitative and quantitative outcome indicators that include different people’s perspectives on the difference your work made.

Selecting a range of qualitative and quantitative indicators will give you better quality information about which outcomes were achieved, and how people perceived and experienced those outcomes. Collecting different perspectives on change allows you to compare different people’s accounts of the difference your work makes, and will make your evidence for change more detailed and robust.

  •  Developing your output indicators in collaboration with key stakeholders and beneficiaries.

Asking beneficiaries about the way that they experienced outcomes from your project will give you useful details that you can turn into relevant indicators capable of collecting good quality evidence of change. For example, if your work aims to reduce isolation for older people, and the people using your project tell you that the difference your work made was that they socialised more often, this suggests that ‘how often people meet up with friends’ would be a meaningful outcome indicator.

1.10 We choose data collection tools that meet our information needs and suit our context

The type of tools and methods you use to collect data should be in proportion to your information needs. If you are testing a new type of project, or you want to gather evidence that is robust enough to influence policy change, you may decide to use a validated tool – that is, one that has been developed and tested by academics and impact measurement specialists. However, if you are collecting straightforward information to report back to your funders and trustees, you may decide to create your own data collection tool that suits your purposes, or to adapt one that already exists.

Once you have prioritised the outputs and outcomes that you want to collect information on, and set your indicators, the next planning stage is selecting data collection tools – for example, questionnaires, focus groups and interviews.

In order to collect good quality evidence, your data collection tool needs to be appropriate to your context. This involves thinking through three key considerations.

1. What sort of information do I need?

If you want to collect basic quantitative data (numbers and statistics), you may find that a simple questionnaire suits your data collection needs. However, if you want to collect in-depth information about how people experienced different outcomes and what they thought about them, you will need to choose more ‘open’ data collection tools, such as interviews and focus groups.

2. Which tools would work best in the context of my work?

Not all data collection tools can be applied equally successfully in different contexts. A questionnaire designed for older people might work well in context, but very poorly in youth work setting if young people find it dull to complete or difficult to understand. Before choosing a data collection tool, you will need to think about your particular context, and how you can make data collection easy and interesting for the people you want to collect information from.

3. Which tools will give me data that I can easily collect, store and analyse?

Think carefully about the skills level and time available to the people who will be collecting the data. Your tool will need to work for your staff and volunteers, as well as for your beneficiaries. Thinking ahead to storage and analysis is also sensible. Before choosing a tool which gives you a large amount of very detailed information, ask yourself whether you have the necessary IT to store it safely, and the skills to make sense of it during analysis.

This criterion is fully met if:

You have selected data collection tools that can capture all of the information you need, which make data collection easy for your beneficiaries, and which suit the time and skills level of those people responsible for collecting and analysing the data.

What next?

If you have met this criterion in full, you could improve your practice by:

  • Consulting your staff, volunteers and beneficiaries about data collection tools.

Asking the people who you want to collect data from about which tools they think work best will help you to select data collection methods that are tailored to your beneficiaries’ needs – which, in turn, will give you better quality evidence. Talking to the people who will be collecting the data about data collection tools will give you a more informed picture of which tool will fit most easily into their day-to-day work.

  • Including a validated tool in your range of data collection tools where appropriate

Introducing a validated tool into your plan for data collection can sometimes improve the standard of evidence that you collect. If many other organisations are using the same validated tool, it may also allow you to compare your outcomes against other people who are doing the same sort of work (benchmarking). However, validated tools only collect good quality data if they are used in a relevant and appropriate setting and in the right way, so you will still need to think carefully about whether or not they are right for your particular context.

1.11 We have a plan which sets out how and when evidence will be collected, and by whom

Having decided on your priorities for collecting evidence and selected your tools, you are now ready to draw up your framework. Your framework clarifies which outputs and outcomes are going to be measured, with what tool, when and by whom. Essentially a road map for collecting evidence, your framework is a crucial tool for making sure that nothing important gets left out of your data collection plan, and that everyone understands their particular role in collecting information.

Sample evaluation framework:

  • Outcomes Indicators Information collection tools
  • Who and when
  • Improved job search skills
  • Numbers with a CV
  • Ability to identify suitable jobs
  • Ability to complete application form
  • Appropriateness of presentation Job log database
  • Case file
  • Self-assessment form
  • Staff observation grid Case worker
  • At assessment/ review sessions

This criterion is fully met if:

You have a framework or mapping document which sets out: what information you want to capture on outputs and outcomes; how this data will be collected; when it will be collected; and who is responsible for collecting it.

What next?

If you have met this criterion in full, you can further improve your practice by:

  • Including a description of how the information will be used

Adding this information to your framework will provide an additional check to make sure that you aren’t collecting anything unnecessary, and that all of your priority outputs and outcomes are included.

  • Linking your framework to your strategic planning processes by including targets

Including targets for your outputs and outcomes will turn your framework into a useful tool for strategic planning as well.

1.12 We have realistic targets that set out what we want to achieve

Your plan for evidence collection should be informed by your understanding of the information you need for strategic planning – that it, the data your trustees require to understand whether or not your organisation is working as effectively as planned.

Setting realistic targets for what you want to achieve as an organisation is a good way of building this link between gathering evidence and strategic planning. Good targets will be time specific, and will clearly set out what success looks like. However, they should not add up to more than your capacity. When thinking about setting reasonable targets, consider:

  • Your resources
  • What you have achieved before
  • What other, similar organisations have achieved
  • The level of need for your work (as evidenced by your needs assessment)

This criterion is fully met if:

You have set realistic targets for your priority outputs and outcomes, that clearly describe what your organisation plans to achieve.

What next?

If you have met this criterion in full, you can further improve your practice by:

  • Setting targets around the quality of the work delivered, as well as the quantity

Including client satisfaction targets will help you to paint a clear picture of what success should look like for your organisation in this area.

  • Consulting with your trustees, staff, and volunteers about targets

Talking to the people responsible for delivering the work and meeting targets should help you to set targets that are realistic and motivating.

1.13 We look at the resources available for focusing on our impact and identify any gaps

Good impact practice requires a number of different resources – including staff time, IT, skills development, and money, (especially if you decide that you need external support from a consultant or specialist). Committing the right amount of resources to impact practice is crucial to doing it well. However, impact measurement should not take up a disproportionate amount of time and money.

Assessing your available resources will help you to build a realistic plan for collecting and making sense of evidence to show your impact. It will also give you a more informed picture of the costs involved, which will help you to come to an agreement with your funders and commissioners about the amount of information that it is reasonable to expect you to collect.

In the longer term, understanding where the resource gaps are will help you to develop a plan for meeting these gaps, and to make a better case for adequately resourcing impact measurement within your organisation.

This criterion is fully met if:

You have a clear picture of the resources available, including IT, money, staff time, and skills. You are able to identify where the resource gaps are and to explain how these could be met.

What next?

If you have met this criterion in full, you can further develop your practice by:

  • Building a clear plan for how to develop your resources in the short term.

Once you have identified the gaps in your resources, you are in a better position to think through how best to meet these in the short term, for example through staff training. This will allow you to make some relatively quick improvements that will increase your capacity to measure your impact.

  • Building a clear plan for how to develop your resources in the long term

Building a longer-term plan will give you more of a strategic view on how your organisation could manage resources for impact measurement in the future – for example, by acquiring a new database or hiring a consultant to help with a specific area or task. This can help to make a case for developing your capacity in this area.

Resources for this section

Using local information to provide evidence of the need for your work

Neighbourhood Statistics is an online government portal that allows you to search 2011 census data. Enter your postcode to find statistics on demographic profile, crime rate, education, health, housing, deprivation, lifestyles, work and environment.

Knowing which information to collect to evidence that goods and services have been delivered/ to evidence the changes that beneficiaries experience

Charities Evaluation Services, Keeping on Track: A guide to setting and using indicators (2008)

This booklet provides a step-by-step guide to setting and using indicators.
The guidance is illustrated by good-practice case studies, and takes account of what funders and commissioners regard as good quality indicators. It also offers practical guidance on how to set and use indicators that will help you to monitor your work effectively.

Mapping the description of how and why your work makes a difference (your theory of change)

Charities Evaluation Services, Making Connections: Using a Theory of Change to Develop Planning and Evaluation (2011)

The document provides an overview of the theory of change approach and explains more about how the approach is used. It goes on to provide a greater detail through the several steps of developing a theory of change and discuss what you will need to include to make it most useful.
The report is free to download from the CES website.

New Philanthropy Capital, Theory of Change (2011)

This short paper introduces theory of change, explains the origins of the technique, and discusses how it can be used by charities to improve their work.

Developing a plan

Sample evaluation frameworks can be downloaded from the CES website.

 

Sign-up to the principles

We need your help to put principles into practice