The Qualitative Impact Assessment Protocol
The QuIP addresses the issue of how to attribute changes to different stakeholders or events, whilst minimising pro-project and other sources of bias, and avoiding the need to interview a control group. It is designed to work alongside existing quantitative monitoring of key indicators, adding qualitative, self-reported attribution of impact to provide sufficient evidence to test the theory of change behind the activity being evaluated.
Put simply, the QuIP provides a straightforward and cost-effective mechanism to ask intended beneficiaries about significant drivers of change in their lives, and to analyse and present the data collected.
There are strong ethical grounds for asking people directly about the effect of actions intended to benefit them, and doing so can also contribute practically to detailed learning, innovation and wider accountability within your organisation. However, this approach entails finding credible ways to address potential response biases. The QuIP does so by arranging for qualitative data collection to take place without any reference to the project being evaluated, ensuring that field researchers and respondents are not briefed on the project being evaluated and thereby reducing confirmation bias as much as possible up to the point of data analysis. The analysis is then carried out by a separate party, who is fully briefed and can therefore interrogate and code the data against the theory of change. The aim of separating these roles is to ensure that the analysis remains as independent as possible.
The second key problem with recording detailed feedback from beneficiaries is what to do with all the data? Qualitative information is difficult to process, analyse and condense into a credible and transparent report. We have developed a simple and semi-automated approach to coding QuIP data which allows for easy analysis and reporting, ensuring that reports are brief, readable and that the frequency of findings cited and the source data are completely transparent. The coding system is detailed in the briefing paper and full guidelines.
By creating a systematic approach, we have speeded up the time required to undertake the assessment thereby reducing the costs considerably. Whilst it isn’t possible to fix a price on a QuIP as every country has different costs, it is safe to say that it will cost a fraction of the cost of randomised controlled trial, or any assessment requiring a control group. The following timings may help calculate approximate costs:
1 week of design and preparation
2 weeks of 2 field researchers’ time, both in the field and writing up
1 week of analysis and writing up
Training and software to support analysis and ensure the tools remains robust
- Diageo (CSR)
- Acumen (Social impact investors)
- Self Help Africa
- Farm Africa