1. Impact assessment in international development

Most of the literature analysing the question of the impact of aid programmes is based on international development aid, rather than humanitarian aid. The documents available in international development therefore form an important and useful body of reference for humanitarian practitioners.

The most comprehensive study of the question of impact is Chris Roche’s Impact Assessment for Development Agencies, conducted in 1999 for Oxfam UK and Novib. David Hulme also gives a useful account of different impact assessment methods in the context of microfinance. Hulme’s work raises wider questions regarding impact assessment, and can be extrapolated beyond the particulars of microfinance. INTRAC also gives a useful account of the issues related to impact assessment. The OECD/DAC provides the most comprehensive and widely-agreed list of definitions in relation to impact.

• Chris Roche (1999) Impact Assessment for Development Agencies. Oxford: Oxfam/Novib.
• Hulme, D. 1997, Impact Assessment Methodologies for Microfinance: A Review (click here).
• Kirkpatrick, C. et al. (2001) Basic Impact Assessment at Project Level (click here).
• Simanovitz, A. (2001) Virtual Meeting on Impact Assessment Methodologies – Background Paper (click here).
• INTRAC (2001), NGOs and Impact Assessment (click here).
• OECD/DAC (2002) Glossary of Terms Used in Evaluations (click here).

Other key documents are:

• White, H. (2002) A Drop in the Ocean? The International Development Targets as a Basis for Performance Measurement (click here). This is an annex to the UK National Audit Office (NAO) report ‘Department for International Development. Performance Management – Helping to Reduce World Poverty.
• OECD/DAC (1997) Searching for Impact and Methods: NGO Evaluation Synthesis Study (click here).
• Koponen, J. and Mattila-Wiro, P. (1996) Effects or Impacts? Synthesis Study on Evaluations and Reviews 1988 to Mid 1995, Ministry for Foreign Affairs, Department for International Development, Finland (click here).Lund Madsen, H. (1999) Impact Assessments Undertaken by Danish NGOs, Centre for Development Research, Copenhagen (click here).
  
  

2. Questions of impact in context: accountability and results-based management

The humanitarian system’s increasing interest in impact needs to be understood in the context of broader debates about accountability for humanitarian aid, and in the context of public management reforms within Western governments.

There is a vast literature on issues of accountability, not cited here. For general debates, see for Humanitarian Exchange, no. 24, July 2003 (click here), or Forced Migration Review, no. 8, August 2000 (click here). ALNAP’s Annual Reviews provide useful information on accountability, learning and performance in the humanitarian sphere (ALNAP website). With regard to the particular question of impact, a number of agencies have set up impact assessment systems in order to improve their accountability at the organisational level. For a discussion of some of these mechanisms, see a report from the British Agencies Aid Group (see below). The 2003 INTRAC Evaluation Conference was on the theme of Management, Measurement and Accountability. All the papers are available at the INTRAC website. ECHO commissioned a study that compares the different quality management tools used by humanitarian agencies, most of them aiming at increasing the accountability and performance of their work.

• ECHO (2002) Report on the Analysis of ‘Quality Management’ Tools in the Humanitarian Sector and Their Application by NGOs (click here).
• British Agencies Aid Group (2003) Towards Organisational Performance Assessment: Experiences of Strengthening Learning, Accountability and Understanding Social Change (click here).

Increasing interest in the question of impact is largely due to changes in public management in some Western governments: the recent shift from input–output management to so-called results-based management, adopted by several governments, the EU and an increasing number of aid agencies puts stronger pressure on agencies to demonstrate results. The OECD/DAC paper on results-based management provides an overview of the position and systems of a number of agencies, and how their performance measurement system works at project, country and agency levels. A number of agencies and donors have developed their own guidelines for implementing results-based management. Some examples are:

• CIDA (1996) Results-Based Management in CIDA – Policy Statement Prepared by Results-Based Management Division, Performance Review Branch (click here).
• CIDA (2000) RBM handbook on developing result chains (click here).
• OECD/DAC (2000) Results Based Management in the Development Cooperation Agencies: A Review of Experience (click here).
• WFP (2003) Information Note on Results-Based Management (click here).
• WFP (2003) Results-Based Management in WFP: Next Steps (click here).

The following authors provide a general account of the origins of results-based management, highlight some of its limits and explore some of its implications for aid agencies.

• Hailey, J. and M. Sorgenfrei (2003) Measuring Success? Issues in Performance Management, Paper presented at INTRAC’s Fifth Conference (click here).
• Wallace, J and T. Chapman (2003) Some Realities Behind the Rhetoric of Downward Accountability, Paper presented at INTRAC’s Fifth Conference (click here).
• Earle, L. (2003) Lost in the Matrix: The Logframe and the Local Picture, Paper presented at INTRAC’s Fifth Conference (click here).
• Chapman, J. (2003) System Failure: Why Governments Must Learn to Think Differently. London: Demos.
• Onora O’Neill (2002) Called to Account, BBC Reith Lectures, 2002 (click here).
  

3. Tools and methods for measuring impact

Some development agencies have developed their own guidelines for impact assessment, and these can be applied to humanitarian contexts. The various guidelines developed by humanitarian agencies for monitoring and evaluation or needs assessments can also be used to design impact assessments. A comprehensive review of needs assessment practice can be found in Darcy, J. and C.-A. Hofmann, According to Need? Needs Assessment and Decision-Making in the Humanitarian Sector (HPG Report 15, September 2003). Save the Children’s Toolkit provides a useful overview of the different approaches. UNHCR’s Evaluation and Policy Evaluation Unit (EPAU) provides useful information on real-time evaluations. Alistair Hallam’s Evaluating Humanitarian Assistance Programmes in Complex Emergencies (HPN Good Practice Review 7, 1998) has a section on impact assessment.

• Save the Children UK (2004) Global Impact Monitoring Guidelines 2004 (click here).
• CARE USA (1999) Impact Guidelines (click here).
• Darcy, J. and C.-A. Hofmann (2003) According to Need? Needs Assessment and Decision-Making in the Humanitarian Sector, HPG Report 15. London: ODI (click here).
• Hallam, A. (1998) Evaluating Humanitarian Assistance Programmes in Complex Emergencies. Good Practice Review 7. London: ODI (click here).
• UNHCR (2002) Project Planning in UNHCR (click here). Other UNHCR guidelines are available at the UNHCR eCentre.
• USAID (1999) Field Operations Guide for Disasters Assessment and Response (click here).
• UNHCR (2000) Handbook for Emergencies, 2nd Edition (click here).
• OECD/DAC (1999) Guidance for Evaluating Humanitarian Assistance in Complex Emergencies (click here).
• OECD/DAC (2002) Glossary of Terms Used in Evaluations (click here).
• ECHO (2002) Evaluation of Humanitarian Action Funded by the Humanitarian Aid Office of the European Commission: A Guide (click here).
• ODI (1996) The Joint Evaluation of Emergency Assistance to Rwanda (click here).
• WaterAid (2002) Looking Back: Participatory Impact Assessment (click here).
• Humanitarian Initiatives, Disaster Mitigation Institute and Mango (2001) Independent Evaluation: The DEC Response to the Earthquake in Gujurat (click here).
• UNHCR (2001) The WHALE: Wisdom we Have Acquired from the Liberia Experience. Report of a regional lessons-learned workshop, Monrovia, Liberia, 26–27 April (click here).

The use of indicators is a crucial element in determining the impact of an intervention. The following documents provide clarification about which types of indicators ought to be used in humanitarian assistance. The SMART initiative is advocating for the systematic use of crude mortality and under-five malnutrition indicators.

• SMART Improving the Monitoring, Reporting and Evaluation of Humanitarian Assistance (click here).
• The Sphere Project (2004) Humanitarian Charter and Minimum Standards in Disaster Response, 2nd edition (click here).
• World Bank (1996) Performance Monitoring Indicators: A Handbook for Task Managers (click here).
• FANTA has developed a series of guides on the use of indicators for USAID Title II programmes (click here).

Some agencies have developed particular guidelines for evaluating the impact of advocacy activities:

• Mayoux, L. (2003) Advocacy for Poverty Eradication and Empowerment: Ways Forward for Advocacy Impact Assessment (click here).
• Lloyd Laney, M. (2003) Advocacy Impact Assessment Guidelines (click here).
• ActionAid (2001) Monitoring and Evaluating Advocacy: A Scoping Study (click here).

There are a number of methodological difficulties in assessing the impact of humanitarian aid. These include a lack of baseline data and difficulties in attribution. Below are some useful examples of how these obstacles can be overcome:

• Mayne, J. (1999) Addressing Attribution through Contribution Analysis: Using Performance Measures Sensibly (click here).
• Roberts et al. (2001) Keeping Clean Water Clean in a Malawi Refugee Camp: A Randomized Intervention Trial, Bulletin of the WHO, vol. 79 (click here).
• White, H. (2003) Challenges in Evaluating Development Effectiveness, World Bank, Operations Evaluation Department.

4. Impact assessment in humanitarian practice

On impact assessment in health programmes, see:

• Robertson, D. W. et al. (2002) What Kind of Evidence Do We Need to Justify Humanitarian Medical Aid?, The Lancet, vol. 360.
• Spiegel, P. et al. (2001) Developing Public Health Indicators in Complex Emergency Response, Prehospital and Disaster Medicine, vol. 16, issue 4 (click here).
• Griekspoor, A. et al. (2002) Enhancing Health in Complex Emergencies: The Broader Research Agenda (click here).

On impact assessment in food and nutrition programmes, see:

• Emergency nutrition network online, which provides useful information in relation to nutrition and food security.
• Save the Children UK (2000) The Household Economy Approach: A Resource Manual for Practitioners (click here).
• USAID (2003) Tools for Operationalizing Essential Nutrition Actions (click here).
• Maxwell, D. (2001), ‘The Coping Strategies Index: Monitoring Food Security Status in Emergencies’, Field Exchange 13.
  

5. Useful links

Other useful resources include:

Evaluation to Assess and Use Results.

• MEASURE: Monitoring and Evaluation to Assess and Use Results.
• MandE: Monitoring and Evaluation News.
• SMART: Standardized Monitoring and Assessment of Relief and Transitions.
• FANTA: Food and Nutrition Technical Assistance.
• The PARC: Performance Assessment Resource Centre.
• Imp-Act: Improving the Impact of Microfinance on Poverty: Action Research Programme
• EDIAIS: Entreprise Development Impact Assessment Information Service.