Catalogue of Methods for Impact Studies

Overview

This article includes a variety of methods and tools which can be chosen and combined; appraisals of individuals methods and tools to help in the selection of the most appropriate combinations of methods for the desired purpose. The use of a variety (triangulation) of sources and methods is advocated as the best way to obtain complete and reliable information on the issues under study.

Main Evaluation Instruments

Short Household Survey

General Approach:

Short and structured household questionnaire focused on impact-oriented information that can be easily elicited. Comparison of people provided with the modern energy service (stove or household connection) in the intervention area before and after the intervention. Households that have been included in the before survey but finally did not purchase a stove or connect to the grid may as well be included in the ex-post comparison. Complemented by open, semi-structured key informant interviews, which can be conducted in combination with different qualitative appraisal approaches (see below).

Purpose:

• Focused on the usage of the main services and opportunities provided by electricity or improved stoves. Indirect impacts on non-connected households and households without stoves cannot be verified.
• To deliver statements about impacts close to the project’s results that do not necessarily need an extensive survey set up and data analysis with control groups for causal attribution. Relation to poverty impacts is established theoretically by means of results chains. For example, information for electrified households on the average daily time primary school children study at home allows gauging impacts of electric lighting on studying of children. The link of these studying conditions to changes in school performance is, however, only of theoretical nature based on plausibility.

Profound Household Survey

General Approach:

Extensive and structured household questionnaire comprising information on socio-economic conditions, access to services, different wealth and income categories, health, education and gender issues. The survey covers the project region and a control region to credibly create a counterfactual situation and thereby enable a chance for causal attribution of impacts to the intervention. Depending on the information interests, two different approaches are applicable: Control regions may be non-electrified (Option A) or already disposing of electricity access (Option B). For stove projects, option A refers to regions without any former improved stove intervention and Option B to regions where comparable improved stoves interventions have already taken place. In both of the cases, surveys are conducted before and after the intervention. They are complemented by open, semi-structured key informant interviews, which can be conducted in combination with different qualitative appraisal approaches (see below). Furthermore, the wider set of socio-economic household characteristics that are covered by this method facilitates detecting unintended (positive or negative) impacts.

Purpose:

• To provide information that allows for deriving statements about the causal relationship between modern energy provision and development indicators using modern state-of-the-art evaluation techniques.
• Baseline survey to deliver profound knowledge about the target region.
• If Option B is chosen, information from the comparison group that already uses electricity or stoves respectively can be used to assess the beneficiary’s behaviour already before project implementation same as potential difficulties and opportunities to be expected in the wake of the EnDev project.

Qualitative Appraisals

General Approach:

The qualitative complement of the different survey methods to better appraise the changes induced by the intervention. Flexible in terms of the applied methods, be it focus group discussions, Participatory Rural Appraisal (PRA), Method for Impact Assessment of Programs and Projects (MAPP) or open questions, all of which are described in detail below. Although the simplicity of these methods and tools is part of their success it is vital to work with qualified and experienced facilitators in order to achieve viable results.

Purpose:

• To provide background information on the changes induced by the intervention.
• Including more advanced qualitative methods (namely PRA), it provides anecdotal evidence on poverty impacts and transmission channels, i.e. how project outcomes turn into impacts.
• Particularly helpful to elicit non-intended (positive and negative) impacts.

Enterprise Surveys

General Description:

In principle, short and profound enterprise surveys are the analogues of the respective household surveys for micro, small or medium enterprises. The main differences are the sampling procedure and the integrated indicators. This survey instrument does not apply to stove producers, for whom the relevant evaluation approaches are in general more case-specific (see below). Indications are given in the respective indicator table in the following chapter.

Social Infrastructure Surveys

General Description:

Semi-structured interviews with executive social infrastructure representatives, if the respective institutions are being targeted. These respondents are executive officers, priests, directors, or headmasters, but maybe in exceptional cases, other staff members considered equally well-informed. Can be conducted in combination with different qualitative appraisal approaches (see below).

Stove Producer Surveys

General Description:

The effort to provide for this survey corresponds to the household survey for stove users. It addresses the different types of people active in stove business, whether they are producers, installers, builders or dealers, working individually or in production groups, part time or full time. The survey aims to highlight the different aspects of sustainable stove production and the impacts being achieved.

Standardised Tests

General Approach:

Two standardised tests are part of the basic instruments for stove projects’ monitoring: the Kitchen Performance Test (KPT) and the Indoor Air Pollution Test (IAPT). The KTP is the principal field based procedure to demonstrate the effect of stove interventions on households’ fuel consumption. It allows testers to compare the rate of daily fuel consumption per person of traditional and improved stoves as they are used in a normal household environment over an extended period of time. It is conducted with the willing cooperation of individual families. The IAPT measures room pollution levels caused by smoke in the kitchen. Cookstove emissions contain a wide range of harmful pollutants, of which particulate matter (PM) and carbon monoxide (CO) are considered to be the most damaging pollutants to health. Today, portable instruments exist to carry out the measurement of these pollutants in households’ kitchen.

Purpose:

• KPT: to compare the impact of improved stoves on fuel consumption in the kitchen of real households.
• IAPT: to measure the level of indoor air pollution caused by stove emissions from cooking and heating with solid fuels.

Description of Qualitative Methods

Qualitative methods for data collection play an important role in impact evaluation by providing information useful to understand the processes behind observed results and assess changes in people’s perceptions of their well-being. Furthermore qualitative methods can be used to improve the quality of survey-based quantitative evaluations by helping generate evaluation hypothesis; strengthening the design of survey questionnaires and expanding or clarifying quantitative evaluation findings.

Focus Group Discussion

General Description:

In a focus group discussion, people from similar backgrounds or experiences (e.g., mothers, young married men, farmers) are brought together to discuss a specific topic of interest to the investigator. Homogeneous samples are preferred because mixing age/ gender groups may inhibit some people, especially women, from expressing their views.

Purpose:

• To explore the range of opinions/ views on a topic of interest.
• To collect a wide variety of local terms and expressions used to describe a topic (e.g. a lighting device, an informal insurance scheme, a stove type, or disease).
• To explore drivers of change and meanings of survey findings that cannot be explained statistically.

Key-informant Interviewing

General Description:

Key-informant interviewing is a standard evaluation method. The subject of such an interview may be very broad, such as health, or farming, or family structures in the locality; or it may be more specific - which water sources are best for which purposes, for example. Key-informant interviewing can thus provide valuable information on both specific practices and on the context in which they are assessed. The term key-informant may be used for anyone who can provide detailed information, on the basis of their special expertise or knowledge of a particular issue. The choice of key-informants is topic-specific. For example, a village leader could be helpful when discussing ongoing community participation projects. Women may be ideal key informants to discuss the situation concerning firewood collection or children’s health status, and so on. The investigator can as well simply raise a topic for conversation with a respondent and then let the respondent take the lead. If the respondent is highly knowledgeable on the subject raised, she or he can become a key-informant.

Purpose:

• To obtain information on aspects concerning a particular issue (e.g. health) or a subgroup of the population (e.g. women, farmers) to cross-check information obtained from other sources (e.g. household survey).
• To obtain information on the project- or at least livelihood-relevant aspects that do not differ across households (e.g. safety in the community, market access).
• To investigate general and specific issues by questioning informally but systematically.

Participatory Ruaral Appraisal (PRA)

General Description:

Participatory Rural Appraisal (PRA) is a label given to a growing family of participatory approaches and methods that emphasize local knowledge and enable local people to make their own appraisal, analysis and plans. Data collection and analysis are undertaken by local people, with outsiders facilitating rather than controlling. PRA uses group animation and simple tools to facilitate information sharing, analysis, and action among stakeholders. The PRA toolbox comprises a large variety of tools and techniques that have been extended and adapted to all sorts of development contexts over the past years. Main groups are mapping and modelling (social mapping, resource map etc.), sampling (transect walks, wealth ranking, matrix scoring), diagrams (historical diagram, seasonal calendar) and interviewing (focus group discussion, semi-structured interviews, observation). PRA emphasizes qualitative data triangulation (consultation of different sources).

Purpose:

• To facilitate analysis of and learning by the participating groups in a process of dialogue.
• To motivate and enable local groups to analyse their development potentials and constraints, identify common goals and set about achieving them.
• To mitigate the bias of expert-driven field visits.

Method for Impact Assessment of Programs and Projects (MAPP)

General Description:

Method for Impact Assessment of Programs and Projects (MAPP) has been developed based on PRA (see below) principles and tools at theGerman Institute for Development Policy (DIE). It intends to collect information on impacts through focus group discussions among the beneficiaries where a theme is evaluated with the help of simple tools. The survey team restricts itself to process facilitation. The method comprises seven instruments that are applied in a logical sequence. They basically first assess the evolution in a certain field and then attribute the results or changes to the different development activities taking place. In the context of impact assessment of stove projects, a combination of three tools has been successfully applied with groups of women.

Purpose:

• To assess impacts of development actions and their relative importance for the improvement of livelihood in a given local context.
• To gain insights into local people’s perception of impacts, whether they might be positive or negative changes.

Structured (Spot-check) Observations

General Description:

Observation is a relatively unobtrusive and effective method that is often combined with other methods, such as interviewing. Observations can be done in a structured way, using a set of preselected things to observe, or in an unstructured manner by noting down everything observed and then classifying the information according to relevant themes. When the study objectives are specific and clearly defined, as is the case in the stove and rural electrification projects, structured observations are more appropriate than unstructured ones. Spot-check observations are the simplest type of structured observations that can be conducted during a transect walk, as well as during household visits and when interviewing.

Purpose:

• To obtain first-hand information on project-related aspects such as the current status of the improved stove.
• To find out about project-related practices (e.g. concerning cooking) in and around people's homes.

Official Statictics

If available for a sufficiently disaggregated level, official statistics can be consulted as well for information on changes occurring in a certain observation field. In any case, their evolution has to be observed for all highly aggregated impacts to which the project plausibly provides a contribution with its outcomes.

Further Information

• Impact Portal on energypedia
  
• Impact Evaluation - Mixed Methods

References

• The presentation of these methods is based on: Almedom, Astier M., Ursula Blumenthal, and Lenore Manderson. 1997. “Hygiene Evaluation Procedures: Approaches and Methods for Assessing Water- and Sanitation-related Hygiene Practices.” International Nutrition Foundation for Developing Countries.