Project Evaluation Overview

Although donors have been evaluating trade-related assistance for many years, it is not easy to measure the outcomes and impacts of aid for trade. A number of conceptual difficulties, described in Chapter 1, need to be acknowledged and (where possible) overcome. A range of existing techniques and guidelines can be used to do this, and many of the building blocks are already in place. Careful planning of evaluations is crucial and should be undertaken from the beginning of a programme or project.

ALSO SEE

Evaluation Questions (World Bank)
This training module structures questions by type (descriptive, normative (or comparative) and cause and effect and suggests how to brainstorm and then narrow lists of candidate questions.

Developing Process Evaluation Questions (CDC)
This may be helpful for framing mid-project performance evaluation questions.

Utilization Focused Evaluation
This summary of Michael Quinn Patton's book includes useful insights for developing questions, which if addressed by evaluations, are likely to trigger action because the answers are viewed as being important by managers and other key project stakeholders.

At the earliest step in the project design process, the development of a Concept Paper, a USAID design team is expected to identify 1-2 central questions to be evaluated over the course of project execution, considering those identified in the CDCS.

USAID's call for evaluation questions in a project Concept Paper iterates a parallel requirement for a CDCS. Both of these requirements reflect USAID's evaluation policy commitment to improving the degree to which evaluation is an integral element of the program cycle.

When developing project evaluation questions for inclusion in a Concept Paper, project Design teams may find it useful to use the Project Evaluation Questions Worksheet included in this kit section to identify questions by levels in a project's Logical Framework on or other issues, as illustrated below.

Question Focus	High Priority Project Evaluation Questions	Evaluation Timing	Type of Evaluation
Project Purpose and Sub-Purpose	What types of improvements and/or types and level of benefits do assisted firms attribute to their involvement with this project ? disaggregated by male/female firm owners and workers?	Mid-term	Performance
Project hypotheses/ theory of change	What evidence do the records of assisted firms provide concerning a causal linkage between a reductions in the time and cost of moving goods across borders and an expansion of their exports/imports?	On-going	Impact
Sustainability	What has been the project experience to date with the retention of trained customs staff assigned to land border crossings and what does retention experience suggest in terms of the sustainability of project services and benefits?	Final	Performance

When developing questions that might be asked in project evaluations, it is important to remember that USAID undertakes two types of evaluations: performance evaluations and impact evaluations, as defined below.

USAID Evaluation Policy Emphasizes Important Links to Project Design

Impact evaluations measure the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. Impact evaluations in which comparisons are made between beneficiaries that are randomly assigned to either a treatment or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured. Any activity within a project involving untested hypotheses or demonstrating new approaches that are anticipated to be expanded in scale or scope through US Government foreign assistance or other funding sources will, if feasible, undergo an impact evaluation.
Performance evaluations focus on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making. Performance evaluations often incorporate before-after comparisons, but generally lack a rigorously defined counterfactual. Operating units are required to conduct at least one performance evaluation of each large project it implements; where large is defined as larger than the average for the unit's portfolio.

For performance evaluations, which are carried out at a single point in time, rather than over the course of an intervention, the timing of a proposed evaluation has implications for the types of questions to be asked. For example:

In mid-project evaluations process questions may help identify opportunities for improving performance, as may questions about the realism of the project design, adherence to the project's budget and schedule, and early indications about the achievement of expected results and the validity of project assumptions.
Questions about project results, including any differential impact on men and women, or their effects on youth or other target populations are likely to be answered more fully in an end-of-project evaluation, as are questions about a project's actual cost-effectiveness compared to project design stage expectations.
Questions about the sustainability of project services and benefits can be asked at any stage, but must usually be adjusted to take evaluation timing into account. Thus, for example, in a mid-term evaluation a question about the existence of a sustainability plan and early action on that plan might be appropriate. An end-of-project evaluation could address questions about how effective a sustainability plan seems to be and early evidence concerning the likely continuation of project services and benefits after project funding ends. Only an ex-post evaluation, however, can provide empirical data about whether a project's services and benefits were sustained.

ProjectStarter

BETTER PROJECTS THROUGH IMPROVED
MONITORING, EVALUATION AND LEARNING

A toolkit developed and implemented by:
Office of Trade and Regulatory Reform
Bureau of Economic Growth, Education, and Environment
US Agency for International Development (USAID)

For more information, please contact Paul Fekete.