Impact evaluation

The role of the impact evaluation is to:

• support accountability and lesson learning
• encourage strategic engagement by internal and external stakeholders on critical management issues
• support the UK Government's process of justifying continued investment in development

DFID's evaluation policy states:

"The UK is committed to more and better impact evaluation of international development. Impact evaluations are rigorous research studies which are designed to look specifically at whether development interventions have resulted in specific outcomes on the ground."

The independent impact evaluation process is led by Tim Robertson - he has good knowledge of RIU as he also led the DFID-commissioned annual review early in 2010.

Tim brings to RIU 23 years' experience working with NGOs, national governments and development partners in Africa, Asia and Latin America in the fields of evaluation, enterprise development, research, governance, social protection/risk reduction and policy reform.

The impact evaluation team includes:

Tim Robertson of TANGO International
Evaluation coordination with specialisms in

• livelihoods
• income

Allyson Thirkell of Social Development Direct
Specialisms

• social exclusion
• gender
• policy

Martin Fowler of TheIDLgroup
Specialisms

• enterprise development
• policy

The team is independent but will work very closely with RIU to ensure that it can effectively capture the learning from the project and that it has accurately reflected RIU's impact. The guiding document for their work will be the revised RIU log frame and the team will be producing selected case studies of impact from across the RIU programme.

There are over 40 live experiments on how to put research into use at scale within RIU. There were other programmes which have been ended on the basis that they had not delivered the anticipated outcomes. The impact evaluation team will have to be selective about the areas they can look at given the time and resources available. To help them with this process the team is supported by a steering group. The initial membership of the steering group is made up of representatives of large funding agencies and evaluation experts and with Sir Gordon Conway (formerly DFID's Chief Scientific Adviser) as a key member. The evaluation team is working to address issues of diversity on the steering group and also looking for private sector input.

The specific brief for the steering committee is to assist in

• validating the evaluation process
• ensuring the evaluation of RIU effectively addresses the issues of accountability and focuses on issues of critical strategic relevance to wider policy debates on innovation in the agricultural sector.
• helping to ensure the products of the evaluation add value to the strategic/policy debates over agricultural innovation and research.

The team also has peer review arrangements in place which will be provided by IFPRI: this allows for consideration of methodologies and provides strengthened checks and balances.

The evaluation approach will be based on 4-5 core evaluation questions that will set the course for the work.

The evaluation team has stated that they will not be able to validate the '56 million' impact figure, so RIU will need to address this through other mechanisms. The evaluation will log the story of the transition from RNRRS to RIU in terms of value for money at RIU.

The approach proposed is to look at issues of support for and development of food producers - rather than the impact of food production on health, wellbeing or food security in the population.

The team has come up with a three phase process.

Phase one
May - September 2010

• Recruit evaluation team (June)
• Establish governance structures (July)
• Scoping study (July)

Literature review Evaluation of Innovation Systems and Agricultural Research Programmes 2 August 2010 (PDF 800KB)

• Initial evaluation questions (July)
• Evaluation framework (July/August)
• Approach paper (September)

During phase one the evaluation team will become familiar with the history and trajectory of RIU.

The team will undertake a comprehensive literature review of evaluations that have had a similar focus and objectives. Analysis of the literature review should be used to inform the design of the evaluation and selection of evaluation questions.

A set of summary tables outlining theories of change for RIU, and indicators for success for each RIU entity and network will be developed. The evaluation team will analyse these tables and use them to inform the methodology, and selection of evaluation questions, with particular reference to enterprise development and institutional/policy impact.

A review of RIU monitoring databases will be undertaken. This review will assess the quality and availability of data currently within RIU.

The key output from this first phase will be an evaluation approach paper. The paper will include sections on evaluation background, methodology, scope and sample of RIU projects to be reviewed, and key evaluation questions.

Second Phase
October - December 2010

• Implement survey design
• Train evaluation teams
• Consultant with national stakeholders
• Data collection survey
• Initial survey report

The precise details of the location, survey design and methodology cannot be determined until the evaluation approach paper has been agreed and finalised. However, the evaluation team will undertake the following:

• design survey modules and methodologies to address evaluation questions agreed in the approach paper
• design and agree a process for analyzing data and the format for reporting evidence
• design training modules and participate in the evaluation training session for the evaluation teams
• assist in the selection of national consultant(s) required to support the evaluation process
• the evaluation process will include consultation with key stakeholders including NGOs, National Government officials, Private sector actors, etc.

The key output from this phase will be an initial survey report.

Third Phase
January - May 2011

• Drafting final report
• Comments
• Redrafting
• Final Report

The structure of the final report will follow standard evaluation report templates required by DFID Evaluation Department.

The main audience for their work is DFID, other international development donors and investors (some of whom are on the steering committee) and the RIU management and programmes.