External Evaluation of the Multipurpose Cash Programme Consultancy

Background:

Founded in 1956, the Danish Refugee Council (DRC) is a leading international NGO and one of the few with a specific expertise in forced displacement. Active in 40 countries with 9,000 employees and supported by 7,500 volunteers, DRC protects, advocates, and builds sustainable futures for refugees and other displacement affected people and communities. DRC works during displacement at all stages: In the acute crisis, in displacement, when settling and integrating in a new place, or upon return. DRC provides protection and life-saving humanitarian assistance; supports displaced persons in becoming self-reliant and included into hosting societies; and works with civil society and responsible authorities to promote protection of rights and peaceful coexistence.

The Danish Refugee Council (DRC) is the lead agency for Cash Consortium of Yemen(CCY) .The Cash Consortium of Yemen (CCY) was set up in 2020 to improve the living conditions of crisis-affected people in Yemen by maximizing the effectiveness of cash assistance and supporting the transition to social protection mechanisms. The Cash Consortium of Yemen (CCY) is led by the Danish Refugee Council and includes Acted, the Norwegian Refugee Council, Mercy Corps, Solidarities International, and the International Organisation for Migration (IOM). CCY’s program has operated under two core pillars:

• Distributing MPCA to crisis-affected HHs to meet their critical basic needs and reduce negative coping strategies. CCY promotes MPCA as its proposed modality based on its demonstrated effectiveness in Yemen, leveraging on functioning and accessible markets; and the flexibility that cash assistance provides.
• Establishing, expanding, and utilizing referral mechanisms to increase access to other tailored assistance and/or longer-term assistance.

The complex situation in Yemen requires ongoing evaluation of implementation strategies and approaches to ensure that programs achieve their desired impact effectively. The Evaluation Plan provided here clearly describes the evaluation activities of CCY’s program. It precisely outlines the important aspects of each activity. These include defining the purpose and scope, formulating crucial evaluation questions, specifying the methodology, choosing the evaluation team, identifying the primary audience, setting a timeline and budget, coordinating with other consortium partners, and detailing the dissemination of findings and recommendations.

SECTION I – ProGRAMME Overview

Building on the CCY partners’ extensive experience in cash-based transfer programming within Yemen, the CCY proposes to a) provide MPCA as a first-line humanitarian response and b) support longer-term assistance to address more protracted needs by employing a strong referrals network to open pathways for vulnerable populations to access a wide range of services. The MPCA beneficiaries targeted under this award will include socio-economically vulnerable Yemeni IDPs, returnees, refugees, host communities, and those most affected by food insecurity and epidemic outbreaks. This proposed activity directly supports the Yemen 2021 Humanitarian Response Plan Yemen and its focus on ensuring maximal coverage of the most urgent, life-saving assistance integrated with a longer-term perspective.

a) Goal: To increase the ability of vulnerable conflict-affected populations of Yemen to build stable foundations through addressing the most acute humanitarian needs and linking humanitarian and social protection programming.

b) Theory of Change Statement: The proposed Theory of Change (ToC) states that: if individuals have access to food, essential services, non-food items and dignified shelter, if cash flow is increased in local markets, if the political, security and protective environment remains conducive, and if sequencing of humanitarian and development efforts is facilitated, then conflict-affected and vulnerable communities become more resilient and are better able to cope with shocks and crises.

c) Purpose of Program: To improve the dignity and living conditions of affected populations in Yemen through harmonized, coordinated, timely, safe and accountable needs-based MPCA and linkages to longer-term assistance.

As of December 2023, the CCY program had assisted beneficiaries in 16 governorates across both North and South Yemen. These beneficiaries comprise various groups, including members of the host communities, households internally displaced by conflict or natural disasters, persons who have returned to their places of origin, as well as refugees and migrants. The table presented below outlines the number of households that received support from the CCY program between 2020 and 2023.

SECTION II – Project Monitoring System

CCY Yemen has established a robust MEAL system to align with the country’s context and meet the needs of stakeholders. The project ensures a continuous intake of beneficiaries by receiving numerous referrals from multiple agencies. CCY has devised its criteria for selecting beneficiaries by assessing their vulnerability, food insecurity, coping strategies, and economic capacity to ensure eligibility and identify unique recipients. CCY currently uses four different tools, i.e., vulnerability assessment, baseline, post-distribution monitoring and endline to collect the indicators. Below is a list of outcome indicators from the CCY indicator performance table:

Result 1

1

% of HHs who received the assistance within 14 days from the registration

2

# of households receiving the full amount of the MPCA, by one-off, 3-month and 6-month MPCA

3

Total value (in EUR) transferred to beneficiaries in MPCA, by one-off, 3-month and 6-month MPCA

4

% of households who report being able to meet the basic needs of their households (all/most/some/none), according to their priorities.

5

% of beneficiaries with acceptable FCS (>80%).

6

% of households with total monthly expenditure which exceeds the MEB

Result 2

1

Number of referral pathways mapped.

2

Number of sectoral referral pathways integrated into the CCY system.

3

Number of households successfully referred through the CCY system, by referral type.

Result 3

1

Remittance trackers, Exchange rate trackers, Price Monitoring Tool, CCY Special Report

2

Number of learning products (studies, evaluations, research) produced and disseminated by CCY

3

Percentage of feedback/complaints received which have been timely and effectively acted upon

4

Number of key CCY program adaptations introduced as a result of consortium learning activities

5

# of price forecast data updates integrated into the online dashboard

To monitor the implementation and desired outcomes of the activity, the targeted beneficiaries undergo surveys for pre-assistance baseline, post-distribution monitoring PDM, and endline evaluations. Remarkably, CCY maintains one of the largest beneficiary databases in Yemen, making the program exceptionally cost-efficient by reducing duplicate assistance and specifically targeting the most vulnerable individuals. Additionally, the project utilizes software to keep track of outputs, ensuring efficient maintenance and monitoring throughout the process

SECTION III- CCY LEARNING AGENDA

The Evaluation Plan aligns with the interest of USAID and DG ECHO in expanding the knowledge base related to multipurpose cash assistance practices. The evaluations are intended to contribute valuable evidence to the decision-making process based on their Learning Agenda questions. This evaluation will align with the majority of USAID’s learning agenda, with a specific focus on the following areas:

• RESILIENCE TO SHOCKS: The evaluation will explore how CCY enhanced the resilience of households, communities, and the country against various challenges such as climate-related issues, conflicts, economic downturns, and health crises like COVID-19 and other global pandemics.

The research question from the Learning Agenda will be incorporated into the evaluations. CCY’s program will concentrate on reinforcing the multipurpose cash assistance and implementing livelihood approaches for the affected community by 2024. The findings obtained from these evaluations will enable the consortium body and donors to gauge the extent to which cash assistance helped the beneficiaries build resilience against natural and man-made shocks and stressors.

SECTION Iv- Purpose and SCOPES of the Evaluation

The main objective is to evaluate the program’s performance against the desired results as articulated in the project’s result framework. Specific objectives of the evaluations include:

1. Assess whether the programme has adequately responded to the short, medium and long-term challenges as expressed in the project documents, including through expected synergies of a multipurpose cash assistance approach.
2. Assess the overall project’s performance from planning, implementation and knowledge management by identifying the key strengths and areas of gaps and making the necessary recommendations for future improvement
3. Document vital lessons learned, innovations/best practices resulting in future strategies and interventions
4. Understand better what the project has achieved concerning gender and disability inclusion and protection mainstreaming and overall results
5. Support the use of relevant and timely contributions to organizational learning, informed decision-making processes resulting from the analysis, conclusions or recommendations as well as accountability for results.

In order to achieve these objectives the evaluator should focus on the criteria and evaluation questions in Section V below:

SECTION v- Criteria and Evaluation Questions to move forward:

Criteria and Evaluation Questions

Sources of Information

Data Collection Tool

Relevance and Coherence

1. To what extent are the objectives of the project consistent with the beneficiaries’ requirements, and needs?
2. Has the CCY partnership approach been appropriate or effective?
3. Does the programme strategy and intervention package respond to the priority needs of displaced and crisis-affected beneficiaries?

• Project documents
• CCY staff
• Local authorities
• Literature Review
• KIIs
• IDIs

Effectiveness

1. Are the projects’ outputs and outcomes on track to be achieved in accordance with the stated plans?
2. Did the project’s activities and outputs lead to the intended outcomes?
3. To what extent are the project implementation arrangement and strategies adequate for achieving expected results (budget structure, reporting lines, human resource allocation in line with project design?)
4. Are the program management strategies effective in delivering program plans and results?
5. Did the program/project achieve the intended outcomes, and to what extent was the result achieved by this time?

• Project documents
• CCY staff
• Partner NGO staff
• Local authorities
• Literature Review
• KIIs
• FGDs
• IDIs

Efficiency

1. Were activities implemented on schedule and within budget? – outputs delivered economically
2. How efficiently were the resources used to achieve the intended results in line with the implementation context?
3. How well are the resources (funds, expertise and time) being converted into results?

• Project documents
• CCY staff
• Partner NGO staff
• Literature Review
• KIIs

Sustainability

1. To what extent are the project activities connected to longer-term development concerns?
2. Are the benefits likely to be maintained for an extended period after assistance ends?
3. What is the likeliness of increased gender equality beyond the project end?

• Project documents
• CCY staff
• Literature Review
• KIIs

Impact (Early Signs of Impact)

1. What changes did the project bring about against each outcome category?
2. Were there any unplanned or unintended changes?
3. To what extent will the gender-sensitive?

• Project documents
• CCY staff
• Partner NGO staff
• Government officials
• Literature Review
• KIIs
• FGDs
• IDIs

Cross-cutting Issues

1. To what extent did the project design incorporate gender considerations and the needs of persons with disabilities?
2. What were the gender equality results and objectives achieved? Were gender mainstreaming principles adhered to by the project?
3. To what extent did the project adhere to humanitarian protection principles in designing and implementing the project?

• Project documents
• CCY staff
• Partner NGO staff
• Literature Review
• KIIs
• FGDs
• IDIs

In addition to that CCY is interested to see the results of the following questions:

Coping Strategies:

• Was the project successful in preventing targeted households from adopting negative coping strategies?
• With which groups was the project more successful in preventing negative coping strategies?
• With which groups did the project have less success, and why?
• Were these changes temporary or permanent?
• What factors contributed to these outcomes?

Resilience Strengthening:

• How has the project contributed to strengthening household-level resilience as targeted through the action?
• What factors have helped or hindered households in achieving a reasonable level of resilience?

Basic Needs and MPCA:

• What basic needs were beneficiaries able to cover with the project’s Multi-Purpose Cash Assistance (MPCA)?
• How did beneficiaries manage to cope with these needs after the project ended?

Impact on Food Security:

• What impact did the project have on the food security at the household level of the targeted groups?

Other aspects:

• What are the top 5 learnings from the project?
• Provide a list of recommendations to further improve the relevance, effectiveness, efficiency, and sustainability of the project and its results.

SECTION VI – Key Evaluation Stakeholders and Users

Several internal and external stakeholders to CCY have interests in the evaluation results, and some stakeholders will be asked to play a role in the evaluation process. Those include CCY CMU, IOM, DRC RO Middle East, and CCY Partner agencies. Externally, USAID BHA and ECHO would be two leading stakeholders as they are the major funders of the CCY program. Additionally, UNFPA, IRC, UNICEF, and WFP might find the evaluation interesting as beneficiary referral partners. Moreover, many sectors, for instance, RRM, CCCM, Cash and Market working group, Food and Nutrition, are involved with the CCY operationally and strategically. Local authorities in Yemen, specifically in the North (De-Facto Authority) and South (Internationally Recognized Government) have been providing critical support for this program and might find the evaluation results meaningful.

SECTION VII – Methodology for the evaluation

• The evaluation methodology should be aligned with OECD DAC criteria. It should be developed based on the requirements outlined in the TORs, along with further analysis conducted during the inception phase and consultations with key stakeholders. The evaluation team will collaborate to formulate an appropriate evaluation design, sampling strategy, and methodological approach, which will be documented in the Inception Report.
• Desk Review: The scope of the desk review should go beyond project documents and encompass external sources, providing a thorough grasp of the context in which the project operated and the factors that either facilitated or hindered its objectives. Complementary to this, CCY outcome data will be made available to augment the evaluators’ information sources.
• In-depth Interviews (IDIs): Interviews should be conducted with staff and partners familiar with the project. These interviews aim to benefit from the staff’s detailed understanding of CCY operations.
• Key Informant Interviews (KIIs): Interviews should be conducted for community leaders, local authorities, and other key stakeholders external to the project management. These interviews will tap into individuals with knowledge of the project’s purpose, outcomes, and impact in the communities where it was implemented.
• Focus Group Discussions (FGDs): The primary focus of FGDs should be the heads of households, community leaders, camp leaders, representatives of the Supreme Council for the Management and Coordination of Humanitarian Affairs and International Cooperation (SCMCHA) in the North, as well as the Ministry of Planning and International Cooperation (MOPIC) and the Executive Unit (ExU) for IDP Camp Management in the South. The targeted stakeholders are expected to share their perspectives and experiences as they should have knowledge of the project’s purpose, outcomes, and impact in the communities where it was implemented.

SECTION VIII – ROLES AND RESPONSIBILITIES

CCY MEAL Manager will assume the role of the Evaluation Manager, ensuring no direct involvement with the implementation partners. This position will provide continuous support to the evaluator throughout the entire process, assisting in developing a robust methodology, supporting in establishing efficient data collection systems and facilitating communication with stakeholders for interviews.

Meanwhile, DRC’s Regional MEAL Coordinator will provide additional support to the evaluation, ensuring alignment with DRC’s evaluation policies, protocols and subjective expertise. The data collection must be conducted independently by the evaluation agency, and whether it will be done remotely or in person will be decided based on the security situation and necessary permissions. The CCY Chief of Party and MEAL Manager will serve as the overall focal points for this evaluation, taking charge of initiating the selection process, connecting the evaluator with internal/external stakeholders, and validating the final report.

SECTION IX – Quality Assurance and Quality Assessment

CCY has established clear quality standards for this evaluation, incorporating processes with built-in steps for quality assurance, along with providing templates for evaluation products and checklists for review. The principles and criteria expected for this evaluation should align with the OECD DAC criteria, ensuring a consistent and robust approach.

To enhance the quality of the inception/evaluation report, the Evaluation Manager will provide inputs, with the support of the CCY’s Chief of Party. Throughout the analysis and reporting phases, the Evaluation Manager, along with CCY’s MEAL/IM team, will provide the required support to ensure data quality.

The evaluation team can be assured of access to all relevant documentation within the framework of the directive on disclosure of information. This ensures transparency and enables the team to have access to the necessary resources for conducting a comprehensive evaluation.

SECTION X- Selection of the EvaluatoR

The evaluation for the CCY will be undertaken by a qualified and independent evaluation agency. The selection will include a competitive tender process. The evaluation agency will be selected according to the following criteria:

• The evaluation team should comprise a team leader and other national and international team members, as necessary, to ensure a complementary mix of technical expertise in the evaluation’s focus areas (resilience, cash assistance, food security, gender and protection, capacity development etc.). In addition to the core evaluation team, a local field team can be engaged by the evaluator to support the process of data collection.
• The evaluator should have previous experience in evaluating cash assistance, food security, livelihood or resilience projects. Prior experience in evaluating ECHO and/or BHA projects is expected.
• The evaluator should have the capability and permissions to collect data in Yemen either directly or through local field teams.
• All team members will be independent consultants and will include a mix of international and host-country national consultants (if relevant) who will work jointly (remotely if travel restrictions are still in place) to deliver evaluation products.
• The lead evaluator should have a minimum of five years of experience in impact evaluation, research, and survey design.
• The evaluation team should ideally have conducted baseline/mid-term/endline studies/evaluations within the framework of a program evaluation.
• The team leader must also demonstrate strong leadership experience.

SECTION XII –timeline for delivery

The items are detailed in the table below:

Date

PHASES

Apri 2024

Assign roles/responsibilities and establish an Evaluation Committee (CCY)

Develop Terms of Reference (TORs) and budget (CCY)

Procure independent evaluation firm (CCY)

INCEPTION PHASE FOR EVALUATION

April 2024

Desk review of key project documents (evaluation team)

Inception planning with KIIs conducted (evaluation team and CCY)

Prepare Inception Report including quantitative and qualitative data collection tools (evaluation team)

DATA COLLECTION AND ANALYSIS

May 2024

Determining the sample size

Enumerator training, data collection, data cleaning

Conducting desk reviews

Data analysis (evaluation team)

FINAL EVALUATION

June 2024

Present the preliminary findings

Draft and finalize final evaluation report (evaluation team with inputs from CCY evaluation committee)

Disseminate final evaluation findings to internal stakeholders

Disseminate final evaluation findings to external stakeholders

Prepare a response to the CCY Consortium Management Unit’s comments on the recommendations (CCY)

**The timelines are subject to be revised based on the onboarding of an external evaluator and discussion with CCY

The payments will be made in two instalments. The first payment is 40% of the total money to be disbursed after the inception report. The second instalment is the rest of the 60% money to be paid after receiving the final reports.

SECTION XIII – Guideline for Deliverables

• The technical proposal during the bid submission should not exceed 6 pages and must have key evaluation questions, detailed methodology with rationale, and the role of each team member in this assessment. Providing irrelevant information may be considered as the corrosion of the bidder’s strength.
• The inception phase is set for approximately 10 working days. During this time, it is recommended to conduct 2-3 IDIs, enabling evaluators to gain a deeper understanding of the project and develop more realistic evaluation tools
• A detailed inception report (maximum 20 pages) with evaluation questions, methodology, link between the evaluation questions and methodology with rationale, timeline, data collection methods, analysis methods/theories, etc.
• A brief of the preliminary results and findings to the evaluation committee
• A first draft report comprising the whole picture of the evaluation for review
• A final presentation to the stakeholders (CCY partner agencies and donors)
• Two final evaluation reports (internal and external versions). The internal report should not exceed 40 pages, and the external report should be very concise, with 2-3 pages highlighting key findings, learnings, conclusions, and recommendations should be provided. Two of these reports should be validated by all the CCY executive committee members and the donors.
• Publishing the evaluation reports in ReliefWeb

Dissemination Plan

Date

Tasks

June 1, 2024

Presenting preliminary results and findings to the CCY CMU

June 3, 2024

Presenting preliminary results and findings to the CCY partners

June 4, 2024

Presenting preliminary results and findings to the donors

June 7, 2024

Submission of internal and external versions of the evaluation report

June 14, 2024

CCY provides feedback received internally and externally

June 21, 2024

The evaluator submits the final reports after addressing all the comments and suggestions

June 22-25 2024

CCY disseminates the final report to all the stakeholders and publishes it externally

SECTION XIV – Terms and conditions

• The payment instalments entirely depend on handing over quality deliverables. If the expected quality is not produced, then DRC can terminate the contract with justification.
• The evaluation is expected to be completed within 3 months; however, if the situation requires, the evaluator should remain flexible for short extensions to the timeline to deliver the products.
• DRC will not provide any visa, accommodation, or safety support to the evaluator or the data collector team in Yemen. There are no payment provisions aside from the amount budgeted for the assessment.

How to apply

Bids can be submitted by email to the following dedicated, controlled, & secure email address:

[email protected]

Click Here to download full package.