End of project Evaluation Consultant

Position: End of project Evaluation Consultant

Location: Yemen

Duration: May 1, 2022 through November 1, 2022

Reports to: Deputy Country Director – Programs

Start Date : May 1, 2022**

About Relief International.

Relief International (RI) is a leading nonprofit organization working in 16 countries to relieve poverty, ensure well-being and advance dignity. We specialize in fragile settings, responding to natural disasters, humanitarian crises and chronic poverty.

Relief International combines humanitarian and development approaches to provide immediate services while laying the groundwork for long-term impact. Our signature approach — which we call the RI Way—emphasizes local participation, an integration of services, strategic partnerships and a focus on civic skills. In this way, we empower communities to find, design and implement the solutions that work best for them.

Relief International includes the four members of the RI Alliance in the USA, UK, France, and Belgium. Under our alliance agreement, we operate under a single, shared management structure with one CEO and one Board of Directors.

BACKGROUND

RI is implementing 14.5-month program to address the Nutrition, Health, WASH, Multi-Purpose Cash Assistance (MPCA) and Protection needs for the affected population in Milhan and Al Tawila district in Al Mahweet governorate, Khayran Al Muharraq district in Hajjah governorate and Ataq districts in Shabwah governorate.

The project provides services and support to affected communities across five sectors i.e. Health, Nutrition, WASH, MPCA and Protection. Under health and nutrition, RI is supporting primary health care including mental health, management of acute malnutrition through Community-Based Management of Acute Malnutrition (CMAM) approach, infant and young child feeding activities and referral to secondary healthcare. The WASH component is deployed at health facility level through rehabilitation, improvement of sanitation and water supply, supporting Integrated Food Security Phase Classification ‘IPC’ and waste management for services provision; while community level hygiene promotion and distribution of hygiene kits is implemented in supported communities. RI is providing MPCA through cash distribution for households affected with severe acute malnutrition admitted into supported health facilities for treatment, and newly arrived IDPs in targeted districts. Protection mainstreaming constitutes a cross cutting aspect of all activities, while protection services provision and awareness has been set up in Shabwah governorate through the establishment of a Women and Girls center (safe space), integrated with the reproductive health services provision component.

PURPOSE

The end of project evaluation will be shared with BHA and relevant stakeholders to give a clear vision of the project achievements, outcomes and performance of implementation, lessons learnt about the implemented modality, best practices that can be replicated and challenges faced. The main objectives of the evaluation are:

1) To measure the immediate outcome of the project and its performance.

2) To assess the achievement made in the program implementation and the progress made in achieving the overall goal of the project.

3) The evaluation will identify the best practices, challenges, recommendations and lessons learned related to projects in the targeted areas. The information obtained will enable RI to use the knowledge attained for future programming.

SCOPE **

The evaluation will be conducted in the project locations in Shabwah, Al Mahweet and Hajjah governorates, taking into consideration the project’s indicators as identified in the proposal.

Specifically, RI wishes to confirm the following general points:

● In the opinion of direct beneficiaries and local stakeholders, was RI’s programming responding to the needs of the beneficiaries.

● Evaluate the effectiveness of Health and Nutrition support to beneficiaries, their access to health and nutrition services, and the impact on the communities over the longer term.

● Evaluate the results of the program in linkages between attendance to Anti Natal Care (ANC) and choice of place of delivery, motives for attending ANC, and barriers faced in accessing Health Facilities (HFs) for delivery.

● Evaluate the satisfaction of beneficiaries with the services supported across activities and sectors.

● Evaluate the effectiveness of the complaints response mechanism (CRM) in terms of accessibility and usage by the beneficiaries, and identify what barriers might have prevented awareness and/or access.

● Evaluate MPCA usage and understand its impact on the targeted beneficaires, and targeting of cash for nutrition under the rapid response mechanism (RRM) which targeted newly displaced populations

● Evaluate the reach of hygiene promotion activities and adequacy of messaging provided to address barriers encountered in practices.

● Evaluate the contribution of WASH rehabilitation on IPC and capacities on waste management improvement.

● Assess the degree of integration and possible additional leverages to be explored between the health and protection sector, within the Women’s Centers (WC) activity component.

● Evaluate the perceptions of access and safety of participants to WC activities, both in terms of the physical space and services provided.

● Evaluate whether the activities were designed and implemented in a gender-sensitive manner considering the specific context and societal norms of served communities and locations, especially focusing on whether males and females were adequately engaged to participate.

In relation to the above objectives, RI requests the External Firm to apply the five OECD DAC evaluation criteria of Relevance, Effectiveness, Efficiency, Impact and Sustainability of the project.^[1] The intervention logic (i.e. logframe) must also be properly taken into consideration. The Consultancy firm needs to make sure that both qualitative and quantitative methods are used to gather data in order to investigate the components and answer the questions in the OECD DAC criterion mentioned in Table 1.

Table 1: Assessment criteria for the External Evaluation

Evaluation Criteria

Questions to be considered

Relevance

· Were the provided services and program objectives within the prioritized needs of the community and aligned with the targeted response of the health cluster?

· Was the program aligned to the priorities and policies of RI, target group(s), donor(s) and other stakeholders?

· Were the objectives of the program still valid and are the outputs of the project consistent with the objectives?

· Was RI able to practice its own values and principles such as ensuring a “Do No Harm” approach?

Efficiency

· Could the activities and outputs have been delivered with fewer resources without reducing their quality and quantity?

· Were the outputs delivered within the agreed budget and timeline?

Effectiveness

· What are the major factors identified that were influencing the achievement or non-achievement of the objectives?

· To what extent have beneficiaries been satisfied with the program?

· Did the program have effective monitoring mechanisms in place?

Impact/Outcome

· Does the intervention contribute to reaching the higher-level objectives (the goal of the project) and make a significant long-term change?

· What real difference has the program made to the target groups (and beyond)?

· What changes can be identified that may (in part) be caused by the program? To what extent can the changes be measured?

· Was the modality of the HF used in this project the most relevant modality in regard to its effectiveness, impact on access to health and nutrition services, impact on the communities in the longer term?

· What are the lessons learnt from the response using this modality or overall achievements and challenges in this program?

· Are there any unintended effects of the program?

Sustainability

· Are the positive effects or impacts sustainable?

· Are we building the capacity of local stakeholders to sustain the project?

· What is the likelihood that the benefits from the program will be maintained for a reasonable period of time if the program were to cease?

· Is the program supported by local/national institutions and adapted to local/national conditions? Are requirements of local/national ownership satisfied?

METHODOLOGY

The methodology for this evaluation should use a mixed-methods approach. Tools will be developed by the External Firm in close collaboration with RI, taking into consideration themtatic and geographical aspects of the program The Consultancy firm should have the capacity to conduct the number of surveys in compliance with the minimum required sample size, both at the HFs targeted areas and at the community level.

The Consultancy firm is recommended to use the following qualitative and quantitative methodologies, however is not limited to these, any additional methods will be developed in cooperation with RI during the design phase:

1. Literature Review and Secondary Data Analysis: to get facts and overview of the project context; this includes an analysis of past project reports, monitoring reports, knowledge, attitudes and practices (KAP) survey results, and program reports from RI.**

2. Primary Data Collection: will include field visits, observations, surveys, participatory and focus group discussion with all beneficiary groups, not limited to patients but including, community representatives, and staff worked with HFs, governorate and districts health directorates, and local councils where possible and applicable. **

a. Quantitative Methods

Household Survey

The Household Survey, will consist of various components. First it will seek to understand beneficiaries access and coverage to services as well as the direct beneficiary satisfaction by measuring the level of awareness and physical, social and financial accessibility to the health facilities and provided services.

· Identifying the most important barriers to accessing health facilities and services.

· Identify the barriers to services by Community Health Volunteers (CHVs).

· Measuring their knowledge on good hygiene practices.

The second component will focus on assessing knowledge, attitudes and practices of the communities located in and around the HFs targeted areas on Health and Nutrition-related topics. Specifically, the households should be assessed to see if they have been visited by a CHV, measuring their knowledge on health and nutrition topics to see if they have adapted any positive health and hygiene practices as a result of the awareness sessions.

The survey will include the section of questions directed to beneficiaries who confirm that they/their families were targeted by services provided within this project. This patient/beneficiary part of the survey will cover both Health and WASH components in all targeted HFs, the targets for the survey will be patients who have visited the HFs.

b. Qualitative Methods

Key Informant Interviews

The objective of these Key Informant Interviews (KIIs) should be to analyze the quality of RI’s implementation of the HFs in the community. In particular, they will provide an insight into whether standard operating procedures set by RI in relation to WASH, Health and Food Assistance are being followed, and whether the intervention meets specific quality standards such as those provided by SPHERE Core Humanitarian Standards and WHO. Moreover, gauging staff’s opinion on available resources, the ability of HFs to effectively support the community, and any challenges related to the referral system will provide important programmatic insights.

Focus Group Discussions

Focus Group Discussions (FGDs) will be conducted with males and female patients separately in each of the targeted communities. These FGDs will aim to capture qualitative data with some semi-quantitative data. This will include, approximates of the total community who have access to services; who use services; who have awareness of specific issues relating to health programming at the facilities; the involvement of women and children; challenges preventing part [or all] of the community from benefiting from services; why the community is not following referrals etc.

Health and Nutrition Component: Two FGDs will be conducted with both male and female patients separately in each of the targeted health facilities Each FGD will cover between 8-10 participants and will aim to capture primarily nuanced qualitative data with some semi-quantitative data (i.e. approximates of the total community who have access to services; who use’s services).

WASH Component: Two FGDs will be conducted at each project site, one with male and one with female beneficiaries. The FGDs will be conducted at household level in the communities where beneficiaries were supported by hygiene promotion activities.

Protection Component: Two FGDs will be conducted in Women and Girl Centers with both women and girls and each group should allow for the participation of about 8-10 participants (minimum required is 8). The FGD questions will be aimed at exploring protection issues, assessing what the available solutions at the community level are and at the RI level respectively, if there are barriers to access to those services and solutions.

ROLES AND RESPONSIBILITIES

RI Responsibilities:

RI will be responsible for any support relevant to the review and development of the data collection tools, draft reports and approving finalization of reports. In addition, RI will provide:

· Any relevant background materials and documents to support in designing tools and completing the desk review.

· Interviewees contact information, including RI field staff, to mitigate problems or challenges during the evaluation team’s field visits.

· RI MEAL staff will ensure that the selected consultancy firm is conducting the data accurately and in a safe manner, ensuring a high quality of data.

· RI will be responsible in reviewing any draft reports, especially in case some of the findings could be explained further and reported accordingly.

Evaluation firm Responsibilities:

· The consultancy firm will be responsible for coordination with the local authorities for permissions to conduct the evaluation, including on-site data collection and survey activities.

· The consultancy firm will be responsible for logistics, administrative support, telecommunications, printing of documentation, and logistics for fieldwork, etc. The firm is also responsible for the dissemination of all methodological tools such as surveys;

· Mobilize and deploy the survey teams and enumerators in the field to ensure quality of data collected according to the samples established and agreed-upon with RI.

· The firm should budget for all necessary expenses in their Financial Proposal.

· The firm will be leading and managing the data collection in the field.

· Data collection and analysis will be the sole responsibility of the selected consultancy firm.

· Presentation to RI team on findings of evaluation and highlight area of focus.

Confidentiality

All external consultants/firms involved will commit to strictly complying with the confidentiality of information obtained during the evaluation process. This includes but is not limited to: program documents including datasets, reports and annexes and technical proposal, given the sensitivity of the program and context. Deliverables generated throughout the consultancy, including the final evaluation report, remain the property of RI and BHA unless otherwise requested to and approved by RI.

DELIVERABLES AND SCHEDULE

Four major deliverables that will be requested are:

1) An inception report which contains the intervention logic of the program (based on desk study), an evaluation plan, and a list of reviewed documents. The evaluation plan should contain the proposed data collection methods and data sources to be used for addressing each evaluation question/dimension, taking the above methodology section into consideration, with a clearly defined timeline and resources required.

2) Data collection in the field (see methodology section).

3) A draft evaluation report: stakeholder workshop should be facilitated between the evaluation team and RI to discuss preliminary evaluation findings and conclusions at this stage. The timeline of the evaluation should allow sufficient time for stakeholders to review and discuss the draft report. Holistic and final sector-specific analysis and feedback will be provided to enable the technical units to analyze the findings before completion of analysis and validation.

4) The consultant will share the final evaluation draft report with RI to be revised and commented by RI for finalization in view of concluding the consultancy. The final evaluation report, and a final findings presentation, should be prepared by the firm. The final evaluation report should be written in English language and should not exceed 30 pages (excluding annexes). It should be submitted electronically in both an MS-Word document and PDF. It may include:

RI aims to adhere to the following tentative timeframe for some components feeding in to the deliverables per each evaluation to take place:

Deliverables/ Components

Dates

Deliverable #1: Inception report for project

In a week (7 days) after signing the contract

Deliverable #2: Primary data collection: interviews with stakeholders (key informants, RI staff members), interviews with beneficiaries and vendors (including FGDs); household and patient surveys and observation checklists

Raw data for all of the conducted activities including paper consent forms and any other voice or video recordings.

Within the first month (30 days) after delivery of the inception report

Deliverable #3: Draft report to RI (including raw data)

Within three weeks (21 days) after completion of data collection.

Deliverable #4: Final report & presentation

RI will provide feedback and consolidated remarks to the firm within 14 days after reception of the Draft evaluation report.

The firm will provide a revised Final Evaluation Report including necessary adjustments within one week (7 days).

Additional engagement, review and adjustment will be entertained between RI and the firm for an additional week (7 days) maximum.

Final presentation of results will be organized within one week of the final approval from RI on the final evaluation report and appendices.

The consultant is also expected to present the final report to RI’s Deputy Country Director – Programs, MEAL Technical Advisor and the BHA Program Manager. In addition, raw data from field interviews, surveys, or focus groups results should be provided to RI. The reports will highlight good practices, areas of improvement, and recommendations. Draft versions of the reports will be submitted to RI for review before final versions are submitted.**

Annex 1: Project Locations of HF

S.N

Governorate

District

health facility

Catchment population

1

Al Mahwit

At Tawila

Bayt Maneayn HU

5,640

2

Al Mahwit

At Tawila

Wadi Laea HU

6,654

3

Al Mahwit

Milhan

Bani Hajjah HC

4,371

4

Al Mahwit

Milhan

Kinanah HU

5,976

5

Hajjah

Khayran Al Muharraq

Al Shawarfah HC

3,680

6

Hajjah

Khayran Al Muharraq

Al Modeirah HU

6,582

7

Shabwah

Bayhan

Alrokbah HU

6,120

8

Shabwah

Bayhan

Alkaraa HC

7,200

9

Shabwah

Ataq

Qura Al Mahdi HU

4,350

10

Shabwah

Ataq

Ba Kabearah HU

4,150

Relief International’s Values:

We uphold the Humanitarian Principles: humanity, neutrality, impartiality and operational independence. We affirmatively engage the most vulnerable communities.

We value:

Inclusiveness

Transparency and accountability

Agility and innovation

Collaboration

Sustainability

^[1] The Organization for Economic Co-operation and Development (OECD) Development Co-operation Directorate (DCD-DAC) Criteria for Evaluating Development Assistance can be found at the following link: http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm