INDEPENDENT EVALUATION FOR DEC PROJECT IN NORTH EAST SYRIA – CARE INTERNATIONAL

Terms of Reference

Independent Evaluation for DEC Project In North East Syria – CARE International

phase 1 from June 2020 to Jan 2021

phase 2 from Feb 2021 to July 2022

1. Introduction

1.1. Background

CARE is a humanitarian non-governmental organization committed to working with poor women, men, boys, girls, communities, and institutions to have a significant impact on the underlying causes of poverty. CARE seeks to contribute to economic and social transformation, unleashing the power of the most vulnerable women and girls.

CARE International in Syria has several years of experience in humanitarian interventions in Food Security and Livelihoods (FSL), Water, Sanitation and Hygiene (WASH), protection, shelter/Non-Food Items (NFIs), and multi-sector emergency response interventions. Response activities throughout Syria have taken a resilience approach, providing basic needs in moments of crisis, and working to help build mechanisms to support populations to cope with the crisis. CARE’s Northeast Syria (NES) operational hub implements programming directly and refines approaches to partnership, capacity building, and remote management through hybrid modalities to access hard-to-reach areas.

Under the DEC project COVID-19 appeal in Syria. Phase 1 activities, which were implemented between 14th June 2020 and 31st January 2021, consisted of the distribution of 19,685 COVID-19 kits and 496 full hygiene kits. In addition, CARE through a local partner “Peace Civil Society Center” (PCSC), provided protection services that benefitted 1,428 individuals.(917 Female, 511 Male)

The current Phase 2 includes the rehabilitation and maintenance of water treatment units and the provision of handwashing stations and COVID-19 hygiene kits, is being implemented between 1st February 2021 ending 31st July 2022.

Under Phase 1, CARE’s intervention had two expected outcomes:

1. Increased access to WASH NFIs and protection services for Internally Displaced Persons (IDPs) and host community members in Al-Hassakeh and Deir-ez-Zor governorates
2. Increased access to COVID-19 specific WASH NFIs for IDPs and host community members in Al-Hassakeh

Under Phase 2, there is an additional or 3rd expected outcome:

3. Increased access to WASH services for IDPs and host community members in Al-Hasakeh governorate

To achieve these outcomes, CARE implemented the following key activities during Phase 1:

• Distribution of hygiene kits to vulnerable families in Al-Hassakeh and Deir-ez-Zor
• Distribution of COVID-19 specific hygiene kits to vulnerable families in Al-Hassakeh
• Relevant sensitization and information sharing on critical protection issues for 1,150 individuals (686 Female, 464 Male).
• Provision of psychosocial support (PSS) and Prevention of sexual and gender-based violence (SGBV) sessions, and referrals to other service providers such as legal aid and medical support to 351 (227 Female, 124 Male) individuals

Additionally, in Phase 2, CARE is implementing the following activities:

• Establishing 6,000 handwashing stations at HHs in Al-Hassakeh to help families maintain safe and dignified personal hygiene and prevent the transmission of COVID-19
• Increase access to safe water supplies through rehabilitation and maintenance of 2 Water Treatment Units in Al-Hassakeh district, Al Hasakeh sub district in “Abdan water treatment plant and Hole water treatment plant”.
• Rehabilitation of Makhroum water station in Al Hasakeh district, Al Hasakeh Governorate
• Distribution of 27,095 COVID-19 kits in Washokani and Abu Kashab camps in Al-Hassakeh governorate, Al Hasakeh district.

CARE has well developed tailored Monitoring, Evaluation, Accountability & Learning (MEAL) frameworks for all of its programs. CARE intends to conduct an independent (external) evaluation of Phases 1 and 2 of the DEC COVID-19 project to measure “impact”, ensure external, third party analysis for greater accountability of the response – both to the donor and to the affected communities, as well as to consider the relevance, appropriateness and effectiveness of the project at the community level, the added value of CARE’s relationship with local partners (in Phase 1), and to document lessons learned for improved COVID-19 response and WASH programming in the future. To achieve this, CARE will contract an independent organization with a well-developed network within NES, access to project implementation areas, and the capacity to reach identified beneficiaries and other stakeholders in Deir-Ez-Zor and Hassakeh governorates). The organization should have a proven track record in independent evaluations and methodological rigor in order to measure the impact of CARE’s DEC COVID-19 project and develop a quality evaluation report.

1. Scope of Work

2.1. Purpose and Objectives

Purpose:

The purpose of this external evaluation is to assess the program design, implementation methods, and operational modalities based on the Core Humanitarian Standard (CHS), in order to improve the quality and accountability of future WASH and emergency response programming to affected populations. The evaluation will primarily examine the relevance, appropriateness, effectiveness, and timeliness of program activities in achieving the intended project outcomes. Additionally, this evaluation will provide clear analysis of considerations given to protection, gender, disability, safeguarding, diversity and inclusion, and environmental sustainability under the project.

Objectives:

1. Review the relevance, appropriateness effectiveness and timeliness of the project’s distributions, psychosocial services, and WASH interventions – this will cover both CARE and its implementing partner – with a particular focus on the use of monitoring and evaluation to analyze the changing context beneficiaries faced and the coordination and complementarity of the activities with those of other actors in the targeted areas. (CHS 1, 2 & 6)
2. Assess the appropriateness of targeting strategies, beneficiary selection processes, programmatic approaches, and identify factors that appear to have enhanced or detracted from the quality of project interventions. (CHS 1 and 2)
3. Assess the capacity of CARE and local partner PCSC’s programming to address potential or actual unintended negative effects in a timely and systematic manner, including in the areas of safety and security, dignity and rights, prevention of sexual exploitation and abuse, and gender. (CHS 3)
4. Assess the project’s accountability mechanisms, including CARE and PCSC’s focus on communication, participation and feedback from the affected population and the use of complaints and feedback mechanisms throughout the program cycle. (CHS 4 and 5)
5. Provide programmatic and operational recommendations to improve CARE’s ongoing and future programming both in NES and other applicable contexts.

2.2. Main Evaluation Review Questions

The evaluation will seek to address the following questions (subject to adaptation):

Table 1. Summary of Evaluation Questions and Existing/Needed Data to Analyze the Question(s)

COMPONENTS OF THE PROGRAM WE WOULD LIKE TO LEARN MORE ABOUT

QUESTIONS WE HAVE THAT WE WOULD LIKE ANSWERED

EXISTING DATA TO HELP ANALYZE THIS QUESTION

FURTHER DATA NEEDED

WHO SHOULD BE INVOLVED?

Relevance (CHS 1)

[OECD/DAC: The extent to which the objectives of a development intervention are consistent with beneficiaries’ requirements, country needs, global priorities, and partner’ and donor’s policies.]

• To what extent have the project’s activities met the needs of the targeted population? And how?

Post-distribution monitoring report

The perspective of targeted beneficiaries on whether the project’s activities met their needs. How and why.

Beneficiaries, Partner staff, Project staff

Effectiveness (CHS 2 & 6)

[OECD/DAC: The extent to which the development intervention’s objectives were achieved.]

• To what extent were the project objectives achieved?
• What factors contributed to the achievement or non-achievement of the project objectives/performance indicators?
• Has the project been effective in improving access to protection services, WASH services, and WASH NFIs for IDPs and host communities in Northeast Syria?

Updated MEAL plan, post-distribution monitoring report, project database

• Project’s achievement against indicators and to determine the reasons for the results.
• Perspective of targeted respondents on the effectiveness of project’s approaches.

Beneficiaries, Partner staff, Project staff

Efficiency (CHS 9)

[OECD/DAC: A measure of how economically resources/ inputs (funds, expertise, time, etc.) are converted to results.]

• To what extent were the project management approaches, structures and use of resources (e.g., CARE’ partnership with PCSC, budgeted costs) effective in meeting project objectives and/or addressing project-related concerns?

Experience of similar projects, the budget comparative report, detailed implementation plan (DIP)

Qualitative information

Partner staff, Project staff, Head of Office

Impact (CHS 3)

[OECD/DAC: The positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended.]

• Has the project achieved its planned impact? Why or why not? Did the project have any unintended effects, either positive or negative?

Post-distribution monitoring report

Changes witnessed by the targeted respondents in the project. The perspective of targeted beneficiaries and project staff on any unintended project’s impact

Beneficiaries, Partner staff, Project staff

Accountability (CHS 4 & 5)

[Good Enough Guide: ‘Accountability’ is all about how an organization balances the needs of different groups in its decision-making and activities.]

• To what extent were beneficiaries involved in the project (project design, implementation, monitoring)?
• Were beneficiaries able to provide feedback? How was it used to promote accountability to the community and improve programmatic activities?

Post-distribution monitoring report, feedback log

Qualitative information

Beneficiaries, Partner staff, Project staff

2.3. Methodology:

The evaluation team will outline the appropriate evaluation methodology in their proposal and further develop it in the inception report. However, as a good practice, the evaluation team needs to design a mix of qualitative (using participatory method wherever feasible) and quantitative tools to gather information based on key evaluation questions. This includes targeted households/individuals for quantitative data collection, and community leaders and staff (CARE and partner) for quantitative data collection. Focus group discussions (FGDs) and key informant interviews (KIIs) should be used to further assess the quantitative response and key evaluation questions. A survey with direct beneficiaries along with discussions will help understand the spillover effect of the project interventions as envisaged in the project proposal.

The evaluation team will be responsible for defining and carrying out the overall evaluation approach. This will include specification of the techniques for data collection and analysis, structured field visits and interactions with beneficiaries, CARE/partner staff and other stakeholders, and composition of the evaluation team. Evaluation tools, methodology and findings should be reviewed and validated with various stakeholders and approved by the CARE NES MEAL Manager.

Table 2 below outlines the detailed data collection matrix for the evaluation (to be further discussed and finalized with the evaluation team). The evaluation team can further improve the sample size based on technical expertise and the COVID-19 situation on the ground when the evaluation takes place.

For the activities complete in phase one, no data collection requires as the data source will be from MEAL system (PDMs, distribution monitoring, etc.)

The selected evaluator will be expected to submit a work plan with final tools and forms (preferably in KOBO), to be agreed with CARE.

The selected vendors should take into consideration those point when collecting data:

a) Protection: all BNFs data should be protected and safe and not collected any sensitive data if it’s not necessary.

b) Disability: data to be collected from people with disapplied.

c) diversity and inclusion: data collected from different group of people including male and female.

e) environmental sustainability: avoid using papier in data collection and used Kobo.

Table 2. Detailed Data Collection Matrix

Key Activity to Monitor

Target Beneficiaries

Locations

Proposed Method

Proposed Sample

Water treatment stations and handwashing stations

6,000 HHs

Hole, Shadadah and Abdan

HH visits

375 visits

FGDs

6

KIIs

6

COVID-19 Hygiene Kits

4,431 HHs

Washokani and Abu Khashab camp

HH visits

354 visits

FGDs

6

KIIs

6

Water rehabilitation and maintenance

12,484 HHs

Abdan, Hole

HH visits

373 visits

FGDs

6

KIIs

6

2.4. Intended Users/Audience:

The evaluation findings will be used by CARE’s Senior Management, MEAL and WASH and emergency program teams to integrate recommendations for improvement into ongoing and future projects. The findings will also be shared with other relevant stakeholders. The following table outlines the expected communications to be produced based on the evaluation findings, the purpose of the communications, as well as the intended users.

Table 3. Communication and Reporting Plan

Communication Format

Purpose of Communication

User

Person Responsible

Timing/Dates

Presentation

Present preliminary findings

CARE Syria staff, internal stakeholders in NES (relevant local government depts)

Evaluator

TBD based on final evaluator chosen

Presentation

Present completed/final findings

CARE Syria and DEC staff

Evaluator

TBD based on final evaluator chosen

Report

Document the evaluation and its findings

CARE Syria and DEC staff, NES WASH and Shelter/NFI Working Groups, NES Forum, DEC members and readers in public domain, CARE staff via online evaluation library

Evaluator, after review by CARE MEAL Manager WASH Manager and Emergency Manager

TBD based on final evaluator chosen

Email

Document actions taken because of the evaluation

CARE Syria and DEC staff

CARE NES Area Director

TBD based on final evaluator chosen

2.5. Final Report Requirements:

The evaluation report will consist of a maximum of 35 pages, excluding cover page, acronyms page, table of contents page and appendices, with the following outline:

• A Title: A title that conveys the purpose of the report, the name of the project, location, implementation period, as well as the main impact or key finding of the report.
• An executive summary that focuses on impact in terms of quality and accountability that is no more than 2 pages in length and is formatted so that it can be printed as a standalone 2-pager.
• A description of impact early in the report, including 3-5 key impacts/findings: What changed because of the program? What happened in the project areas and in the lives of targeted beneficiaries, and why did it matter? What are the most significant achievements, supported by solid evidence? Each area of impact should be written as one or two sentences. Talk about impact early in the report so that the audience does not have to read the entire report before seeing evidence of change.
• A clear methodology section: The methodology should include the evaluation questions and explain how the methodology was selected to appropriately answer those questions. It should also contain key ethical considerations and a description of how the evaluators protected participants and personally identifiable information.
• 3-5 key lessons learned: These should be short, actionable, and the most important aspects of what the program evaluation found. They need to be relevant and new for people outside of the direct program. They should also include highlights on what to improve in the future.
• 3-5 bullets describing how the project made an impact/3-5 recommendations for CARE and local partners, DEC, and other stakeholders inside and outside of Syria, depending on the scope of the findings: It is important to have non-jargon descriptions of what a project did to make an impact. These are highlights of the most effective, relevant, and scalable approaches and tools.
• Shareable evidence: Evidence collected by the evaluation team supporting the evaluation findings and recommendations must be submitted along with the final report. All datasets, qualitative interviews, and underlying data are owned by CARE and are included in the final deliverables. Sources of all evidence must be identified, and conclusions must be based only on evidence presented in the report, and recommendations must directly correspond to the conclusion.

2.6. Data Disclosure:

The external evaluation team should deliver all files, including (at a minimum): quantitative datasets (raw and refined products), transcripts of qualitative data and other data in an easy-to-read format, and maintain naming conventions and labelling for the use of the files by all authorized stakeholders.

All documents should be compliant with the following conditions:

• CARE requires that the datasets that are compiled or used in the process of external evaluation are submitted to CARE when the evaluation is completed.
• Data must be disaggregated by gender, age, disability, and other relevant diversity.
• Datasets must be anonymized with all identifying information removed. Each individual or household should be assigned a unique identifier. Datasets which have been anonymized will be accompanied by a password protected identifier key document to ensure that users are able to return to households or individuals for follow up. Stakeholders with access to this document will be limited and identified in collaboration with CARE during the evaluation inception.
• CARE must be provided with a final template of any surveys, interview guides, or other materials used during data collection. Questions within surveys should be assigned numbers and these should be consistent with variable labelling within final datasets.
• Formats for transcripts (for example: summary; notes and quotes; or full transcript) should be defined in collaboration between CARE and the external evaluator at the evaluation inception
• In the case of tabular datasets variable names and variable labels should be clear and indicative of the data that sits under them. Additionally, the labelling convention must be internally consistent, and a full codebook/data dictionary must be provided.
• All temporary or dummy variables created for the purposes of analysis must be removed from the dataset before submission. All output files including calculations, and formulae used in analysis will be provided along with any Syntax developed for the purposes of cleaning.
• We require that datasets are submitted in one of our acceptable format types
• CARE must be informed of and approve the intended format to be delivered at evaluation inception phase. Should this need to be altered during the project CARE will be notified and approval will be needed for the new format.

3.Ethical Considerations

The precarious security situation, COVID-19 risks and access challenges should not hinder the full consideration of ethical principles of this evaluation. Therefore, it is the responsibility of the contracted evaluator to ensure that all respondents and stakeholders participating in evaluation activities do so voluntarily. This can be assured through written or verbal informed consent from participants before commencing the interviews as well as at the end of the interviews. Prior to the commencement of the assignment, the contractor will sign CARE’s ethics policy and abide by those standards throughout the contract duration.

The safety and security of the respondents and on-the-ground enumerators should be always considered, particularly during data collection. Names or details of respondents must be confidential and protected. Do No Harm principles should be respected during and following the implementation of the evaluation. Permission to use any visuals or personal details must be obtained, and the contractor is responsible for anonymizing data collected to protect the identities of the individuals interviewed. All staff and volunteers involved in evaluation activities are responsible for reporting incidents of Sexual Exploitation and Abuse.

4. Key deliverables and activities

The successful applicant will undertake the following tasks:

1. Conduct meetings with key project staff of CARE, partners and/or other stakeholders.
2. Design/finalize the evaluation plan, including data collection tools for the project and relevant program indicators (including translation into Arabic) and sampling technique and size.
3. Conduct desk reviews of secondary information and project documents including the project proposal (including logframe), implementation plan and M&E plan, project budget and financing agreement, progress and financial reports, existing data collection and monitoring tools, and any other relevant documents.
4. Obtain feedback on data collection tools from key CARE staff and finalize draft data collection tools to be tested.
5. Train enumerators who will pre-test the data collection tools. If necessary, make final adjustments to data collection tools in consultation with CARE.
6. Collect data from a representative sample of individuals from the target groups and key project relevant stakeholders using household questionnaires, KIIs and FGDs.
7. Data processing (data entry, verification, and analysis).
8. Report writing and submission of first draft report.
9. Presentation of findings and recommendations to and validation by key stakeholders.
10. Finalize the report incorporating feedback and submission of final report.

There should be adequate women’s representation and participation throughout the data collection process. Where necessary, especially in rural areas, FGDs should be conducted separately for men and women.

5. Proposal details, selection criteria and submission

Parties are requested to submit the following documentation to CARE International in Northeast Syria (address below) no later than 15th July-2022:

• Technical proposal, including proposed methodology and approach and detailed plan of action for field work indicating staff-days require (maximum 8 pages)
• Short biographies of key personnel on the proposed evaluation team, including specific roles and responsibilities of the Team Leader, supervisory chain, and other core members of the evaluation team (maximum 4 pages)
• Schedule of key activities preferably in a format such as a Gantt chart
• Detailed cost proposal, with costs broken down to reflect evaluation scope and administrative costs
• Updated CV of Team Leader and other core members of the evaluation team
• A profile of the consulting firm (including a sample report if possible)

Applications will be assessed against the following selection criteria:

• Relevant skills and previous research, monitoring, and evaluation experience in Syria (specifically Northeast Syria will be an added advantage)
• Proven track record in independent research and methodological rigor
• Appropriateness of proposed methodology and its correspondence to these Terms of Reference
• Attention given to gender considerations and the specific needs of women, men, boys, and girls
• Proven access to project implementation areas and linkages with local Syrian actors
• Value for money and demonstrated cost-effectiveness
• Expertise regarding (monitoring/evaluating) protection outreach, protection case management and basic needs support projects. Familiarity with Prevention of Sexual Exploitation and Abuse related issues
• Familiarity with the principle of Accountability to Affected Populations and the CHS commitments, including experience evaluating complaints and feedback mechanisms.
• The data collection team should be gender balanced and ensure same-sex interviews at all times

Annex A – CHS Table

CHS commitment

Proposed draft question

Supporting notes

CHS 1 – Is humanitarian response appropriate and relevant?

How has the project design (outcomes, activities and indicators) been relevant to addressing underlying causes of the identified problems?

CHS 2 – Is humanitarian response effective and timely?

How has the project been effective in achieving its planned activities and outcomes? If not, why?

CHS 3 – Is humanitarian response strengthening local capacity and avoiding negative effects?

Has the project followed a conflict sensitivity approach so as not to create/trigger conflict due to the implementation of the project?

CHS 4 – Is humanitarian response based on communication, participation and feedback?

Have the approaches/modalities used in the project been effective in engaging with the communities affected by the crisis and ensuring they participate throughout the project cycle? Is their feedback collected? How and why?

CHS 5 – Are complaints welcomed and addressed?

Review and assess the quality of the project feedback and complaints mechanisms? How many complaints addressed for total received?

CHS 6 – Is humanitarian response coordinated and complementary?

Review and assess the coordination mechanisms with other partners and relevant stakeholders. How many stakeholders involve in coordination to implement the activities

How to apply

The interested applicants should send their financial and technical offers to this email [email protected], as the deadline for submission offers will be June 28th, 2022.