Evaluator for for End-of-Project Evaluation – Jordan

Background Information

Project Title: “Access, Inclusion, & Empowerment: Support for Refugees and Host Community Members with Impairments & Disabilities in Jordan”

Implementer(s): International Orthodox Christian Charities (IOCC) and King Hussein Foundation (KHF)/Noor al Hussein Foundation Institute for Family Health (IFH)

Life of Strategy/Project/Activity: September 1, 2021, to August 31, 2023

Active Geographic Regions: Jordan: Sweileh, Amman; Russeifa and Zarqa city, Zarqa; Irbid city, Irbid, Karak; Karak city; and surrounding areas via mobile clinic

Sector(s): Health; Protection; Education; Persons with Disabilities

Project Summary

As of the project design in March 2022, 760,963 refugees were actively registered in Jordan – nearly 83% reside in the host community. Of refugees, 64% live below the poverty line[1]. The World Health Organization (WHO) estimates that the incidence of disability within any population is 15% and 20% in a humanitarian or crisis setting. Jordan’s 2015 census found disability prevalence at 13%, which is likely a conservative estimate.[2] UNHCR’s Vulnerability Assessment Framework (VAF) 2021/22 found that 34% of refugee families have at least one member with a disability. Due to inaccessibility, stigma, lack of transport, medical costs, and other challenges, PWDs are less likely to access basic services and engage in their communities, which increased due to the impacts of COVID-19, further contributing to marginalization,.

PWDs require sustainable, specialized services. Through disability programs for more than 50,000 people in Jordan, assessment of needs, and coordination with the Government of Jordan (GoJ),[3] IOCC confirmed gaps persist in local capacities to provide medical (including assistive devices), rehabilitation, education, and PSS services for PWDs and caregivers. Hosting a significant number of refugees the past 12 years, compounded by the impacts of COVID-19, has overburdened Jordan’s services and infrastructure, particularly inclusive healthcare and education. While the GoJ has made progress in developing the National Rehabilitation Strategic Plan (2020–2024) and 10-year Strategy of Inclusive Education, support is required in implementation. Additionally, overwhelmed clinics are unable to provide specialized services needed at the frequency required and the quality expected.

Finally, rehabilitation and PSS services are key to comprehensively addressing PWDs’ needs. For example, the Inter-Agency Standing Committee “Guidelines on Mental Health and PSS in Emergency Settings” notes PWDs are at an increased risk of PSS challenges in emergencies, and the Women’s Commission for Refugee Women and Children found children with disabilities are often excluded from PSS services, which are crucial, particularly due to additional stressors as a result of the COVID-19 pandemic.

IOCC’s two-year project, funded by the United States State Department Bureau of Population, Refugees, and Migration (PRM), responded to PWDs’ needs and filled gaps in services that were exacerbated by the pandemic. The project aimed to improve the quality of life of refugees and vulnerable Jordanians with hearing, visual, and intellectual impairments/ disabilities in Amman, Zarqa, Russeifa, Irbid, and Karak in Year 1 and with the addition of a mobile vision clinic to reach those residing in more remote areas in targeted governorates in Year 2.

Objective 1: Improve identification of impairments and disabilities and increase access to medical diagnosis and treatment for 3,868 refugees and 1,657 Jordanians with impairments and disabilities

This has been accomplished through the following activities:

• Identification of disabilities through screening and referral to project activities;
• Recruitment, training, and deployment of community-based rehabilitation workers (CBRWs) who perform outreach and act as a link between the project and communities;
• Diagnosis and treatment of hearing and visual impairments, including the fitting for and provision of assistive devices (glasses, hearing aids, and low vision devices) and, as necessary, repairs and maintenance to devices; referrals to other specialists, such as diagnosis specialist, physical medicine and rehabilitation specialist, ophthalmologist and/or ear, nose, and throat (ENT) doctor for specialized medical treatment and/or referral for surgeries.

Objective 2: Enhance inclusion of PWDs by equipping 3 local health clinics and building the capacity of 40 education institutions to offer inclusive services and increasing knowledge and awareness of disabilities at the community level among 2,730 refugees and 1,770 Jordanians

This has been accomplished through the following activities:

• Equipping of one vision clinic and one hearing clinic in Karak, in addition to a mobile visual clinic;
• Delivery of a two-day workshop to 600 school staff at 40 schools in communities surrounding each of the clinics to promote increased inclusive education;
• Community awareness-raising events and campaigns on disabilities.

Objective 3: Empower PWDs through provision of rehabilitation services, support for caregivers, and PSS services for PWDs and caregivers, reaching 2,380 refugees and 1,020 vulnerable Jordanians

This has been accomplished through the following activities:

• Rehabilitations sessions for 1,650 PWDs and/or their caregivers in speech therapy, occupational therapy, physical therapy, and/or special education;
• Workshops for 900 caregivers and support in the development of home-based care plans;
• PSS sessions for 775 caregivers and/or PWDs;
• Peer counseling sessions for 200 caregivers and/or PWDs.

Organizational Background

Working in Jordan since 2005, IOCC has implemented development and resilience activities for vulnerable Jordanians, as well as humanitarian programs in response to influxes of Iraqi and Syrian refugees. IOCC works in the host community and Azraq Refugee Camp in health and protection (focused on PWDs), livelihoods, and shelter programming with support from PRM, Austrian Development Agency (ADA), Diakonie ACT Austria, and IOCC private funds.

In Jordan, IOCC has a strong reputation for high quality disability programming among refugees and Jordanians in coordination with local actors. Since 2015, IOCC has scaled up its disability work, reaching more than 50,000 people. Through its disability programs, IOCC has trained and deployed 224 community-based rehabilitation workers; diagnosed more than 22,229 people and treated 18,473; conducted awareness raising activities for 16,276 people, peer counseling groups for 100, and self-help groups for 1,005; trained 1,181 caregivers and supported their development of home-based support plans; and provided rehabilitation sessions for 4,569 people and PSS sessions for 1,714 people.

Evaluation

Purpose of the Evaluation

This evaluation will assess the effectiveness, efficiency, coherence, relevance, impact, and sustainability of the project, “Access, Inclusion, & Empowerment: Support for Refugees and Host Community Members with Impairments & Disabilities in Jordan.”

The results of the evaluation are intended for IOCC and IFH’s use to inform future program design and implementation and to identify and transfer best practices and lessons learned to similar programs that IOCC and IFH implement. The results of the evaluation will be shared with the project donor (PRM) to show the impact of the project on the targeted population. As such, PRM, IOCC, and IFH will be particularly interested in the results and impact the project has had on beneficiaries, as well as its sustainability when the project comes to an end, ensuring that all benefits of the project for beneficiaries continue.

Evaluation Type

This will be an ex-post performance evaluation. The project term to be examined will be limited to the period between September 1, 2021, and August 31, 2023.

Evaluation Questions

• Effectiveness:

  • To what extent was the implementation of activities successful?
  • Did all of the intended target groups, including the most vulnerable, benefit equally from the project?
  • Were services provided in a gender equitable way?
  • To what extent did the intervention flexibly respond to changing circumstances?

• Efficiency:

  • Was the project implemented in the most efficient way (i.e. time, personnel, and resources)?
  • To what extent were financial and human resources converted into results?

• Coherence

  • To what extent were the project objectives consistent with refugees’ and vulnerable Jordanians’ needs?
  • To what extent did the project fill gaps in services for refugees and vulnerable Jordanians in the targeted areas?

• Impact

  • To what extent were the project objectives achieved?
  • How and in what ways did the project impact the lives of the targeted populations?
  • Which institutions have been influenced to become more inclusive by the project and how?
  • What has changed for whom (immediate impact) as a result of the project?
  • How has the project contributed to changing communities in the targeted areas for the better?

• Sustainability:

  • To what extent are the benefits of the project likely to continue after it ends?
  • Has the project achieved any positive or negative lasting effects on the targeted populations and stakeholders (i.e., teachers and/or principals, PWDs, caregivers, community members, etc.)?
  • To what extent was local partner IFH’s ability to provide services to PWDs strengthened and capacity built as an overall institution as a result of the project?

• Learning:

  • What were the major lessons learned from the project?
  • What other services could be added to future, similar projects?
  • What recommendations can IOCC and IFH include in future program design, implementation, and monitoring?

Evaluation Design and Methodology

The methodology will include review of project documentation, including project proposals, reports, monitoring and evaluation (M&E) plans, previous evaluations (i.e. midterm review), and desk monitoring review; development of an evaluation approach and data collection tools, including detailed timeline, work plan, and budget; field visits to project sites for data collection purposes, utilizing the most suitable tools agreed by the evaluator and IOCC; debrief summarizing preliminary findings; draft evaluation report; and a final evaluation report in English.

Limitations

Details and logistics of site visits, interviews, focus group discussions, etc. will be dependent on any occurring security risks or threats and mitigation measures initiated by GoJ, IOCC, IFH, and the evaluator’s risk assessment.

Deliverables and Reporting Requirements

Dates are tentative and open for discussion with evaluator.

• Evaluation Design/Final Inception Report: June 1
• Data collection: August 13 – 23
• Final Report: September 15

Evaluator Profile

The evaluation will be conducted by an external evaluator with the following qualifications:

• University degree in social sciences or relevant fields;
• At least five years of experience, with familiarity for standards of ethical conduct in research/evaluation settings;
• A strong knowledge of humanitarian and development issues and the socio-economic and political context of Jordan and the impact of the Syria crisis;
• Expertise in evaluation/research relating to PWDs, education, and health;
• Experience in project cycle management and preparing and analyzing a theory of change;
• Experience in conducting external evaluations in mass displacement settings, such as Jordan;
• Must be able to articulate evaluation methods to elicit feedback from people with impairments and disabilities (particularly hearing and visual);
• Demonstrated ability to handle sensitive information with discretion and professionalism;
• Sound and proven experience in conducting learning-focused evaluations with qualitative and quantitative data collection techniques;
• Ability to self-obtain a visa for Jordan and ability and willingness to conduct the work in Jordan, including in the field;
• Excellent English writing skills required;
• Experience evaluating projects funded by the U.S. Government is preferred;
• Fluency in both English and Arabic languages is preferred;
• Evaluation management must not have been involved in the design or implementation of the project being evaluated.

Period and Timeline

The evaluator(s) is expected to complete the assignment within approximately 20 to 25 working days, including up to 10 days of field work (between August 13 and 23).

Work may begin on May 3, 2023 (with the evaluation kick-off meeting) and should conclude in September 2023 (with the final report), depending on the agreed upon dates for the field work (target timing between August 13 and 23). An initial summative inception report (approximately 10 – 20 pages without annexes) should be submitted for discussion by June 1, 2023. A precise timetable (by days) will be set by the evaluator(s), in coordination with IOCC, according to plans made and finalized with the inception report.

The final evaluation report (no less than 35 and no more than 55 pages), including a two-page executive summary with key findings and recommendations and excluding annexes must be submitted to IOCC no later than September 15, 2023.

An outline of the final report structure will be agreed upon during the inception phase. At least the following must be included: cover page; executive summary (should be included in the report and also as an independent document); background and objectives of the assignment; methodology (with mention of limitations, if any); summary of findings; analysis of findings addressing guiding questions; gender analysis; conclusions, lessons learned, and recommendations; and agenda of all activities carried out (annex).

The quality of the report will be judged according to the following criteria:

• Does the report contain a comprehensive and clear executive summary?
• Was the ToR fulfilled and is this reflected in the report?
• Were all evaluation questions mentioned in the ToR answered?
• Are the methods and processes of the evaluation sufficiently documented in the evaluation report?
• Are cross-cutting issues analyzed in the report (i.e. gender)?
• Are the conclusions and recommendations based on findings and are they clearly stated in the report?
• Does the report clearly differentiate between conclusions, lessons learned, and recommendations?
• Are the recommendations realistic and is it clearly expressed to whom the recommendations are addressed?
• Were the most significant stakeholders involved consulted?
• Does the report present the information contained in a presentable and clearly arranged manner?
• Is the report free from spelling mistakes and unclear linguistic formulations?
• Is the report fully proofread and edited?
• Can the report be distributed in the delivered form?

[1] Poverty is calculated according to the international poverty line for higher middle-income countries at 3 JOD/day per a collaboration between UNHCR and the World Bank (https://www.unhcr.org/jo/18841-64-percent-of-refugees-in-jordan-survive-on-less-than-3-dinar-a-day.html).

[2]“Fafo: The Living Conditions of Syrian Refugees in Jordan – Results from the 2017-2018 Survey of Syrian Refugees inside and outside Camps.”

[3] Ministry of Health (MoH), Ministry of Social Development (MoSD), Ministry of Education (MoE), Ministry of Labor (MoL), and Higher Council for the Rights of Persons with Disabilities (HCD)