Project Evaluation – Empowering Persons with Disabilities (PWDs) through Education and Income Generation in Amman, Jordan

Terms of References (ToR) for Project Evaluation

Introduction

The Austrian Development Agency (ADA)-co-funded “Empowering PWDs through Education and Income Generation in Amman, Jordan” project is a collaboration between DAA, IOCC, and IFH.

DAA is a member of Diakonie Österreich, the umbrella organization of all Diakonie organizations in Austria. DAA was founded in 2009 as successor of the department for development cooperation at the Evangelisches Hilfswerk in Österreich. The organization’s aim is to contribute to the international process of peace, justice, and protection of the environment, primarily through support of projects in developing countries and humanitarian aid. DAA runs two brands: Diakonie Katastrophenhilfe for humanitarian aid and Bread for the World for development assistance.

IOCC was founded in 1992 as the official humanitarian organization of the Assembly of Canonical Orthodox Bishops of the U.S. and as a way to engage the philanthropic work of the Orthodox Church on a global scale. IOCC responds to humanitarian emergencies, as well as makes long-term commitments to communities through development programs. Operating in Jordan since 2005, IOCC has implemented long-term development and resilience activities and humanitarian programs in response to the needs of refugees and vulnerable host community members. IOCC is currently implementing activities for persons with impairments and disabilities, along with caregivers, in the health, protection, education, and livelihood sectors, as well as basic needs and shelter programming. IOCC’s headquarters is located in Baltimore, Maryland, USA, with a country office for Jordan in the capital city of Amman.

IFH was established in 1986 as a national model for primary healthcare services for mothers and children. IFH was the first health center in Jordan to provide comprehensive training for medical professionals, which included a focus on early detection and intervention for children with disabilities. Since 2002, IFH has expanded its services to address gaps in family health needs. IFH has programs in family health, focused on a wide range of primary and secondary medical services, such as family medicine, pediatric care, orthopedics, ophthalmology, and physiotherapy; reproductive health, including ante-natal and post-natal care, family planning, cancer screening, and menopausal care; gender-based violence; mental and psychosocial health through which IFH provides basic and advanced psychosocial counseling services, focusing on behavioral and psychological disorders, as well as treatment for those suffering from war-related trauma, torture, and organized violence; child protection services; and capacity building for community-based organizations, government, and non-governmental organizations. IFH’s headquarters is in Amman with clinics across Jordan.

Background

With funding from ADA –- and in partnership with DAA –- IOCC Jordan and IFH are implementing a project entitled, “Empowering PWDs through Education and Income Generation in Amman, Jordan.” The overall objective of the project is: “Refugees and marginalized Jordanians with disabilities are empowered and better equipped to access education and livelihood opportunities through provision of assistance that meets their specific needs and improved access to services regardless of disability or gender.” The project began on July 1, 2019, and will end on April 30, 2022. A community-based approach is utilized to holistically address the needs of PWDs through four key results:

1. Children with disabilities (CWDs) and PWDs are identified and assisted according to their needs

IOCC and IFH utilize their expertise to expand screening and assessment of potential impairments and disabilities –- among both children and adults –- in the targeted area of the project (East Amman). The project utilizes a community-based approach to build networks and trust within communities that enables IOCC and IFH to improve identification of disabilities among refugees and vulnerable Jordanians (taking into account movement and other restrictions in place due to COVID-19). Trained community-based rehabilitation workers (CBRWs) conduct assessments of potential impairments and disabilities, and disability specialists from IFH provide treatment (i.e., assistive devices, such as hearing aids and glasses; specialized medical treatment; and/or surgeries).

2. CWDs and PWDs are supported to participate in society through rehabilitation and PSS sessions

CWDs and PWDs in Jordan face limitations to participating in society due to challenges accessing educational and healthcare services, and caregivers’ and community members’ lack of awareness and knowledge on how address the needs of their PWDs or CWDs. To improve PWD and CWDs’ participation in society, IOCC and IFH hold awareness sessions for community and family members, workshops for teachers, and rehabilitation and PSS sessions for CWDs, PWDs, and/or caregivers.

3. PWDs, with a focus on women with disabilities (WWDs), women with impairments, and/or female caregivers, have equal opportunities (regardless of gender or disability) to livelihood training and support

While PWDs, especially WWDs, face a number of challenges accessing the labor market in Jordan, IOCC is equipping WWDs and female caregivers of PWDs with skills that are in demand in the labor market through the provision of employability skills training, enrolment in vocational training courses, business development training, job coaching, and/or small grants.

4. PWDs have increased household income

Due to a number of barriers, PWDs are not fully integrated in Jordan’s workforce. However, following internships, a job fair, and awareness raising, PWDs are better equipped to meet employers’ needs and make valuable connections with employers and strengthen their networks. In addition, IOCC is working directly with employers to raise awareness on the importance of hiring and retaining PWDs – both as it relates to laws and regulations in Jordan and as PWDs are an integral part of Jordan’s society and workforce.

Purpose

The purpose of the evaluation – which is as mandated by ADA and to meet project partners’ accountability requirements – is to provide an assessment of overall project progress at the conclusion of the project’s second phase, review results toward the project’s Phase II objectives and indicators, and provide conclusions, recommendations, and lessons learned for the final, consolidation phase of the project.

Objectives

The objectives of the evaluation are to assess project objectives and results, thus far, and provide actionable recommendations that will ensure sustainability of these results and solidify gains among beneficiaries through the project’s final phase.

• To determine the extent to which targeted Syrian refugees and marginalized Jordanians with impairments and disabilities have been supported and enabled to participate in education and livelihood opportunities (taking into consideration the planned consolidation phase)
• To identify recommendations, which will be utilized for the final, consolidation phase of the project, with a particular focus on solidifying the impact of project activities thus far

Scope of the Assignment

Stakeholders to participate in and benefit from the assignment:

• DAA, IOCC, and IFH project and organization staff
• Representatives from Ministry of Health, Ministry of Education, Ministry of Labor, Ministry of Social Development, and the Higher Council for the Rights of Persons with Disabilities
• Project beneficiaries and surrounding communities
• Other humanitarian and development actors in Jordan (i.e., as reached through UNHCR-led working groups and task forces)

Geographical scope of the assignment:

• Evaluation of project activities: Implementation areas of East Amman

Target Audience of the Assignment

• DAA, IOCC, and IFH project staff and organizational leadership
• ADA program staff

Assignment Approach & Methods

The assignment should include:

Desk Study:

• Review the project’s history, location, and implementation context;
• Review project documentation, such as the proposal, monitoring/site visit reports, and project reports;
• Review publicly available similar project reports and documents on impairments and disabilities –- particularly pertaining to livelihoods, health, and education access –- in response to the Syria crisis in Jordan by UN, NGOs, and other local and international organizations in Jordan

Inception Phase:

• Provide an inception report (including a description of the evaluation; a preliminary desk review summary; an evaluation matrix; a stakeholder mapping; and a workplan)
• The field phase will only take place upon the official approval of the inception report

Field Phase

• Including involvement of project beneficiaries and other key stakeholders
• Data gathering and triangulation: it is expected that the evaluation utilize mixed methods, including quantitative and qualitative data. Various methods of data collection, such as analysis of documents, structured interviews, semi-structured interviews face-to face or by remote modalities, group discussions, surveys (online or in-person) (if applicable), and other methods shall be used.

Data review and analysis

Report:

• The document should include:
  • Presentation of the evaluation with key findings;
  • Presentation of recommendations
  • Completion of ADA “Results-Assessment Form for Mid-Term and Final Project Evaluations/Reviews” (see page 53 of the ADA Guidelines for Project and Programme Evaluations via link below)
• All data must be disaggregated by sex, age, and disability;
• Submission and presentation of the draft report to DAA, IOCC, and IFH
• Period of comments and feedback from DAA, IOCC, and IFH
• Submission final draft report, inclusive of comments from all partner organizations (DAA, IOCC, IFH) for final review
• Submission of final report
• Presentation of final report (via PowerPoint presentation)

The Guidelines for Project and Programme Evaluations developed by ADA must be considered throughout the entire evaluation process. (Please see: https://www.entwicklung.at/fileadmin/user_upload/Dokumente/Evaluierung/Evaluierungs_Leitfaeden/Guidelines_for_Programme_and_Project_Evaluations_ADA_2020.pdf.)

Evaluation Criteria

The evaluation will follow ADA and OECD/DAC norms and standards, as well as ethical guidelines for evaluations. The evaluation should be able to answer the following questions on the project and activities implemented during the project period:

OECD/DAC Evaluation Questions

Relevance

• To what extent and how did the project address needs identified during design?
  • Which planned activities were particularly relevant to the target group?
  • Have these needs changed, and, if so, how can they be addressed in the final consolidation phase?
• To what extent are the objectives of the project still valid for the context, partner organizations, and targeted populations? In cases in which they are not, what modifications can be made to ensure relevance for future activities?

Effectiveness

• To what extent has the project achieved expected results?
  • If results were not achieved, why and what were the key factors that influenced this? If they have been, what were key drivers for success?

Impact

• How and in what ways has the project impacted targeted communities and the lives of targeted populations?
• How was the project affected by the intersection and interaction of gender and disability issues among the targeted populations and in the targeted communities?
• Which positive and/or negative impacts in terms of gender and environment can be possibly attributed to the project?

Sustainability

• What can be done in the final consolidation phase to ensure the sustainability of project benefits, results, and impacts and to increase the larger, overall impact of the program (Phase I, Phase II, and Consolidation Phase)?

Consultant/Consultancy Team

To achieve the objectives of this evaluation, the consultant or team should be an experienced individual or professional firm with the following skills:

• A strong knowledge of humanitarian and development issues and the socio-economic and political context of Jordan and the impact of the Syria crisis.
• Experience with European-donor-funded projects and evaluations.
• Has conducted at least three evaluations in the last five years ideally in the relevant field.
• Experience on projects and evaluations targeting PWDs, experience specifically on projects targeting PWDs with hearing or visual impairments, a plus.
• Must be able to articulate evaluation methods to elicit feedback from people with impairments and disabilities (particularly hearing and visual).
• Experience on livelihoods, health, and/or education projects.
• Experience in social science methods.
• Experience in project cycle management and preparing and analysing a theory of change.
• Culturally sensitive.
• Ability and willingness to conduct the work in Jordan, including in the field.
• Ability to work within constraints that may arise due to COVID-19 (including experience with utilizing remote modalities, if necessary).
• Excellent communication skills, fluent in English with a good command of Arabic.
• Excellent English writing skills.
• In case of team/group applications, a gender-balanced and diverse team is required.
• Evaluation management must respect ethical standards and guiding principles for evaluations, including impartiality and independence.

Note: Evaluation management must not have been involved in the design or implementation of the project being evaluated.

Period and Timeline

The consultant(s) is expected to complete the assignment within approximately 35 working days, including up to 13 days of field work (May 8 – 24).

Work may begin on April 4, 2022 (with the evaluation kick-off meeting and may conclude in July 2022 (with the final report and presentation), depending on the agreed upon dates for the field work (target timing between May 8 and 24). An initial summative inception report (approximately 10 – 20 pages without annexes, including an evaluation matrix – template to be provided) should be submitted for discussion on April 10, 2022. A precise timetable (by days) will be set by the evaluator(s) in coordination with DAA and IOCC, according to plans made and finalized with the inception report.

For the inception report, structure and guidelines will be provided, in line with ADA Guidelines for Programme and Project Evaluations.

A draft of the final evaluation report [no less than 25 and no more than 40 pages, including a two-to-three-page executive summary with key findings and recommendations and excluding annexes and the results-assessment form (Annex 1)] will be submitted to IOCC and DAA no later than June 12, 2022. Findings and recommendations must be structured according to the evaluation questions. An outline of the final report structure should be agreed upon during the inception phase. At least the following must be included: cover page; result assessment form (as an annex); executive summary (should be included in the report and also as an independent document); background and objectives of the assignment; methodology (with mention of limitations, if any); summary of findings; analysis of findings addressing guiding questions; conclusions, lessons learned, and recommendations; and agenda of all activities carried out (annex).

For the evaluation report, structure and guidelines will be provided, in line with ADA Guidelines for Programme and Project Evaluations.

DAA and IOCC will coordinate with the evaluator on the review process, which should include a presentation of initial findings (following the field work and processing and analysis of the data), review and comments from project partners on the draft, revisions (as needed) by the evaluator, and review and comments from ADA on an updated draft, revisions (as needed) by the evaluator, and then final review of the report by DAA, IOCC, IFH, and ADA. The final report with all annexes must be submitted by July 11, 2022, and will be accepted, pending ADA’s final sign-off on the report by July 15, 2022. If ADA is still not satisfied with the report, additional revisions could be required. A presentation (including PowerPoint overview of the evaluation report) should be planned for DAA, IOCC, IFH, and other stakeholders on July 17, 2022.

All documents should be submitted in English. If an Arabic version (i.e., of survey questions) is provided, an accompanying version in English must be provided.