Terms of Reference Endline Survey services for Waxbar Carurtaada (Educate Your Children) II Project

Terms of Reference

Endline Survey services for Waxbar Carurtaada (Educate Your Children) II Project

Job title

EYC II Final Evaluation Consultant

Location

Banaadir, Puntland, Galmudug, Hirshabelle, Jubaland and South-West states of Somalia

Reporting to

MEAL Manager who in turn will work closely with the Program Manager with support of CARE USA’s Director of Research Advocacy and Learning

Expected time frame

14th February – 25th March 2024

BACKGROUND INFORMATION

About CARE International

CARE is a leading humanitarian organization fighting global poverty and providing lifesaving assistance in emergencies. Its programs go beyond meeting basic needs at the onset of an emergency to helping affected communities recover and rebuild their lives long after the crisis ends. CARE has been assisting communities in Somalia since 1981. CARE works in partnership with the government, international NGOs, civil society, community leaders and local authorities in order to bring effective and lasting change to the most vulnerable communities. CARE currently works through three main programs: firstly, the Rural Women program which supports poor, rural women, and girls in addressing long term underlying causes of poverty and vulnerability as well as social, cultural, political, and economic obstacles towards positive change. We help women and girls improve their economic status, access education, and support them to play a greater role in local leadership and conflict resolution. Secondly, the Urban Youth program focuses on job creation and livelihood opportunities for poor youth through interventions such as secondary education, vocational training, small business development and microfinance. Thirdly, the Emergency program provides direct humanitarian relief to victims of drought and conflict in Puntland, Mogadishu, Lower Juba and Galmudug state. This project falls under the Urban Youth programme.

1. About ***Waxbar Carurtaada (***Educate your Children) II project

Operating in Somalia since 1981, CARE has an extensive track record in implementing successful education programs focusing on increasing access, improving learning outcomes and boosting completion rates for the most marginalized groups of the population, with a particular focus on girls’ education. CARE commits to support the development of the education system in Somalia, contributing to building inclusive, efficient, and effective systems that support the most marginalized to access school and succeed in their education.

In partnership with Educate A Child, a program of Education Above All Foundation, CARE, in a consortium with WARDI and GREDO, is implementing Waxbar Carurtaada II (Educate your Children II/ EYC II) project which aims to address obstacles to educational access for out-of-school children affected by instability, displacement, social exclusion, and poverty in Somalia. The project seeks to enrol 80,600[1] out of school children across six states/region (Galmudug, Hirshabelle, Jubaland, South West, Puntland, and Banaadir) of Somalia. Waxbar Carurtaada II Project seeks to strengthen local governance structures and community support around school safety and inclusive and gender-sensitive education, increasing enrolment rates among marginalized groups, including girls and the extremely poor. Through Accelerated Basic Education (ABE), the project provides viable options for learners unable to access education or to attend regularly because of seasonal and out-migration, domestic demands, and income-generating activities. The project works closely with the Federal Ministry of Education, Culture, and Higher Education (MOECHE) and Federal member states Ministries of Education (MOEs) across the six states/region and other education stakeholders to achieve its objective.

1. Projects overall goal/objective

The overall project goal/objective is to ensure that 80,600[2] out-of-school children affected by instability, displacement, social exclusion, and poverty in Somalia have increased opportunities to enroll and complete a quality primary education. The project aims to achieve this goal/objective through the following strategies and interventions:

1. Outcome 1: 70,600[3] vulnerable OOSC in Banaadir, Puntland, Galmudug, Hirshabelle, Jubaland and South-West states of Somalia are enrolled and retained in formal primary education
2. Outcome 2: 10,000 overaged OOSC children in Banaadir, Puntland, Galmudug, Hirshabelle, Jubaland and South-West states of Somalia are enrolled and retained in alternative basic education
3. Outcome 3: Enhanced quality of education for grade-appropriate learning outcomes
4. Outcome 4: Community Education Committees and MOE staff are strengthened in capacity to improve the management of schools and ABE centers.
   1. Program scope

The project is implemented in 25 districts of six region/states. The project covers rural areas of Puntland (Galkacyo North, Galdogob and Jariban districts) and Galmudug (South Galkacyo, Dhusamareeb, Guriel, Abudwak, Cadaado and Hobyo districts), South West (Baidoa, Dinsoor, Hudur, Walanweyn and Afgoye districts), Hirshabelle (Beletweyn, Bula Barde, Mataban districts), Jubaland (Kismayu, Baardheere and Dollow districts) and Banaadir (Bondheere, Hodan, Waberi, Yaqshid and Deynile).

1. Overview of the project implementation timescales

Project start-date: April 25, 2021

Project end-date: March 2024

1.6 Project Beneficiaries

The project direct program beneficiaries include: (i) 57,600 out-of-school children from poor marginalized households who face barriers in accessing and attaining education; (ii) 145 Community Education Committees; (iii) 300 primary education teachers; (iv) an estimated 150 ABE teachers; (v) 25 education officers who will benefit from professional development to improve skills in planning, coordination, and monitoring. The project has enrolled 38,808 former OOSC (cohort 1 and 2) in formal primary school and 11,012 (cohort 1 and 2) in ABE. The project is in the process of finalizing verification of cohort 3 children enrolled in formal primary. Table 1 below gives a summary of the project’s Key Performance Indicators (KPIs) targets and achieved and reported to date.

Key Performance Indicators (KPIs)

Target

Achieved

OOSC newly enrolled in formal primary pathway

47,600

38,808[4]

OOSC newly enrolled in ABE pathway

10,000

11,012

Retention of OOSC newly enrolled (%)

80%

88.6%

Temporary Learning Sites (TLS) constructed

15

15

Schools rehabilitated

100

104

Formal primary teachers trained

300

231[5]

ABE facilitators trained

100

177

MoE staff trained

25

25

Teachers paid incentives

300

287[6]

ABE facilitators paid incentives

150

222

CONSULTANCY PURPOSE AND DESIGN

1. The rationale for the evaluation:

The Waxbar Carurtaada (Educate your Children) II project is in its third and final year. CARE seeks to contract an external consultant to conduct the endline evaluation in close collaboration with state level MOEs and Community Education Committees (CECs) in all target locations. The findings from the evaluation will be used:

• To determine the achievement of expected outcomes against benchmarks established at baseline and progress against targets.
• To identify factors affecting expected outcomes and if/how these map out against the outputs, thus testing its Theory of Change (ToC).
• To demonstrate accountability for the support received from EAC and EAA Foundation to the donor, MOECHE, state MOEs, and communities.
• By partners, stakeholders, and the Government to learn lessons from the project for the purpose of informing education programming and sector planning in country, particularly in relation to the MOECHE Compact’s priorities.
• By other donors, academic institutions, and education networks to inform the wider policy debate concerning access to education in Somalia, especially for marginalized children and girls.
  1. Objective of this consultancy

The program is seeking to procure the services of a consultant to conduct a mixed-methods, gender-sensitive evaluation that is inclusive of children with disabilities and other marginalized sub-groups within the targeted population. The evaluation will assess the design, theory of change of the programme), assess the implementation process of the programme and impact / results brought by it. The results of this evaluation are expected to provide a detailed measure of the degree to which stated objectives and results were achieved (or not) in the project. It should generate relevant findings, lessons learned and recommendations which will be shared with key stakeholders of the project and used by the implementing agencies to guide and inform on the best practices and strategies in for improving other future phases and education programs. It will also conduct an extensive exploratory analysis and review of secondary data to identify the factors affecting the project’s outcomes.

Specific objective: To provide information on the performance of the project against all indicators and the project’s overall relevance, effectiveness, efficiency, validity of design, sustainability, factors affecting performance, alternative strategies and its strengths and weaknesses.

TECHNICAL APPROACH AND METHODOLOGY

1. Evaluation design

The evaluation will use a mixed-methods, complexity-aware and inclusive approach. The quantitative component is expected to use a combination of primary data collected through individual-level (OOSC; school principals; CEC members; household surveys; ABE and primary school teachers) and institutional level (school and MoEs/REOs/DEOs) surveys; and secondary data obtained through EMIS and other sources (Education Sector Analysis; ESPIG and AGES baseline, midline and endline surveys; EMIS reports). The evaluator is expected to determine representative sample sizes for respondent groups, adequately powered to identify the expected level of impact on program outcomes, while also considering how modifications to the baseline sample size may affect the analysis. Quantitative data will be triangulated with qualitative data obtained from a variety of stakeholder groups through key informant interviews and focus group discussions. The evaluator will also conduct an extensive literature review. The project will provide with relevant project documents (proposal, logframe, baseline report, and various projects implementation reports) as well as relevant MOECHE/MOEs documents to contribute to the desk review.

The following methods will be used:

• Household survey with parents / caregivers which includes questions related to the household’s socio-economic background; a module on disability; questions on education practices etc
• School-level tools – A school survey will be conducted, including an assessment of school conditions; teacher data; aggregated school enrolment and attendance; classroom observations using structured timed tools. The school survey will also include tracking of all the 59,763 students who have been enrolled by the project in 170 schools to establish their status.
• Qualitative data collected from:
  • Parents (focus groups, conducted separately with mothers and fathers);
  • Teachers (focus groups);
  • CECs (focus groups);
  • Religious leaders / authorities (in-depth interviews).

1. Sampling design

The consultant is expected to conduct the survey in the same locations sampled at the baseline to strengthen comparability of results across the two evaluation rounds. The consultant will use a cross-sectional approach to sample parents and teachers. The cross-sectional parent sample acknowledges the project’s limited capacity to support the additional costs of recontacting parents, particularly in the case of ABE students who may have transferred to primary school after completing L1. Tracking of enrolment/transition which is also part of the evaluation will not involve sampling as all the 170 schools in which 59,763 students have been enrolled will be assessed.

1. Ethical protocols

The evaluation approach must consider the safety of participants and especially children at all stages of the process. The bidder will need to demonstrate how they have considered the protection of children and vulnerable individuals through the different data collection stages, including recruitment and training of research staff, data collection and data analysis and report writing.

Bidders are required to set out their approach to ensuring complete compliance with CARE’s Safeguarding Policy. The evaluator will be responsible for designing and enforcing measures to ensure compliance with the above-mentioned policy and international good practice with regards to research ethics and protocols, particularly safeguarding of children, vulnerable groups (including people with disabilities) and those in fragile and conflict-affected states. Consideration should be given to:

• Administrative, technical, and physical safeguards to protect the confidentiality of those participating in research, including but not limited to a complete ban on photography and video records of respondents and anonymization of data records/ reports.
• Physical safeguards for those conducting research.
• Data protection and secure maintenance procedures for personal information obtained from the beneficiaries.
• Parental consent concerning data collection from children or collation of data about children.
• A do-no-harm approach for collecting data from women and vulnerable individuals.
• Age and ability appropriate assent processes based on reasonable assumptions about comprehension for the ages of children and the disabilities they intend to involve in the research; and age-appropriate participation of children, including in the development of data collection tools.
• Approaches that ensure equal participation from individuals from marginalized groups, including language; abilities; and adequate representation of women and minority groups.
  1. Risk and risk management

Risk management plan: The successful bidder must take all reasonable measures to mitigate any potential risk to the delivery of the required outputs for this evaluation. Therefore, bidders should submit a comprehensive risk management plan covering:

• The assumptions underpinning the successful completion of the proposals submitted and the anticipated challenges that might be faced.
• Estimates of the level of risk for each risk identified.
• Proposed contingency plans that the bidder will put in place to mitigate against any occurrence of each of the identified risk.
• Specific child protection risks and mitigating strategies, including reference to the child protection policy and procedures that will be in place; and
• Health and safety issues that may require a significant duty of care precautions.
  1. Data quality assurance

Quality assurance plan: bidders are required to submit a quality assurance plan that sets out the systems and processes for quality assuring the research process and deliverables from start to finish of the project. This plan should include the proposed approaches to:

• Piloting of all evaluation tools, particularly regarding modifications to the baseline tools (i.e. inclusion of new questions/ items).
• Training of enumerators in conducting the mixed-methods primary research, including in research ethics; adherence to sampling guidelines; consistency and completeness of tools; and qualitative probing.
• Use of electronic data collection tools whenever possible, allowing for overnight checks of data quality and timely resolution of issues.
• Logistical and management planning.
• Fieldwork protocols and data verification including back-checking and quality control by supervisors.
• Verification of data validity (considering the limited accessibility of locations).
• Data cleaning.
  1. Data Analysis

An in-depth approach to data analysis is required. The statistical analysis of the quantitative component is expected to identify factors influencing project outcomes, thus establishing relationships between different variables and the outcomes of interest, as well as validating or challenging assumptions about the current status of access and school structures in the regional states. It is expected that qualitative data will be analysed using emerging codes, thus allowing for identification of unanticipated factors/ patterns. Extensive triangulation of qualitative and quantitative data will be required for validation of results and in-depth understanding of the effects observed. Findings should be disaggregated at multiple levels, including but not limited to gender, disability, location/regional states, type of school, IDP/non-IDP condition and where relevant, most prevalent community livelihood (such as pastoralism, agro-pastoralism, agriculture).

4. EVALUATION SCOPE OF WORK

1. Professional Skills and Qualifications

Qualifications: Bidders are required to identify and provide CVs for all those proposed in the team, clearly stating their roles and responsibilities for this evaluation.

The proposed evaluation team should include the technical expertise and practical experience required to deliver the scope of work and evaluation outputs, in particular, with regards to:

• Data collection management: The team should include skills and expertise required to design, plan and conduct mixed-methods data collection, including, whenever possible, the use of electronic data collection. Enumerators should be able to collect data across the six region/states (Puntland, Galmudug, Hirshabelle, Jubaland, South West and Banaadir). Enumerator teams should include (i) male and female enumerators and (ii) individuals with adequate linguistic abilities to conduct interviews in the mother tongue of respondents.
• Relevant subject matter knowledge and experience: Knowledge and experience required on mixed methods data collection and analysis, including advanced statistical skills; researching with children, the education sector, disability, and gender to ensure that the data collection method is applied in a relevant and meaningful manner, given the aims and objectives of the project and the context in which it is being delivered.
• Primary research: Gender-sensitive management and implementation of primary quantitative and qualitative research in potentially challenging project environments, such as fragile and conflict-affected states – this could include the design and application of household surveys, in-depth interviews and focus groups etc.
• Country experience: the team must have the appropriate country knowledge /experience and language proficiency required to conduct the research required.
• Information management: extensive experience in data quality control and database management (including collating multiple large datasets for analysis).
• Ability to manage English to Somali and Somali to English translation with high quality and accuracy, including for transcription and translation of large volumes of qualitative data.
• Safety considerations: Ensuring the whole evaluation process adhere to best practice for research with children including the implementation of CARE’s Safeguarding Policy, and procedures to ensure the safety of participants. Note that all bidders are expected to be able to show that they have child protection and safeguarding policy in place to safeguard children that the research team would come into contact with through the evaluation activities. All bidders are expected to comply with CARE’s Guidelines for Responsible Data Management as well.
  1. Expected Tasks

The MEAL Manager, who is the principal point person for monitoring and evaluation for the Waxbar Carurtaada II / Educate Your Children II Project will be responsible for the day–to–day management of the evaluation. CARE USA’s Director of Research, Advocacy and Learning will provide technical assistance to tool development, data quality control and data analysis, and offer technical feedback on all deliverables. The consultant will receive technical supervision and guidance from the MEAL Manager and CARE USA’s Director of Research, Advocacy and Learning and will be accountable to CARE and the MoECHE for quality results and for timely delivery of these. The consultant will be required to conduct the following tasks:

• Submit an inception report inclusive of the proposed tools, sampling approach, detailed analysis framework, quality control measures and work plan.
• Submit a detailed work plan as part of the inception report and updated data collection plan following training of enumerators. The work plan should include plans/framework for data collection, data entry, data cleaning, transcription, and translation.
• Submit a comprehensive risk management plan highlighting all reasonable measures to mitigate any potential risk to the delivery of the required outputs for this evaluation.
• Submit a quality assurance plan that sets out the systems and processes for quality assuring the research process and deliverables from start to finish of the project.
• Translate data collection tools and training protocols from English to Somali. Note that back translation will be required for accuracy.
• Develop electronic versions of all relevant data collection tools.
• Pilot the tools in the respective target region/states, track changes and submit revised tools for approval.
• Take the lead in training research assistants, following guidelines and training plans outlined in the inception report, as well as quality control protocols and guidelines for researchers.
• Prepare all resources and related supplies for data collection, including mobile phones/ tablets with e-versions of the tools uploaded into a platform of choice.
• Undertake data collection in the field following the approach outlined in the approved inception report.
• Support in data collection related logistics, including but not limited to deployment of researchers to the target region/states, and supervisors; and where required coordination with CARE to access each area with approval from MoECHE and regional states.
• Upload datasets completed by the team daily for verification by CARE’s technical team.
• Conduct qualitative data entry and coding.
• Conduct all necessary data quality and ethical control measures.
• Analyse the data as per agreed upon data analysis framework.
• Provide a draft comprehensive technical report that also includes the status of all the 59,763 students enrolled by the project; a draft summary report; and a draft presentation of findings for review. Incorporate technical feedback received from CARE, MoECHE and relevant stakeholders and provide revised final versions of all three documents for final approval.
• Provide a complete set of physical and electronic data collection documents, filed, and organized as per guidelines provided by the CARE, upon completion of data collection and entry. This includes but is not limited to complete datasets (anonymized and non-anonymized versions), script and analysis files; datasets used for analysis; transcribed and translated versions of FGDs and KIIs; filled surveys, questionnaires, FGD/ interview notes, daily survey logs; voice files (for electronic qualitative data collection), pictures (noting that no pictures of respondents will be allowed under any circumstances, including during data collection or before/ after data collection), etc.
  1. Deliverables

Project deliverables: the main deliverables for this project are as follows:

• Inception report.
• Work plan outlining all tasks to be completed within the duration mentioned for this assignment, responsible persons, timeframe for completion of each task and resources required for each task
• Detailed data collection and analysis plan, a comprehensive risk management plan and a quality assurance plan as part of the inception report.
• Translations of survey instruments from English into Somali, to be verified through back-translation (conducted by the evaluator) and secondary spot-checks conducted by CARE.
• Training plan (including summaries of training sessions and relevant presentations/ exercises).
• Complete list of enumerators/supervisors per location, including contact details.
• Training report on the data collection process with recommendations for future data collections. The report should catalogue and include, in annexes, the final survey and training tools used (with translations where available).
• Complete set of files, hard copies of surveys, audio/ image/ video files and daily data collection reports/ data collection logs signed by enumerators and field supervisors.
• Complete transcriptions of qualitative data both in Somali and English.
• Clean data sets, to be reviewed and approved by CARE (data quality verified and accepted by CARE). 100% of the planned data is included and the final version of the database should not include any typos, out of range responses for any variable and/or illogical responses.
• Analysis and script files.
• Final report, based on the agreed-upon analysis framework. The methodological section should include a discussion of data collection challenges and limitations, including a summary of data collection activities and the timeline of data collection, a number of total surveys collected and the number of refusals to participate for each location, as well as detailed data collection log per region/state.
• Final summary report.
• PowerPoint presentation of findings.

Evaluation milestones: bidders are required to include in their detailed work plan, the number of days for each milestone set out below, and the deadline.

S/n

Milestones

Days

Deadline dates

1

Inception and briefing meetings

⁴

Desk review of project documents and reports

3

Development and submission of the inception report and data collection tools

4

Present the evaluation design, inception report and data collection tools

5

Training of research assistants and relevant MOECHE officials

6

Fieldwork (pre-test of tools, data collection and debriefing)

7

Data analysis, report writing and submission of the first draft report

8

Incorporation of comments into the evaluation report

9

Revision, finalization of the report, submission, and approval

The evaluator will be expected to identify a team leader for communication and reporting purposes. At the inception meeting, the team leader will be expected to submit a full contact list of all those involved in the evaluation.

The evaluator will be expected to report to the MEAL Manager. The design of the evaluation and, regular updates will be presented to the project’s steering committee. The team will be required to submit to the MEAL Manager weekly progress reports (by email) summarizing activities /tasks completed (per cent achieved), time spent etc. Any challenges are expected to be communicated immediately to the project’s MEL Manager prevent any delays.

CARE Somalia will facilitate coordination with state MoEs and the MOECHE on requesting approvals for the activity and seeking feedback on the design. The evaluator is expected to work closely with Regional Education Officers and District Education Officers during this assignment.

FINANCIAL PROPOSAL

The budget prepared by the bidder should cover all the activities outlined above, including training, data collection, data cleaning, transcription and translation, analysis, and reporting. This budget is inclusive of all costs covering team member costs, travel, research costs and any other costs associated with the completion of the work including where required costs for reasonable adjustment.

1. HOW TO APPLY

The deadline for submission of bids is 10-feb-2024. Interested bidders are welcome to request for the Project Logframe or submit questions until 04-feb-2024, Questions to this TOR should be submitted to [email protected], the bidders should send their applicants to [email protected] the Subject line Consultancy for the Endline Evaluation services for Waxbar Carurtaada II (Educate Your Children II) project.

Bids should include the following:

• Technical proposal in word or PDF covering bidders approach to the evaluation; workplan with milestones, how the bidder meets the qualifications, including CVs of all key involved persons etc
• Financial proposal in excel or PDF format covering all the costs
• Risk Management

How to apply

1. HOW TO APPLY

The deadline for submission of bids is 10-feb-2024. Interested bidders are welcome to request for the Project Logframe or submit questions until 04-feb-2024, Questions to this TOR should be submitted to [email protected], the bidders should send their applicants to [email protected] the Subject line Consultancy for the Endline Evaluation services for Waxbar Carurtaada II (Educate Your Children II) project.

Bids should include the following:

• Technical proposal in word or PDF covering bidders approach to the evaluation; workplan with milestones, how the bidder meets the qualifications, including CVs of all key involved persons etc
• Financial proposal in excel or PDF format covering all the costs.
• Risk Management

NOTE: please download the RFQ templet from the link below and fill where appropriate and share it back with all other requirement documents

RFP for EYC II Final Evaluation_.docx