Second Midline Evaluation Consultancy Services for AGES project

Terms of Reference

Second Midline Evaluation Consultancy Services for CARE Somalia’s Adolescent Girls’ Education in Somalia Project, funded by UK’s Girls’ Education Challenge and USAID

1. SUMMARY

CARE Somalia (henceforth referred to as “CARE”) is seeking to procure the services of a local or international evaluation company to manage the second midline evaluation of Adolescent Girls’ Education in Somalia (AGES), an initiative funded by the United Kingdom’s (UK) Foreign, Commonwealth and Development Office (FCDO) through its Girls’ Education Challenge- Leave No Girl Behind (GEC/ LNGB) initiative and by USAID. The GEC is a global initiative focusing on ensuring the expansion of education opportunities for marginalized girls at primary and secondary level, using rigorous evaluation practices. The AGES project seeks to improve learning and transition outcomes for 82,975 severely marginalised girls in conflict-affected areas of Banaadir, Hirshabelle, Jubaland and South West states of Somalia. AGES supports formal, non-formal and accelerated education for some of the most affected out-of-school girls, including internally displaced populations (IDPs), girls with disabilities, minorities, married and divorced girls, adolescent mothers, and those in child labour. AGES has high demands in terms of the quantity and quality of data to be collected to generate robust evidence which can be used by multiple stakeholders.

This tender includes two components:

1. Data collection, including: Training of enumerators; data collection; quality assurance of fieldwork; data cleaning and collation; and regular reporting to CARE;
2. Evaluation design, data analysis and reporting, including: Modifications to baseline and 1st midline quantitative tools; design of qualitative tools; design of the analysis framework; technical guidance to the data collection company on data collection methodologies, in case the two components are awarded to separate companies; data analysis; reporting; and preparation of dissemination products.

CARE will accept bids inclusive of both components or only one of them.

The second midline evaluation will be an independent and rigorous study, which will enable the project to identify its effectiveness on achieving the expected outcomes and intermediate outcomes; assess to what extent the expected outcomes are being achieved by different subgroups of the population (equity); identify the extent to which different components of the intervention are contributing (or not) to the said outcomes and intermediate outcomes, testing the project’s Theory of Change (ToC); and assess delivery processes (efficiency and economy). The evaluation will ultimately assess if the AGES investment represents Value for Money (VfM) for the UK, US and Somali governments, while also identifying valuable lessons learned for girls’ education programming in fragile and conflict-affected contexts.

The second midline evaluation will use a pre-post design with a mixed methods approach. Quantitative data will be obtained from learning assessments conducted with two student cohorts (1 and 4) representative of treatment interventions; a household survey conducted with students’ families; a school survey; head counts; and classroom observations. Quantitative results will be triangulated with qualitative data obtained through focus group discussions, interviews, and participatory exercises with girls. The second midline evaluation will also include an analysis of contextual changes influencing outcomes, using secondary data collected by other development partners and monitoring data collected by the project during the course of the project. Findings from the contextual analysis will inform the interpretation of the second midline results and the preparation of recommendations based on those. The results of the second midline evaluation will be used to inform adaptations to project design and delivery modalities, seeking to maximise and equalise impact for all subgroups of girls targeted by the project.

This consultancy will also include a baseline evaluation for a representative sample of a third cohort of students enrolled by AGES in non-formal education classes. The baseline study will be conducted using a similar methodology to the second midline study and the findings will be included as a section of the second midline report.

2. BACKGROUND TO THE GEC PROGRAMME AND THE AGES PROJECT

2.1. GEC Programme Background

FCDO leads the UK’s work to end extreme poverty. FCDO is tackling the global challenges of our time including poverty and disease, mass migration, insecurity and conflict. FCDO’s work is building a safer, healthier, more prosperous world for people in developing countries and in the UK too.

FCDO is working to reach the Sustainable Development Goals (SDGs) by 2030. Progress on girls’ education is critical to the achievement of these targets. SDGs 4 and 5 specifically relate to education and achieving gender parity. SDG 4 specifically notes ‘inclusive and quality education for all and promote lifelong learning’.

USAID has joined efforts with FCDO’s Girls’ Education Challenge in its commitment to address barriers to the achievement of equitable education outcomes for all. In doing so, USAID contributes to the US Government’s Strategy on International Education, which commits American aid towards the achievement of SDG 4, expanding access to education for marginalised and vulnerable populations, such as children and youth affected by crises and conflict; improving learning outcomes and prepare girls for a productive future; and facilitating girls and women’s empowerment through gender-transformative educational opportunities. USAID’s partnership with FCDO’s GEC is aligned with the principles of its Education Policy, which prioritizes leveraging resources, promoting equity and inclusion, and focusing investments on improving learning and educational outcomes. In the specific case of AGES, it is expected that the programme will contribute to the expected outcomes of USAID Somalia’s Country Cooperation Development Strategy (CDCS) and USAID’s Youth in Development Policy.

Globally, 31 million primary age girls have never been to school[1]. The majority of these girls come from the poorest and most marginalised communities in the most disadvantaged locations, ethnic groups, etc.[2] Over the last 20 years, primary enrolments for girls have improved along with boys but completion rates are equally low for both sexes. At the secondary level, the differences between boys’ and girls’ participation rates really start to show. Significant disparities exist within countries, with the poorest girls from rural areas most severely subject to educational disadvantage – even at the primary level.[3] In the specific case of Somalia, such disparities are already observed in early primary grades, where the gross enrolment rate stands at 14%.[4] The GEC is helping the world’s poorest girls improve their lives through education and supporting better ways of getting girls in school and ensuring they receive a quality of education to transform their future.

PricewaterhouseCoopers LLP (PwC) and alliance partners have been contracted as the dedicated Fund Manager (FM) and is responsible for the day-to-day operation of the GEC. This includes establishing the recipient tendering process, supporting bidders, sifting and scoring proposals, monitoring Value for Money (VfM) and making project funding recommendations for FCDO approval. The FM also manages the relationships with the selected projects and oversees their Monitoring, Evaluation, and Learning operations.

Through the GEC, the UK government provided £355m between 2012 and 2017 to the FM to disburse to 37 individual projects across 18 countries across sub-Saharan Africa and South Asia to help girl’s education. In 2016 the GEC Transition window has been set up with additional UK funding to support the original GEC beneficiaries continue their journey through stages of education and further transit to upper grades and improve their learning[5]. In Somalia, the US government has contributed $15m to the GEC-funded AGES project to further expand its reach and impact.

CARE Somalia is implementing AGES, one of the GEC-funded projects. AGES’ intervention focuses on providing tailored, sustainable solutions to develop literacy, numeracy, and key life skills (financial literacy and knowledge of reproductive health) for 82,975 of the most marginalised girls and female youth in South Somalia, including girls with disabilities, those from minority groups, IDPs, married and divorced girls and adolescent mothers.

2.2. Project Background

Operating in Somalia since 1981, CARE currently works through three main programs: the first, CARE’s rural vulnerable women’s program, supports poor rural women and girls in addressing long term underlying causes of poverty and vulnerability, addressing social, economic, cultural and political obstacles to positive change. We help women and girls improve their economic status, access education and support them to play a greater role in local leadership and conflict resolution. Our second program area, the youth program, focuses on job creation and livelihood opportunities for poor youth through e.g. secondary education, vocational training, small business development, and microfinance. Thirdly, the emergency program provides direct humanitarian relief to victims of drought and conflict in Puntland, Mogadishu, and Lower Juba. CARE has its main offices in Hargeisa, Garowe, Mogadishu, and Erigavo, and satellite offices in Burao, El Afweyn, Kismayo and Bossaso. We work with a large number of local partners and maintain excellent relations with local governments.

Led by CARE in partnership with three local NGOs, AGES works with schools, communities, individual students, religious leaders, the Federal Ministry of Education, Culture and Higher Education (MOECHE) and State Ministries of Education (MoEs) in Hirshabelle, Jubaland and South-West, as well as the Banaadir Regional Administration to:

1. Increase the availability of flexible, quality learning opportunitiestailored to the needs of ultra-marginalised out-of-school girls:

• Accelerated basic education (ABE) including compressed primary content;
• Increased access to quality formal education through agreements with education providers and teacher training (reading, numeracy, child protection and gender-equitable practices);
• Non-formal education (NFE/life skills course) for older girls (ages 15-25), teaching literacy, numeracy, financial literacy, savings, and basic business planning.

1. Facilitate a social norm change movement towards broader life opportunities for adolescent girls and boys:

• Religious leaders trained on Islamic principles to support gender equity and education for girls;
• Community leaders and Community Education Committees (CECs) sensitised and mobilised to work with parents to identify and support extremely marginalised girls to enrol in learning opportunities and attend classes regularly;
• Mothers and mothers-in-law mobilised through village savings and loans associations (VSLAs) and enrolment drives;
• Girls trained on leadership skills, menstrual hygiene management, savings and forming peer support networks at Girls’ Empowerment Forums (GEFs);
• Boys mentored through Boys’ Empowerment Forums (BEFs);
• Messages reinforced via radio and social media.

1. Build institutional capacity: Support to strengthen monitoring of education services; develop decentralised management capacity; strengthen mainstreaming of inclusive approaches and safeguarding mechanisms; and increase technical capacity for planning and management of non-formal education services.
2. Facilitate the engagement of girls and female youth in collective action to claim rights through support to interventions led by GEFs and GEF networks.
3. Support increased access of female youth to other development opportunities, including employability skills through technical-vocational education and training (TVET); economic empowerment opportunities; and participatory accountability processes.
4. Increase girls and female youth’s access to critical services, such as legal services, governance institutions and safety nets through humanitarian support.

AGES is also providing the means for girls who are unable to attend school consistently to study remotely, through guided lessons and remote support; mobilising communities to follow up on dropouts and ensure their return to school; providing hygiene supplies to participants and schools; and supporting the implementation of psychosocial first aid through mentors, teachers, and peers.

The use of flexible learning tracks by the AGES project, complemented by remote education, allows girls engaged in labour, married and divorced girls, and pastoralist girls to attend learning sessions despite the limitations on their time. The work with religious leaders and Quranic teachers addresses the resistance to the inclusion of older and married girls and issues with their mobility and approval from family to attend classes, particularly in areas where recent influence from armed groups has resulted in severe curtailing of women’s freedom.

Physically disabled girls with limited mobility benefit from (1) access to ABE/ non-formal education courses close to home; (2) renovations to schools which include accessible features such as ramps; (3) access to remote learning allowing them to learn from home; (4) increased awareness of parents and community members of the importance of education for the disabled. Girls with special learning needs are benefitting from scholarships to attend special needs schools. Girls facing severe anxiety and depression are supported through the combination of psychosocial first aid and support networks through the GEFs.

Girls from minority clans and ethnic groups develop their self-confidence and aspirations and build connections beyond their own clans through participation in the GEFs. The participation in GEF networks will expand the engagement of the most marginalised adolescent girls and female youth in civic action. Furthermore, teachers have been trained to support girls who are lagging behind and first-generation learners, enhancing the likelihood of enrolment and retention in both ABE and formal education. Girls from displaced and ultra-poor families as well as orphaned children are overcoming financial barriers through the participation of parents, older adolescent girls, and female youth in VSL. Through the expansion of AGES activities funded by USAID, female youth also have the opportunity to further expand their economic empowerment through VSL networks and participation in skills training, as well as enhanced access to safety nets.

See Annex 1 for an overview of the project’s outcomes, intermediate outcomes, outputs, and associated indicators as well as the key activities under each output.

2.3. Overview of the project implementation timescales

Project start-date: September 7, 2018

Project end-date: August 31, 2024

2.4. Project beneficiaries

AGES seeks to support out-of-school adolescent girls and female youth ages 10-25 facing multiple barriers to enrolment, attendance, and learning. The project will enrol two cohorts of students in accelerated basic education (ABE), as well as five cohorts of students in formal school and non-formal/ life skills courses, reaching a total of 82,975 girls. The first cohort enrolled by the project included 20,468 girls[6]. The pre-enrolment assessment of Cohort 1 students indicated that 31% are IDPs; 16% belong to marginalised minority groups; 40% are Af-Maay speakers (including 18% who are Af-Maay speakers in areas where this is a minority language); 13% are orphans; 10% do not live with parents; 4% are married and 4% are divorced. The enrolment of girls with disabilities was prioritised by the project, with extensive sensitisation of stakeholders and multiple rounds of identification being conducted in order to maximise the likelihood of enrolment. A total of 253 Cohort 1 girls have disabilities other than mental health issues, while 6,140 girls were estimated to face severe anxiety and/or depression. The following cohorts included progressively larger proportions of minority students and IDPs, reaching 27% minority girls and 43% IDPs in NFE cohort 4. A breakdown of the out-of-school girls enrolled to date by cohort is provided on Table 1 below.

Intervention pathway

Which girls follow this pathway?

Cohort 1 enrolment

Cohort 2 enrolment

Cohort 3

Cohort 4

Cohort 5

How long will the intervention last?

How many cohorts are there?

What literacy and numeracy levels are the girls starting at?

What does success look like for learning?

What does success look like for Transition?

Formal school

Girls aged 10-12

6,623

5,594

6,479

To be enrolled

4 years

3

Grade 0-1 for literacy-numeracy

Girls achieve grade 4 level for numeracy

Girls enrolled and retained in formal school

Accelerated basic education (ABE)

Girls aged 13-16

7,241

6,035

2 years

2

Grade 0-1 for literacy-numeracy (original estimate; baseline results show otherwise)

Girls achieve grade 4 level for numeracy

Girls enrolled and retained in ABE; a proportion of the girls transition into formal school upon completion

NFE/Life skills

Girls aged 15-25 (17-19 in Cohorts 1 and 2; 15-25 in subsequent cohorts[7])

6,604

4,319

4,198

13,443

To be enrolled

11 months

5

Grade 0-1 for literacy-numeracy (original estimate; baseline results show otherwise)

Girls are able to read with comprehension and achieve mastery of basic operations (addition and subtraction with problem-solving)

Girls enrolled and completing life skills training; transition into higher levels of education, TVET or employment/ self-employment

Table 1 – Breakdown of participants by cohort

2.5. Approach during the COVID-19 pandemic

Following the closure of schools in March – August 2020, movement restrictions and a severe economic crisis, the project team and the Fund Manager have agreed on a set of midterm adaptations to mitigate the impact of the pandemic on the students and schools supported by AGES. Those include:

-Social protection and safety: Support to parents engaged in savings groups, including coaching of adult and adolescent savings groups; linkages with banks; provision of seed funding; provision of hygiene materials to families; training of community health workers to support GEFs;

-Continuation of teaching and learning: Remote learning content provided to students; strengthening supervision and professional support to teachers by regional and district education officers; printing and dissemination of ABE curriculum materials; provision of teaching and learning materials to students in ABE and formal schools;

– Return to school and learning centres: Follow-up on dropout cases; provision of PPEs and hygiene supplies to schools; cleaning of schools prior to reopening; coordination with other development partners on provision of WASH supplies;

-Wellbeing: Provision of grants to GEFs to implement girl-led activities; awareness raising on COVID-19 prevention; provision of psychosocial support to children;

-Combatting exclusionary norms: Community mobilisation to support remote learning; mobilisation of religious leaders to promote equity in education; strengthen official tracking of out-of-school children and gender-sensitive and inclusive quality assurance processes.

Several adaptations made in response to the pandemic continue to be applied to date in response to other shocks, namely the ongoing drought (2021-2022) and the escalation of violence in parts of the country, which have contributed to school closures, displacement, and restrictions to girls’ movement. Continuing adaptations include support to remote and remedial learning; early warning and dropout prevention systems; provision of water to schools; psychosocial first aid; and integration with social protection systems.

1. CONSULTANCY PURPOSE AND DESIGN

3.1. Rationale for the Evaluation

The findings from the second midline evaluation of the first cohort (of all three education pathways) and the fourth cohort of NFE will be primarily used:

• To determine the achievement of expected outcomes against benchmarks established at baseline and progress against intermediate outcome targets;
• To identify factors affecting expected outcomes and if/how these map out against the intermediate outcomes and outputs selected by the project, thus testing its Theory of Change (ToC);
• To gain up to date understanding of the project context to inform ongoing project strategy;
• By the project management team, project partners and stakeholders to inform improvements in the delivery of the project during its lifetime;
• To demonstrate accountability for the funding received to FCDO, other UK and US Government Departments, UK and US taxpayers, UK and US media;
• By the project management team to leverage additional resources from existing and new partners and stakeholders in order to scale-up and sustain the activities /benefits delivered by the project;
• By the project management team to support the on-going development and implementation of the project’s sustainability and succession strategies;
• By partners, stakeholders and the Government to learn lessons from the project for the purpose of informing education programming and sector planning in country;
• By the Fund Manager to feed into and identify insights in order to inform programme level questions; and
• By other donors, academic institutions, and education networks to inform the wider policy debate concerning the education of girls and marginalised girls.

In addition, findings from the baseline evaluation of the fifth cohort of NFE students will be used to establish baseline values for this cohort, enabling the project to determine progress at the final evaluation. The baseline for this cohort will also allow the project to compare the trajectories of this and other cohorts at the final evaluation and better understand how they differ, particularly considering the interruption of project activities experienced by the first cohort and the major shocks (drought and conflict) faced by the fourth cohort. The baseline evaluation of the fifth cohort of NFE students will also identify the factors affecting the expected outputs and intermediate outcomes of the expanded activities with older adolescent girls and female youth participating in cohorts 4 and 5 of the NFE component, assessing the validity of the project’s revised Theory of Change and informing adaptations of those subcomponents of the AGES intervention.

1. 1. Objectives of this Consultancy

The project is seeking to procure the services of an independent external evaluation company to conduct a mixed-methods, gender-sensitive second midline study that is inclusive of persons with disabilities and other marginalised sub-groups within the targeted population (pastoralists, minority clans, displaced girls, girls affected by early and forced marriage, girls in child labour). The second midline study will assess the economy and efficiency of delivery, effectiveness, and equity of the intervention in achieving its outcomes, thus determining VfM, and establish the results of the project at outcome, intermediate outcome and impact levels. It will also conduct an extensive exploratory analysis and review of secondary data to identify the factors affecting the project’s expected outcomes. The study will also include a baseline for a newly enrolled NFE cohort, which will determine baseline values for future assessment of progress against expected outcomes and comparison of results with those of prior cohorts at the final evaluation.

1. 1. Evaluation Questions

The following evaluation questions have been agreed with the GEC Fund Manager. An additional set of questions has been added to enable the project to understand the combined impact of the ongoing crises on the students and targeted schools. Additional questions may be added by the external evaluator as appropriate.

1. What impact did AGES have on the transition of highly marginalised girls into education/learning/training or work opportunities?
2. What works to facilitate transition of highly marginalised girls into education / training / employment and to increase learning?
3. How sustainable were the activities funded by the GEC and was the programme successful in leveraging additional interest, investment, and policy change?

-Likelihood of continuity of interventions, particularly for ABE and NFE;

-Policy shifts in domains of change (gender in education; safeguarding; inclusive and special needs education; access to alternative/non-formal education)

– Leveraged investment in access/ gender

-Sector capacity building

-Social norm changes

1. How successfully did AGES reduce barriers to full participation in education or vocational education for highly marginalised girls?
2. ToC

• Which factors at household/ school level are affecting the acquisition of literacy, numeracy, and financial literacy skills? How are these factors being affected by socio-economic and political changes in South Central Somalia?
• Is the project’s ToC adequately reflecting the factors driving learning/ transition/ sustainability?
• To what extent is the acquisition of social-emotional learning (SEL) competencies affecting learning and transition outcomes? Which subgroups of the population are benefitting the most from SEL competencies?
• To what extent has participation in the project affected transition pathways for girls enrolled in formal education, ABE, and life skills/NFE programming?
• What are the key factors driving female youth’s successful transition into wage and self-employment?
• What are the key factors affecting female youth’s economic empowerment? What additional components would be necessary to maximise success for the subgroups of participants who are lagging behind?
• To what extent are the skills acquired in the NFE course contributing to economic empowerment? And to the transition into employment?

1. Inclusiveness

• Are there subgroups of the targeted population for whom the interventions have been particularly effective in terms of learning and/or transition? If yes, who are they and why have they benefitted the most?
• Are there subgroups of the targeted population who have not benefitted at all from the intervention, or who have benefitted to a much lesser extent than others, in terms of changes in learning and/or transition? If yes, who are they and why have they not benefitted / benefitted less from the interventions?
• Are students enrolled in ABE/NFE acquiring literacy and numeracy skills at a similar pace as their peers enrolled in formal education?
• Are there any differences in learning outcomes between Af-Maay and Af-Mahatiri speakers?
• To what extent are project interventions effective in addressing barriers to the participation, learning and retention of students with disabilities? Are the interventions particularly effective in addressing barriers for some types of disability, but not others?
• To what extent are project activities effective in addressing underlying causes of exclusion, including inter-clan / ethnic dynamics and gendered practices limiting girls’ voice and time to study/attend class?

1. Teaching skills and practices

• Are there specific literacy / numeracy skill areas that are not improving? Why?
• Are there ‘plateaus’ in the acquisition of literacy and numeracy skills? To what extent are project interventions effective in addressing those?
• In a context where education provision is mostly community or privately owned, what works in engaging private sector providers to increase equality and quality?
• To what extent is positive discipline being applied in classrooms, and what approaches are being adopted by teachers?

1. Governance

• To what extent are community education committees addressing factors related to girls’ learning and transition?
• Have State-specific differences been observed in learning and transition patterns?

1. Community practices and girls’ empowerment

• Is the participation of parents/ adolescent girls in economic empowerment activities contributing to increase sustainability of education outcomes, including for other members of the household?
• To what extent are girls’ roles shifting at the household/ community level as they participate in the project? What are the opportunities and barriers emerging for girls as they take on new roles through the application of academic and leadership skills? To what extent these new roles may be harmful for girls from extremely marginalised groups, such as minority clans and ethnic groups?

1. Impact of crises and shocks

• What has been the impact of the ongoing drought and escalation of conflict on household and school conditions? Which subgroups have been particularly affected and why?
• Are there differences in learning and transition outcomes for girls from the most affected households?
• How likely are the mid- and long-term effects of these shocks to affect the sustainability of the interventions at school, community, and system levels?
• To what extent are the project interventions being successful in mitigating the impact of shocks, including for the most vulnerable subgroups of girls?

The following evaluation questions refer only to the NFE cohorts 4 and 5.

1. Civic action and self-efficacy

• Is the participation in GEFs contributing to boost girls’ self-efficacy? If yes, to what extent, and for whom?
• What types of civic action are GEFs involved in? Have those activities contributed to address barriers to adolescent girls and female youth? If yes, how and to what extent?
• Are project activities contributing to mitigate mental health issues, particularly among the most marginalised girls and female youth? Has the intervention been able to reduce the stigma around mental health issues and increase support-seeking behaviour?
• To what extent are girls’ empowerment and leadership skills intervention and opportunities perceived as positive by local leaders? Are there specific subgroups whose empowerment is more supported than others?

1. Access to other development opportunities

• To what extent has the participation in VSLA contributed to increase female youth participation in income-generation activities? What are the key skills and resources contributing to boost female youth participation in self-employment?
• To what extent has the inclusive investment in VSLA business plans been successful? What is the profile of the successful/ u