SOS Children's Villages International
Terms of Reference
Final Evaluation of GREEN+ Project
Implemented in Ethiopia and Uganda 1980-00/2019
CONTENTS
Abbreviations
- Context and Background
- Purpose and Objectives
- Scope
- Evaluation questions
- Design and Approach
- Work plan
- Evaluation Management Arrangements
- Requirements for the evaluator(s)
- Specifications for the submission of offers.
- Annexes
Abbreviations
ADA – Austrian Development Agency
Community Based Organization – Community-based organization
FGD – Focus group discussion
FSP – Family strengthening program
GBV – Gender-based violence
ICAP – Institutional Capacity Development for Quality Child Care and Protection project
IGA – Income generating activities
IOR ESAF – International Office – Eastern and Southern Africa Region
Key Implementing Partner – Key Implementing Partner
KM&L- Knowledge Management and Learning
M&E – Monitoring and Evaluation
MA – Member associations
NO – National office
People with Disabilities – Persons with disabilities
SDGs – Sustainable Development Goals
SOS CV Ethiopia – SOS Children’s Villages Ethiopia
SOS CV Uganda – SOS Children’s Villages Uganda
SOS CVI – SOS Children’s Villages International
ToR – Terms of Reference
1 Context and Background
1.1 About SOS Children’s Villages
1.1.1 SOS Children’s Villages International
SOS Children’s Villages is a global federation present in 138 countries and territories via national SOS Children’s Villages associations, working in more than 2000 programme locations worldwide. The main focus of the organisation is to provide sustained support to vulnerable children who have lost or are at high risk to lose parental care. In Family Based Care Programmes (FBC), SOS CV provides a family-like environment to the abandoned child or young person and offers quality care and support for their development. Through Family Strengthening Programmes (FSP) SOS CV helps parents and communities build capacities to care for their children and prevent family breakdown. For more detailed information see https://www.sos-childrensvillages.org/
SOS CVI is represented in the Eastern and Southern African region by the regional office IOR ESAF, located in Addis Ababa, Ethiopia. Currently, SOS CVI is implementing similar FSP projects with a focus on Environment and Environmental Protection in Zambia, Somalia, Uganda and Ethiopia.
1.1.2 SOS Children’s Villages Austria (SOS-Kinderdorf)
SOS-Kinderdorf (SOS Children’s Villages Austria, SOS CV Austria) is a member of the SOS CVI federation and has established long-term working relationships with both SOS CV Ethiopia and SOS CV Uganda. The collaboration with SOS CV Uganda dates back to 1991, while the support to SOS CV Ethiopia started over 12 years ago. Specifically, since 2010, SOS CV Austria has partnered with SOS CV Ethiopia and SOS CV Uganda to introduce and promote the Family Strengthening Program (FSP) approach since 2007 and 2010, respectively, with co-funding from the Austrian Development Agency (ADA).
For the GREEN+ project, SOS CV Austria is the contract holder with the donor agency, ADA. In the implementation of this project, SOS CV Austria partners with the two Member Associations (MA) “SOS CV Ethiopia” and “SOS CV Uganda”. All of them are part of the international federation SOS CVI, which has a regional representation, the International Office Regional Eastern and Southern Africa (IOR/ESAF), located in Addis Ababa.
The knowledge management and learning component (called “Umbrella”) of the GREEN+ project is jointly implemented by SOS CV Austria and SOS CV IOR ESAF.
SOS CV Austria, as the contract holder with the donor agency ADA, is commissioning this evaluation in its role as evaluation manager.
1.2 About the GREEN+ Project
Socio-economic Empowerment of Vulnerable Children and Youth as well as their Families with Special Focus on Green Economy (GREEN+) is a five year project implemented in Uganda in Entebbe and Fort Portal and in Ethiopia in Mekelle and Hawassa, with a total budget of 3,500,000 Mio EUR. The project is implemented as strategic partnership, following a series of ADA funded project that SOS, ADA and the Governments of Ethiopia and Uganda have been jointly involved in since 2007.
The final evaluation of the predecessor project Institutional Capacity Development for Quality Child Care and Protection (ICAP) has been completed in August 2018. The key recommendations have influenced the design of GREEN+. In particular GREEN+ focused on the vulnerable/most-vulnerable families and children and on the support of community structures for child care and child protection. Also, under GREEN+ SOS CV developed and applied approaches that make gender equality, diversity and inclusion, work for quality child care and child protection, like through the “role model parents”. Another key recommendations, was to introduce a simple green economy approach into the FS program, linked to the FDP process.
The complete Final Evaluation of ICAP will be made accessible to the successful bidder.
The following are the Specific Objective and the 6 Results of GREEN+:
Specific Objective: To support the socio-economic empowerment of 2,800 children, and 400 youth as well as their families with a special focus on green economy.
% of boys/girls enrolled in and regularly attending formal or non-formal education.
% of male/female parents/ caregivers that usually receive health care treatment when ill, although some services may not be available or accessed.
% of communities structures with a formal system for child protection and capacity to follow-up or support child rights violation cases
% of families exiting the programme that are self-reliant
Results:
1. Income-generating capacity of 1,350 families strengthened through green and other economic activities, with focus on women/girls’ socio-economic empowerment.
1.1 Average monthly household income to cover children’s basic needs for assessed families in Euros
1.2 % of families that have sustainably adopted at least one green practice
1.3 % of male/female parents/ caregivers that consistently provide age-appropriate care, are very accepting of and affectionate to the child(ren), and always recognizes and attend to the needs of all children
2 300 critically vulnerable families, children and other persons have enhanced access to social protection support and critical support to engage in IGA, with a focus on green economic activities.
2.1 % of critically vulnerable male/ female parents/ caregivers that access social protection assistance or similar support.
2.2 % of critically vulnerable boys/girls who access basic needs support and services.
2.3 # of project communities that implement social protection initiatives.
2.4 # and type of policy changes made at local, regional or national level as a result of SOS CV involvement in policy dialogues on social protection (Uganda only)
2.5 % of project communities where SOS CV is an active participant and influential in a network of stakeholders that actively addresses the situation of vulnerable children
3 400 youth have market-relevant skills and capacities as well as enhanced access to income-generating opportunities, with a focus on green economic activities.
3.1 % of male/female youth vocational training participants who use acquired skills for income-generation.
3.2 % of male/female youth that have sustainably adopted at least one green practice
3.3 % of male/female youth who can name at least two contraceptive methods
4 Community structures in 14 communities strengthened to support quality child care and protection, with a focus on protecting the environment and creating opportunities for green economic and community initiatives.
4.1 % of Key Implementing Partner that have reached level 1 or 2 financial and organizational sustainability.
4.2 % of project communities who have developed and are following-up with a joint implementation plan
4.3 % point increase of community members aware of the main factors putting children and families in situation of risk (compared to baseline).
4.4 % of project communities that implement green community initiatives
5 Inclusive participation in community decision-making processes, with a focus on green economy and environmental issues, and inclusive access to basic services improved for 520 vulnerable and marginalized individuals particularly girls and women, including those with disabilities.
5.1 % of marginalized community members (disaggregated by sex, age and disability) that actively participate in community structures and decision making.
5.2 % of girls/boys engaged in school or community fora.
5.3 % of girls/boys and % of male/female parents/caregivers with disabilities receiving attention
6 Learning and Knowledge Management improved in SOS CV and for Key Implementing Partners (Key Implementing Partners)
6.1 # and type of documented changes made to ways of working organizational structure or programmatic approaches by SOS CV as a result of learning processes.
6.2 # and type of documented changes made to ways of working organizational structure or programmatic approaches by Key Implementing Partners as a result of learning processes.
6.3 # of external stakeholders reached with awareness raising/influencing messages.
1.3 Project Locations
In Ethiopia, the project is implemented in two major cities of two regional states: Mekelle, and Hawassa:
- In Mekelle city, the project is implemented in the five most vulnerable communities of Semien Sub-city (Dedebit, Mesfin, Lekatit, Industry, and Meles).
- In Hawassa, the project is implemented in two vulnerable sub-cities, Addis Ketema and Haik Dar.
On this way, the project continued with interventions in Mekelle and Hawassa, where the need to support our specific target group is still high, as suggests the final evaluation from the previous project, ICAP, and government as well as international statistics.
Due to start of the conflict in November 2020 in Tigray Region, the project implementation in Mekelle was largely hampered and funds have been reallocated for emergency response for the targeted families.
In Uganda, the project is implemented in Wakiso district and Kabarole and Kamwenge districts:
- In Wakiso district, the project is specifically in the communities of Kisubi, Nalugala, and Nkumba parishes.
- In Kabarole district, the project is implemented in 3 sub-counties of Karangura, Kichwamba, and Mugusu.
The project targets poverty pockets and marginalized communities in all two districts. Wakiso hosts thousands of poor children due to a number of factors including rural urban migration. The largely urbanized district surrounded by poor fishing communities on the shores of Lake Victoria with extremely poor access to services and public goods. Already poor livelihood opportunities for households in these communities are diminishing rapidly due to the government’s crack-down on illegal fishing. The fishing communities are also affected by land-grabbing. Many of the community members have also migrated to the area from the islands in Lake Victoria. The 2018 report of the equal opportunities commission noted that people living in islands are among the poorest among the Ugandan population with limited access to services and public goods.
Kabarole district is affected by the conflict in DRC as well as the Rwenzori tensions between the Government of Uganda and local indigenous communities that have affected the lives, access to services, and livelihoods in particular for Bakonzo people for many years. In addition to the Bakonzo communities in the Rwenzori Mountains, the project explicitly targets vulnerable children and households in the refugee and host communities, where services and supply with public goods are overstretched.
2 Purpose and Objectives
2.1 Purpose
Why this evaluation is needed:
A subsequent project phase will be developed and presented for funding to the Austrian Development Agency, with the main focus on Gender Equality (DAC Gender Marker II). Therefore, the main purpose of the evaluation is to draw learnings and recommendations from the current project in order to inform the new phase of the project and show accountability towards our stakeholders.
- Learning: We need a thorough analysis of the past implementation regarding the relevance and effectiveness of our planned approaches and methods for the learning and capacity building component as well as the Innovation Incubator for Green Economy. To what extent have they been effective and the right ones to contribute to the planned outcomes and outputs as well as the needs of our targeted beneficiaries?
- Recommendations: The evaluation will be valuable to provide insights on how the main thematic areas of the GREEN+ Project (Green Economy, Gender Equality and Child Protection) worked out and what are their potential regarding sustainability. The recommendations for those areas will be integrated in the planning for the next phase, and in the planning of other future projects.
- Accountability: And of course, this evaluation is also an instrument of showing accountability towards stakeholders, especially the Austrian Development Agency, to what extent the expected results (outcomes and outputs) have been achieved.
2.2 Objectives
The main objective of the evaluation is to assess and present the main results achieved of the Strategic Partnership GREEN+ (2019-2023) in Uganda and Ethiopia to facilitate learning as well as to receive recommendations for a potential subsequent phase 2024-2028.
1. To determine the extent to which the project interventions addressed the target beneficiaries and stakeholders‘ needs. (Relevance)
2. To asses to what extent the objectives defined in the six result areas have been achieved. (Effectiveness) (outcome level/results)
3. To identify facilitating and hindering factors for sustainability and based on this develop …recommendations for … how to increase the potential for sustainability in future, , with a particular focus on gender equality and green economy interventions. (Sustainability)
2.3 Intended users
- The GREEN+ project teams in Uganda and Ethiopia, as valuable input for the project design of the next phase.
- The donor agency, as a means for accountability and communication
- The stakeholders (local government bodies, local partner Community Based Organization’s) as a reference document for knowledge sharing and learning
The evaluation is supposed to establish the level of achievement of specific objective, results and outcomes / outputs as per Logframe, Project Document, Workplan and Budget, taking into account the specific purpose, objectives, core thematic areas and prioritized OECD/DAC criteria, including the specific evaluation questions described in these Terms of Reference (ToR).
The task includes identifying successes and challenges, identifying gaps and good practices and draw lessons that can inform quality improvement and scale up.
3 Scope
3.1 Time:
The evaluation should cover the entire project phase of GREEN+ from project start to April 2023.
The evaluation will tentatively take place from March to July 2023.
3.2 Thematic / Structural:
GREEN+ has an overall learning component and country based project implementation in Ethiopia and Uganda. All three dimensions should be assessed within the Evaluation.
All across the above-mentioned functions of this evaluation (learnings for planning of next phase, and accountability) it is crucial to keep a lens on the core thematic interests of GREEN+, which are (1) green economy, (2) gender equality and (3) child protection. These elements will continue to play a central role in the next phase. They are also of high importance to the donor, hence should be evaluated also for accountability reasons.
Two further areas, which are equally important are youth empowerment and capacity development, which are cutting across the before mentioned 3 thematic areas. Respective evaluation questions are integrated accordingly.
In particular the learning approaches need to be looked at closely and recommendations for a better impact (better concepts, approaches, methods) of future KM&L interventions should be made.
3.3 Geographic:
- Ethiopia
- Samples from Hawassa town in two sub-cities Addis Ketema sub city (Daka and philadelphia kebele) and Haik Dar sub city (Haik dar Kebele).
- At the moment of the publishing of these TORs the situation in Tigray and Mekelle remains volatile. Therefore, the evaluators should present an alternative option for for a remote evaluation concerning this geographic area.
- Uganda
- Samples from Entebbe and Fort Portal will be from communities of Kisubi, Nalugala and Nkumba (in Wakiso District/Entebbe) plus 3 sub-counties of Karangura, Kichwamba, and Mugusu respectively (in Kabarole District/Fort Portal)
3.4 OECD/DAC criteria:
The OECD/DAC evaluation criteria – specifically relevance, effectiveness and prospects for sustainability – will guide this evaluation.
IMPORTANT! All across the evaluation and all across the specific evaluation questions in particular (see chapter 4), data collection and data analysis have to be disaggregated at least by gender, disability status and age. Societal, intersectional inequalities need to come out clearly.
4 Evaluation questions
4.1 Relevance
1. To what extent has the program design proven to be relevant to the needs of the beneficiaries and stakeholders? And what are areas for improvement?
2. To what extent have new approaches, like the Innovation Incubator, Green Economy and Environmental Protection, been relevant to the target group (vulnerable and critically vulnerable families)?
3. To what extent are the objectives and achievements of the project regarding green economy and the Innovation Incubator consistent with the needs and priorities of the targeted beneficiaries, (differentiating clearly between women and men as well as People with Disabilities and intersectionality of those groups), and of stakeholders?
4. To what extent have the design and implementation of the initiatives regarding child protection, social protection, inclusion and gender equality been relevant to the realities and challenges that women and girls, men and boys are facing?
5. To what extent have the initiatives taken regarding inclusion and participation of People with Disabilities and marginalized groups been relevant to the realities and challenges that these particular groups are facing? What can be done to better promote inclusion and participation of People with Disabilities and other marginalized groups at organizational, community and family levels?
4.2 Effectiveness
6. What facilitating and hindering factors in terms of effectiveness could be identified? How have they been addressed? What shall be done differently to improve the effectiveness?
7. Have societal inequalities in the targeted communities influenced the effectiveness of project interventions? If so, which inequalities are those and how should they be addressed in the next phase?
8. Which initiatives regarding economic empowerment and green IGAs did (not) prove to be effective for improving the socio-economic situation of targeted families with a special focus on the difference between the different genders, different age groups, People with Disabilities?
9. To what extent has the project been able to address effectively strengthen youth employability in the targeted communities? What were facilitating and hindering factors?
10. Which initiatives regarding green economy and green IGAs did (not) prove to be effective for improving the awareness about environmental issues and protection in families, communities, Key Implementing Partners and staff?
11. To what extent have the measures addressing issues related to gender equality and inclusion of people with disability turned out to be effective? What has been the contribution of approaches such as P4R, EMB?
12. To what extent have the green economy and gender equality initiatives of the project used methods with meaningful active participation of beneficiaries? What can be done to improve the use of effective participatory methods?
13. To what extent have a) both genders, b) People with Disabilities, c) youth and d) intersectional (a), b) and c)) effectively participated in the innovation incubator and what were the fostering/hindering factors?
14. To what extent are Key Implementing Partners and Community Based Organizations effectively responding to the needs of children in the community, in particular regarding social protection responsibilities?
15. What has been the effect of KML within the program? What were the challenges and what can be improved for the next phase?
4.3 Prospects for Sustainability
16. What approaches and interventions appear most promising and sustainable to be continued in the next phase and why? (Design and Implementation) Which ones should be stopped and why?
17. (by Key Implementing Partners and also beneficiaries)? What have been facilitating and hindering factors for sustainability in innovation incubator projects?
18. What have been facilitating and hindering factors for sustainability for Key Implementing Partners and Community Based Organizations in the context of child protection?
The questions can be refined and restructured in agreement with SOS Austria during the inception phase. Any changes should be explained and should adequately reflect the overall purpose and scope of the evaluation.
5 Design and Approach
The evaluation’s design and methodology shall follow the purposes described above. Both primary and secondary data should be used in the evaluation, and collected from a wide and diverse range of primary and secondary sources. Secondary information should be collected from documents provided to the evaluator. Primary information should be collected from the stakeholders through on-site direct observations, semi structured interviews, and focus group discussions. A mixed-methods approach, combining different methods of data collection and analysis, is assumed, to provide strong evidence of achievement against the key evaluation questions, craft the applicable methodology and identify problems and opportunities in the GREEN+ project in Uganda and Ethiopia and the Umbrella component.
The design of the evaluation shall include but not be limited to the following,:
- Desk analysis of project documents, annual reports and existing evaluative evidence (e.g. recent, context-relevant studies, reports and statistics from governmental and non-governmental institutions)
- Data collection in the field and virtual
- Individual interviews (structured and semi-structured, face-to-face or by phone), (min. 15)
- Focus group discussions with key stakeholders and informants (min. 20)
FGD’s / Interviews need to be held with the following
- Beneficiaries, including children and youth
- Project team
- Key Implementing Partners (Key Implementing Partners) and Community Structures
- Local Government stakeholders
- SOS staff involved in the project (UG, ET, IOR, AT) especially M&E, IPD, PD
- Donor agency
Survey as deemed appropriate.
Data triangulation and quality control are very important and need to be discussed in the inception report.
The Methodology of evaluation should be based on participatory approach involving and engaging a wide and diverse range of stakeholders. Stakeholders` participation is necessary for accountability, promoting ownership and sustainability, facilitating buy in, and further use of the evaluation recommendations.
Disaggregated data collection and analysis including gender and age (and disability status wherever possible) is mandatory and constitutes a central quality criterion for all deliverables. This includes that the applied methodology has to ensure that male and female, and all respondents in all their diversity get equal opportunity to voice their views and that the settings need to be designed in such a way that they allow for sufficiently safe spaces to encourage women, men, boys and girls, People with Disabilities, and of all origins equally to respond freely.
The use of modern technology in the process (mobile devices for documentation, videography, etc.) is welcome but not mandatory.
The evaluation design should follow ethical guidelines (e.g. UNEG Ethical Guidelines for Evaluations) and ensure that measures are in place to fulfil data protection and privacy. Furthermore, the methodology should be human and child rights based, child appropriate, gender sensitive and inclusive.
The evaluation has to be conducted according to the guidance, rules and procedures established by ADA as reflected in the Guidelines for programme and project evaluations. The inception and evaluation report need to be in line with the respective quality check-lists included in the Guidelines (see Annex 5 and Annex 6). Failure in respecting those will result in termination of contract and suspension of payment. More information can be found on the donor website in the evaluation section: https://www.entwicklung.at/en/ada/evaluation
6 Work plan
The evaluation will be conducted between Mach and July 2023. A total of 45 working days based on a team of three members, are estimated for this evaluation (details see below).
Deliverable Responsible Dates
Kick-off and Inception Phase
week of March 6 Presentation at Kick-off meeting Meeting between contractor and consultant
Inception
week 1-2 of March Document review and initial interviews Consultant
week 2 of March Draft inception report Consultant
week 3 of March Online presentation of the draft inception report Consultant
week 4 of March Comments on the inception report Contractor
Week 1 of April Inclusion of comments in inception report and submission of final inception report Consultant
Fieldwork / Data collection
week 2-4 of April
Data collection, including in the field Field visit, conduct data collection (interviews etc.) Consultant
week 1-3 of May
Data analysis and submission of report Analyse data and prepare draft report (see report structure in ADA Evaluation guidelines) Consultant
week 4-5 of May Submission of draft report Consultant
week 1 of June Comments on the draft report Contractor
week 2 of June Validation Workshop for finalizing the final evaluation report Consultant
week 3 of June Collection and inclusion of feedback in final draft report Consultant
Final report
week 4 of June Submission of final evaluation report to contractor (hard and electronic copy) Consultant
Deliverables:
The consultants will submit the following deliverables:
- Kick-Off presentation reflecting the approach suggested in the offer
- An inception report (10-15 pages without annexes), for structure and content see Annex 5, ADA Guidelines for Programme and Project Evaluations;
- Online presentation of the Draft Inception Report , revised inception report with feedback matrix completed (see Annex 8, ADA Guidelines for Programme and Project Evaluations)
- Presentation of the evaluation’s preliminary findings at a Validation Workshop
- A draft evaluation report (about 25-30 pages without annexes) including an executive summary (for structure and content see Annex 6, ADA Guidelines for Programme and Project Evaluations)
- The ADA Results Assessment Form (RAF) for this evaluation (see Annex 9, ADA Guidelines for Programme and Project Evaluations). This form must be completed and submitted together with the draft evaluation report and can be downloaded at: https://www.entwicklung.at/en/ada/evaluation
- final evaluation report with the feedback matrix completed.
All deliverables must be in English.
7 Evaluation Management Arrangements
When at the programme location, accommodation and transport to the field will be organized by the member association, including the full coverage of costs during the entire stay of the researcher(s) (not to be included into price proposal).
SOS Children’s Village and the Austrian Development Agency will provide support (information/interviews; providing relevant documents, feedback to draft of the draft review, participation at presentation of draft findings). The interviews will be arranged and supported after agreeing on the action & time plan. National or location level staff (SOS) will be available at site to help organizing the interviews including contacting SOS, announcement and local preparation of evaluation, linking to community duty bearers and national authorities if required.
External evaluators should not be biased and have any reason for conflict of interests. Evaluation team must respect participating communities’ culture, social norms, values and behaviour; maintain appropriate relationships with participants to this evaluation and keep private information about beneficiaries, acquired during the evaluation, strictly confidential.
8 Requirements for the evaluator(s)
The evaluation team should consist, at a minimum of three members, including national evaluators from.Ethiopia and Uganda. The evaluation team will work closely with the respective national M&E coordinators of SOS Uganda, SOS Ethiopia and the SOS CV Austria and regional office IOR ESAF in order to facilitate a comprehensive understanding of the organization and easy access to information and contact persons.
The evaluator(s) must not have been involved in the design or implementation of the GREEN+ project.
Diverse evaluation teams (gender, People with Disabilities, origin, age, etc.) will have a competitive advantage in the selection process.
Sound understanding of the current discourse in the area of gender equality programming is crucial for this assignment.
Key qualifications required of the evaluation team are:
- Proven, solid experience and competency (submit at least three previous evaluation reports in application) in leading and conducting project/programme evaluations in this thematic area, including final evaluations, in complex and sensitive environments.
- Relevant academic degree (master level) in one or more of the following fields: Development Studies, Monitoring & Evaluation or a relevant, directly r