World Vision
CONSULTANCY SERVICES TO UNDERTAKE THE END LINE EVALUATION FOR SOMREP EU RESTORE AND SIDA PROJECT
The Somali Resilience Program (SomReP) requires the services of a consultant to undertake End Line Evaluation for the EU RESTORE and SIDA project in Somalia.
- About SomReP
The Somali Resilience Program (SomReP) is a resilience building consortium which aims to address the underlying causes and impacts of vulnerability to climatic shocks and other related stressors. Following the Somali famine of 2011, NGOs and donors came together to share best practices to develop a multi-sector, multi-year, area-based program with graduation pathways which bridge the humanitarian-to-development nexus. The consortium undertakes layered and sequenced interventions which build absorptive, adaptive and transformative capacity at household, community, state and system levels. The program targets the ultra-poor across livelihood groups, including agro-pastoralists, pastoralists, fisher-folk and IDP and host communities with a special emphasis on women, youth, persons with disabilities and the marginalized.
SomReP works in the sectors of disaster risk management, climate smart agriculture, productive assets development, natural resource management, basic health/nutrition services, shock responsive safety nets, green technologies, economic empowerment and market systems development. It is an iterative, learning consortium which measures the impact of resilience interventions and employs evidence to adapt approaches, scale effective interventions and promote understanding of best practices at local, regional and international levels with civil society, government and academia. The consortium hosts the Response Innovation Lab (SomRIL), an outward-facing innovation brokerage which convenes NGO, private sector, and government challenge-holders and local, regional and global solution-providers to identify innovations and adapt them to solve Somali problems.
- Project Details
Project Name:
RESTORE 2
Project Number:
T05-EUTF-HOA-SO-47
Country:
Somalia
Start and End date of Project:
01.01.2019 – 31.12.2022
Date Baseline Completed
September 2019
Beneficiaries:
38,707 HHs
National Office Contacts:
- Project Background
Launched in January 2019 for a project period of 36 months, the EU RESTORE and SIDA project has been implemented in Somaliland, Puntland and South Central Somalia in the districts of Badhan, Baidoa, Burco, Bosaso, Doolow, Eyl, Hargeisa, Lasanod, Lughaya and Salahley, Ceel Afweyne, Afgooye, Xudur Belet Xaawo, Ceel Barde, Waajid and Odweyne, and has been addressing specific needs and constraints of pastoral, agro-pastoral, peri-urban, fisher folk and IDP communities. Through the project, a total of 78,613 beneficiaries have been reached through various resilience interventions implemented by our partners ADRA, AAH, CARE, DRC and Shaqodoon.
The consortiums overall project design and implementation has been built upon a conflict sensitive approach that has taken into consideration, cross-cutting issues like the gender equality and environment which are pertinent in ensuring that the resilience gains are felt all round. In respect to this, SomReP ensures that from the onset of the program, issues to do with equal participation of both genders is incorporated in all its interventions such as Good Agricultural Practice (GAP), Animal Health initiatives, cash for work and unconditional cash transfers etc. SomReP strives to ensure that gender mainstreaming and inclusivity is applied during the beneficiary selection process in all the interventions.
In addition to ensuring cross cutting issues are taken in to consideration, in terms of implementation of infrastructural activities, SomReP undertakes prior assessment to inform the risk of damaging the natural resources as well as increase awareness on the importance of preserving natural resources and environment. SomReP builds on partners’ experience in participatory conflict analysis, conflict resolution as well as the Do No Harm approach by continuously working with the target communities in identifying key sources of conflict and peace, particularly relating to the use of natural resources. In so doing, SomReP engages with the communities, the Village Development Committees and local leaders to help identify areas of conflict sensitivity and ensures collaboration with various federal level and member state level government ministries to ensure that project activities are informed by not only the Do No Harm principles but also the Somalia Conflict Early Warning Early Response Unit and which contributes to conflict prevention.
- Project Goal
The project goal for the EU RESTORE and SIDA Project seeks to contribute to the resilience of already vulnerable communities in Northern Somalia, and reduce the effects of forced displacement and irregular migration in the region.
Project Specific Objectives
The specific objective of the project is to sustainably improve food security, livelihoods and build resilience through effective risk management, protection of productive assets and governance of natural resources for pastoral, agro-pastoral and Peri-urban communities in Somaliland, Puntland and the South Central Somalia. To achieve this, the project worked towards the following intermediary outcomes:
Outcome 1: Enhanced risk management and disaster preparedness through community action and contingency planning.
Outcome 2: Enhanced food security and capacity to meet social needs through sustainable cash-based assistance mechanisms and improved access to social capital.
Outcome 3: Enhanced livelihood diversification for women, men, and youth through the restoration and protection of productive assets, value chains and the uptake and adoption of agricultural technology.
Outcome 4: Enhanced management and governance of natural resources, including soil and water systems to support sustainable pastoral livelihoods.
Outcome 5: Program learning and research generated and shared among relevant stakeholders (including communities, NGOs, and government)
The table below provides a summary of the key outcomes and outcome indicators of the two projects.
Table 1: Key outcomes and indicators of the projects
Objective/ Outcome
Key Indicators
Principle Objective: To increase the resilience of already vulnerable households and communities in Northern Somalia, and specifically to sustainably improve food security and livelihoods and build resilience through effective risk management, protection and restoration of productive assets and governance of natural resources in Puntland and Somaliland to protect their livelihoods over continuing shocks.
- % of target households who have improved and/or diversified their livelihood (or income generation) sources by end of the project (Target: 50%) [gender disaggregated]
- % households who are structurally non-poor [1] as measured by stochastic and structural poverty.
- Resilience index 1 ( as measured by Tango framework)
- Absorptive Capacity Index.
- Adaptive capacity Index.
- Transformative capacity Index
- % of targeted households with little to no hunger as measured by Household Hunger Scale
Specific objective/ Outcome: Sustainably improve food security and livelihoods and build resilience through effective risk management, protection of productive assets and governance of natural resources for pastoral, agro-pastoral and peri-urban communities in Puntland and Somaliland.
SoI1: % of households using new community contingency resources.
SoI2: % change in HH income levels per season (seasonal trends)
SoI3: % increase in community assets score.
- % of communities who report to utilize EWEA information to make risk informed decisions (disaggregated by livelihood zone)
SoI4: % change in perception of effectiveness of local leaders/institutions in issues of livelihoods, DRR, conflict mitigation and natural resource management.
- % of people in targeted districts (gender-disaggregated) who view local governments as accountable and responsive to community priorities in providing equitable services and promoting resilience interventions
- % of innovations piloted through SomRIL and adopted for scale-up
Intermediary Outcome 1: Enhanced risk management and disaster preparedness through community action and contingency planning in targeted villages across Somaliland and Puntland.
Io1I1: % of communities with greater involvement by women or marginalized groups in local planning and decision-making processes.
- % of the targeted communities know the early warning signs and know what to do in case of an emergency or disaster.
Io1I2: % of targeted communities indicating improved Knowledge, Attitudes and Practices on contingency planning.
Io1I3: % of community initiatives supported by sub-national and national institutions and authorities.
Intermediary Outcome 2 : Enhanced food security and capacity to meet social needs through sustainable cash-based assistance mechanisms and improved access to social capital in the targeted village
Io2I1: % of target HH who have improved Food Consumption Score (FCS) index.
Io212: % of target households who have improved Coping Strategies Index (CSI). (disaggregated by household head gender, vulnerability type, and livelihood zone).
- Io213: % of households with improved household hunger scale (HHS)
- Io214: % of households with improved household Food Insecurity Access Scale (HFIAS).
Io2I2: % of most vulnerable HHs benefiting from social community transfers during stress and shocks (safety nets).
Intermediary Outcome 3 : Enhanced livelihood diversification for 8,196 number of HH women, men, and youth in targeted villages through the restoration and protection of productive assets, value chains and the uptake and adoption of agricultural technology.
Io3I1: % of HHs newly engaging in diversified livelihood strategies (data disaggregated by sex, livelihood group and strategy employed).
% increase in household asset diversification score (disaggregated by household head gender, vulnerability type, and livelihood zone)
Io3I2: % increase in diversification of asset ownership (access) at HH level (data disaggregated by sex, type of asset and livelihood group.
% increase in household incomes (data disaggregated by household head gender, vulnerability type, and livelihood zone).
% of households experiencing yield improvements (by gender and livelihood zone))
% Increase in yield per unit hectares
% increase in households who realize improved profit gains from their sales
% of HHs engaging in multiple income-generating activities as measured by participation index [1]. (disaggregated by household head gender, vulnerability type, and livelihood zone)
% increase in households who reports easy access to markets
% increase in households who report improved $ sales volume per season.
% increase in households who have access to formal financial services (e.g. those able to open accounts /access loans from formal banks, mobile money providers)
% point increase in the proportion of women that engage in jobs in sectors of sustainable livelihoods (diversified production-related incomes) and economic growth (business development services like transport, milling, bulking, and input supply)
Intermediary Outcome 4: Enhanced management and governance of natural resources, including soil and water systems to support sustainable pastoral livelihoods in target villages.
Io4I1: % of target population with all year access to multi-use water (for irrigation, domestic use & livestock).
Io4I2: % of water infrastructure with operational maintenance systems.
Io4I3: #of hectares of land benefiting from improved agricultural management(water conservation measures
- Extent to which targeted communities are satisfied with delivery of basic services by local government (measured on a scale of 1 to 5)
- % increase of participation of women and other disadvantaged groups in decision making at the district level in the determination of development priorities and CAAP financing.
- Proportion of women with a positive perception on their influence on business and economic-related decision-making
Intermediary Outcome 5: Program learning and research generated and shared among relevant stakeholders (including communities, NGOs, and government)
Io5I1: % of the government stakeholders with knowledge and capacity to participate in Resilience monitoring, evaluation and research studies.
- Extent to which local development annual plans have incorporated priorities from communities identified through CAAPs (measured on a scale of 1 to 5).
- % of targeted government staff who are able to lead CAAPs monitoring and evaluation processes at the community level
- Project stakeholders
The key stakeholders for this project include but not limited to;
Government Institutions
- Ministry of Health, Hygiene and Sanitation
- Ministry of Environment
- Ministry of Agriculture
- Ministry of Water
- Ministry of Livestock
- HADMA
- NADFOR
Local Government
- Community Leaders
- Village Development Committees
- Local Government Offices (Mayor’s office etc)
Local Committees
- Natural Resource Management Committees
- Early Warning Early Action Committees
- Social Affairs Committees
- Water Management Committees
Private Sector Organizations and Diaspora Community
Direct Project Beneficiaries
- Men and women who participated in the different livelihood interventions such as cash for work, GAP, VSLA
- Youth provided with vocational education or enterprise based or entrepreneurial training
- Persons with disabilities
- Community Animal Health Workers
- Government Ministries through structural support
Implementing Partners and project staff
- Purpose of the End line evaluation
SomReP has been implementing resilience activities under the two grants to help assist the vulnerable communities since the project started in 2019. At the beginning a baseline assessment was conducted in 2019 to establish and document baseline values for key indicators of the projects so as to serve as a basis for measuring impact. The end line evaluation will therefore build on the baseline and assess the projects performance in terms of its impact, effectiveness and sustainability as well as capture the project achievements, challenges and best practices to inform future strategic programming and project development. The evaluation will also review the recommendations provided in the baseline assessment and assess the extent to which they have been incorporated into the design of the resilience activities.
The end of project evaluation will therefore look at the following objectives;
Objective 1: Evaluate to what extent the resilience interventions carried out under the 2 grants have delivered on the following;
Effectiveness: The end line evaluation will seek to find out how far the project’s results and its specific objectives were attained by focusing on the following areas;
- To what extent have the planned objectives in the log frame of the project been reached?
- Did the intended beneficiaries participate in the intervention?
- To what extent has the project activities contributed to the resilience of the chronically vulnerable people, households, communities and systems in the targeted pastoral, agro-pastoral, peri-urban, fisher folks and IDPs?
- What were the major factors influencing the achievement of the objectives of the project?
- Were there opportunities for collaboration utilized in delivery of the interventions, which ones and how did they contribute to increased effectiveness?
- Were there any shortcomings that hindered the effectiveness delivery of the interventions?
- Were proper accountability and risk management framework put in place to minimize risks on the program implementation?
Efficiency: This criterion will look at the extent to which the interventions delivered results in an economic and timely manner compared to what was planned. The assessment of efficiency will therefore look at;
- How efficient was the delivery of the project not only in terms of expenditure but also in terms of implementation of activities?
- Was the project activity implementation considered to have been cost efficient while not compromising quality?
- Were there opportunities in the program implementation to reach more beneficiaries with the available budget without compromising on the quality?
- Were there any changes made to the initial program design that were influenced by the reality on the ground? What were the outcomes as a result of this?
Relevant/Impact: This criterion will look at the extent to which the program objectives and interventions responded to the key stakeholders (Government institutions, local authorities, beneficiaries, youth, women, persons living with disability) needs, policies and priorities. The criterion will focus on the following;
- How relevant were the activities implemented by the project in addressing the needs of the stakeholders?
- What is the perception of the project by the stakeholders and how has it impacted on their lives?
- To what extent has the project been able to adapt and provide appropriate response to the needs and priorities of the stakeholders.
- What has been the effects/impact of the project?
- To what extent are the observed changes due to the interventions rather than other factors.
- Have there been any unintended or unexpected impacts and if so, how have these affected the overall impact
- What has been the contribution of the project to the country’s development agenda and the National Development Plan 9.
- Has the project made an impact on terms of cross-cutting issues like gender equality, environment, good governance, conflict prevention etc?
Sustainability: The end line evaluation will seek to find out the extent to which the resilience gains made will likely continue after external funding ends over the medium and long term. The assessment will therefore look at the following;
- Project stakeholders ownership, how far they were consulted from the beginning and whether they agreed and will continue to remain in agreement after the end of the program
- Institutional capacities in existence that can sustain the resilience gains made over time
- Financial sustainability
Coherence: The criterion will look in the project compatibility with other interventions undertaken by Government or other donors within the areas of implementation. In line with this, the assessment will look at the following;
- Likeliness that results and impacts will mutually reinforce one another
- Likeness that results and impacts will duplicate or conflict with one another
Objective 2: Assess the impact of the programme with particular focus on establishing changes that have occurred as measured by key result indicators as stipulated in the logframe
Objective 3: Identify and assess key lessons learned, challenges and draw recommendation for future programming interventions
Objective 4: Assess sustainability of the project interventions beyond donor funding
Objective 5: Conduct tracer study of the TVET graduates to assess their employability and measure the extent to which skills acquired through the program have improved their status
Objective 6: Conduct tracer study on the VSLA groups formed through the program and measure the extent to which members have benefitted from being in the group and the extent to which their livelihood status has been impacted
- Evaluation Questions
- What are the endline values for the key indicators as provided for in Table 1?
- What are the current household and community practices in relation to resilience building, agriculture, natural resources management, income generating activities, livelihood strategies, food consumption practices, and copping strategies?
- What are the current household and community asset holdings in the impact area?
- What are the common vulnerabilities and gender disabilities in the impact area and which groups of people are most affected?
- Which interventions contributed most to the outcome indicators of the project?
- Which interventions demonstrated positive correlation with multiple positive outcomes and why?
- What attitudes and beliefs do people in the communities have towards different vulnerable groups for example women and girls, lesser-clans, people with disabilities, landless farmers, and single-headed households?
- What is the knowledge level of people in the impact area on issues of inclusion and gender?
- What economic and livelihood opportunities exist in the impact area for vulnerable groups (i.e. women and girls, lesser-clans, people with disabilities, landless farmers, single-headed household)?
- Has the project addressed the identified community vulnerabilities and gender disparities?
- Were the implemented interventions in line with the priorities of the households and communities?
- Which livelihood pathways have been observed to be most successful and why?
- What are the thresholds for maintaining various livelihood pathways supported by the programme?
- What is the progress of attaining these livelihood thresholds across gender,minority ,and age?
- What is the relationship between attaining thresholds for maintaining livelihoods and vulnerability?
- Were the funds used economically, effectively and efficiently?
- Was the implementation of activities timely and appropriately sequenced?
- Have there been any unintended outcomes?
- What were the enabling/disabling factors in achieving/impeding project outcomes?
- What are the prospects for sustainability of the project after the external funding ends?
- What were the project best practices and lessons learnt?
- What has been the impact of the project on;
- The direct beneficiaries, disaggregated by gender and disability
- Community
- Natural Resource nad Environment
- Evaluation Methodology
The end line evaluation shall adopt a cross-sectional survey research design that incorporates both qualitative and quantitative data collection as well as analysis techniques. The consultant shall be expected to make extensive use of the available monitoring and evaluation data collected by the project staff during the project implementation period. These will include but not limited to; baseline evaluation report, post distribution monitoring reports and findings from the beneficiary feedback mechanisms.
In order to ensure the quality of evidence is maintained, the evaluation will be designed with reference to the Bond Evidence Principles Checklist annexed to the TOR. Specific reference will be made to the 5 key dimensions of voice and inclusion, appropriateness, triangulation, contribution and transparency.
The qualitative and quantitative data collection will include the use of various approaches to inform the outcomes of the project. These will include;
Document Review
After reviewing the project design, log-frame and baseline, semi and annual monitoring report, donor guidelines, the information gaps will be determined and the scope of other data collection methods will be refined accordingly
Household Survey
Household survey questionnaires will be developed beforehand and approved by the project M&E team. The consultant will also ensure that the survey tool designed covers all Goals and Outcome level and EU indicators stated in matrix as well as add more questions to the questionnaire as appropriate.
Some of the key data to be collected using household questionnaire include but not limited to household demographics, household’s exposure to shocks (both covariate and idiosyncratic), household resilience and wellbeing indicators for example household productive assets, household expenditure/income, household food security and coping strategies –I.e., Food Consumption Score (FCS), Household Hunger Scale (HHS), reduced Consumption Strategy Index (rCSI), and Household Food Insecurity Access Scale (HFIAS)
The Project M&E team recommends that the consultant employs mobile data collection for quantitative data. The consultant will also be responsible for engaging and managing the team of data enumerators during the data collection period. The consultant will make use of SomReP data management platform for the quantitative data collection.
Key Informant Interviews (KII)
The consultant will be expected to develop a KII guide to be approved by the Project M&E team for the key informant interviews with the implementing partners project staff members, village development committee members, different committees groups not limited to; the farmers association, natural resource management, water committees, early warning early action committees as well as representatives from the relevant line ministries in order to gather deeper understanding on the project implementation, delivery as well as impact realized.
Focus Group Discussion
Focus Group Discussions (FGDs) will be held with project beneficiaries and partners to obtain qualitative responses and data of relevance with respect to the scope and requirements of the Evaluation Matrix in the appendices
Field Visit Observation
It is expected that there will be field visits by the evaluation team to more directly observe actual practice or observable results from the project’s activities, which should also be guided by the Evaluation Matrix.
A Field visit and observation guide will be developed by the evaluation lead and oriented to evaluation team prior to the visit.
Study Population and Sampling Frame
The study population will comprise 54 villages in the 11 districts where the project has been implemented. The districts comprise of the following livelihood zones; pastoral, agro-pastoral, peri-urban and Internally Displaced Persons (IDPs). The study sample (for quantitative survey) will need to align to that used during the baseline assessment and be representative of the livelihood zones and the villages.
Sample Technique and Sample Size Calculation
For qualitative survey, a purposive sampling technique will be used to sample direct beneficiaries (pastoral, agro-pastoral, peri-urban and Internally Displaced Persons (IDPs)), community members (i.e., men, women, boys and girls including people with disabilities), government stakeholders, and strategic partners for key informants and focus group discussion. The sample size for qualitative group will governed by the golden rule of data saturation[1]. The focus group discussions at community level shall ensure all gender groups (men, women, boys and girls) and marginalized groups (lesser-clans, people with disabilities, landless farmers, and single-headed households) are given equal opportunity to participate in the study.
For quantitative survey, a probability proportion to size sampling technique will be used to sample beneficiaries. SomReP recommends that at each household, the household head be interviewed. In the event that the household head is not available, the spouse shall be considered.
To determine the sample size for a quantitative survey, two important statistical parameters will be considered: the survey’s margin of error and confidence level. A margin of error of 8% and confidence interval of 95% confidence levels are recommended[2] to arrive at the total number of respondents per district. The sample size will be determined using the formula below given by Krejcie and Smith[3], 1979.
Sample size n=Z2xP(1-p)e2Z2xP(1-P)e2N
Quantitative Data Analysis
- Evaluation analysis for each question should demonstrate gender ,age and livelihood disagregation and explore understanding of the differences in observed in outcomes.
- Relationships between outcomes and interventions must be tested
- All indicators should be analysised by district and programme level
Qualitative Data analysis
The consultant will be expected to apply explorative tools and techniques such as outcome harvesting or outcome mapping to inform the identification of project outcomes both intended and unintended. Through a combination of Focus Group Discussions (FGDs), Key Informant Interviews(KIIs)and In-depth Interviews (IDIs) with key stakeholders including beneficiaries, community leaders, local government offices, civil society organisations , donors, local implementing partners, project staff and others to complement the information collected through quantitative approaches.The consultant is expected to apply narrative analysis for conversations with community groups while apply Phenomenological Analysis for IDIs and detail the questions as well as coding approach that will be applied in each case.Additionally the consulant is expected to demonstrate the use of outcome harvesting in mapping out changes resulting from the programme.
- Authority and Responsibility
SomReP
- Ensure that the consultant and his team adhere to research ethics and child protection policy
- Ensure quality assurance throughout the evaluation period
- Provide the selected consultant with necessary documents to enable a clear reporting and understanding of the EU-Restore and SIDA projects.
- Create linkages with field staff, Implementing Partners, Government line ministries as well as other stakeholders to ensure effective accomplishment of the end line evaluation.
- Review and approve field approaches proposed by the consultant.
- Review the draft report and provide feedback to the consultant.
- Approve the inception and final reports
Consultant
- Acquire all clearances and permissions necessary to collect data in all needed regions where the project was implemented
- Develop inception report to be approved by the Project M&E team before actual roll out of the assessment detailing s